From 2010-2016, Baseball America ranked 18 unique pitchers in the top ten of its rankings. The pitcher with the best performance so far is the deceased Jose Fernandez, although Julio Teheran and Gerrit Cole are likely to overtake him in future years. In addition, pitchers like Shelby Miller, Alex Reyes and Julio Urias have the potential to be great pitchers. Pitchers like Stephen Strasburg, Jeremy Hellickson, Yu Darvish and Arlodis Chapman provided decent value to their club while still being under team-control.
However, when we compare these results to the pitchers ranked in the top ten from 2000-2009, it is easy to see how this crop of pitchers falls short. There were six unique pitchers from 2000-2009 ranked in the top ten that contributed 25 or more wins of surplus value as a team controlled player. The only pitchers with remaining eligibility that have contributed more than ten wins of value are Julio Teheran and Gerrit Cole, both of which are unlikely to contribute even 20 wins of surplus value. If Jose Fernandez was able to stay healthy, then he probably would have reached the 25 wins of surplus value mark, but unfortunately that wasn’t meant to be. Baseball America will need to hope that its 2016 trio of starters, Julio Urias, Alex Reyes and Lucas Giolito, are more successful than the previous 15 starters. Yet, Giolito has already been traded by the Nationals suggesting they don’t think he’ll be successful.
Even more egregious, Baseball America was successful at ranking pitchers from 2010-2016 that were ultimately worth over 20 wins, but they failed to rank them as top ten pitchers. Madison Bumgarner and Chris Sale were both ranked by Baseball America and were worth over 25 wins. However, both of these starters were ranked between 11 and 25th in 2010 and 2011 instead of top 10. In addition, with one year of eligibility remaining, Jake Arrieta was ranked between 51st and 100th in 2010 and has contributed 19.2 wins of surplus value. It’s likely that Arrieta will end up contributing roughly 25 wins of surplus value. This shows that there was a trio of pitchers from 2010-2016 that are likely to break at least 20 wins of surplus value, if not 25, but Baseball America failed to rank them in the top ten.
In addition, there are a lot of up and coming pitchers that are showing promise to break 20 wins that weren’t ranked in the top 10. Matt Harvey has barely thrown for more than 500 innings, but has still contributed 11.75 wins of surplus value and has two years of eligibility remaining. If he can stay healthy (a big if), then he’ll likely contribute more than 20 wins of surplus value. He was ranked between 51 and 100 by Baseball America, which makes him a big miss. It seems like Chris Archer has been around forever, but he still has three years of team-controlled eligibility remaining and has contributed over 10 wins of surplus value. With a team friendly contract, he has a strong chance of contributing more than 20 wins of surplus value as a team-controlled pitcher. He was ranked between 26 and 50 twice and between 51 and 100 once. Zach Britton may be a closer, but has still contributed over 9 wins of excess value and has two years of team-controlled eligibility remaining. He has a strong chance of contributing more than 10 wins of excess value and being a better value than nearly every single pitcher that Baseball America ranked in the top ten. Other starters like Sonny Gray, Aaron Sanchez, Jake Odorizzi, Noah Syndergaard, Yordano Ventura, Kevin Gausman, Michael Fulmer and Steven Matz are showing potential to contribute a considerable amount of excess value but weren’t ranked in the top 10 by Baseball America. There have been pitchers with strong potential to be great, but weren’t ranked highly enough by Baseball America.
Even worse than this is the number of top pitchers that weren’t ranked by Baseball America from 2010-2016 but started their careers during that period. Jose Quintana and Corey Kluber have two years each of team control remaining and have contributed 18.86 and 17.2 surplus value wins. They’re likely to have contributed at least 20 surplus value wins and maybe even 25 before all is said and done. Tanner Roark, Jacob DeGrom and Kyle Hendricks have each been worth more than 10 wins of surplus value. Roark has three years of team control remaining and the other two pitchers have four. These three pitchers are likely to contribute more than 20 surplus value wins and could get to the 25 mark. None of the three were ranked. Iwakuma, Keuchel, Miley, Lynn and Miguel Gonzalez have all contributed more than 10 wins of surplus value and have at least one year of team control remaining. Keuchel and Iwakuma are likely to contribute roughly 15 wins of surplus value.
From 2010-2016, there have been 76 pitchers that have contributed at least 5 surplus value wins to their clubs as team controlled players. Only 37 of these pitchers were ranked from 2010-2016. The ranked and unranked pitchers have contributed roughly the same amount of value so far. It will start to get ugly soon however. Only 3 pitchers out of the top 10 ranked pitchers still have eligibility remaining, Arrieta with one year and Teheran/Harvey with two years. All ten of the top unranked pitchers have eligibility remaining, and 6 of the 10 have more than one year remaining. Once that happens, the unranked pitchers will likely be much more valuable than the ranked pitchers.
More to the point, the top ten pitchers from 2010-2016 so far have probably been Bumgarner, Sale, Quintana, Kluber, Roark, Arrieta, Fernandez, Strasburg, DeGrom and Teheran. Only three of these pitchers were ranked in the top ten and only Teheran has a chance to stay in the top ten in a few years. Four of the ten were not ranked from 2010-2016. None of the top six were ever ranked in the top ten by Baseball America either. Pitchers like Keuchel, Hendricks, Cole and Archer are nearly in the top ten already and have multiple years to break into the top ten while Fernandez and Strasburg are out of eligibility. It’s very possible that only one pitcher ranked in the top ten by Baseball America from 2010-2016 will end up being one of the top ten pitchers that had their rookie year during that period.
Given a poor six year stretch of failing to rank the top pitching prospects in the top ten, it is highly likely that people using Baseball America to rate prospects will notice a problem around 2020. If they can't predict the top pitchers, then their rankings will lose significant value.
Why does only performance under team control matter? To determine whether or not an evaluation is correct, the player's entire career must be measured, not just the first six years.
ReplyDeleteBaseball America ranks prospects. The value of prospects are that you have them under team control for a specified amount of time. If a prospect only becomes good when he's a free agent, then that has little value to the club that developed them. That's why prospect rankings only measure the period where players are under team control.
ReplyDeleteBy the by, very few players actually play for six entire years. Those players typically have something in common and that is that they were productive over those first six years. I mean, you don't see many teams saying, "Oooh, that pitcher has a 7.30 ERA. Let's have him be our ace!!!" Which means that there are very few players like a Porcello who were arguably bad during their period of being team controlled but became good as a free agent. So, measuring an entire career as opposed to the first six years has minimal impact on the final results.
I would also argue that if some said RA Dickey was a Cy Young level kind of pitcher when he was 21, that the evaluation given at that point in time has nothing to do with what he did 15 years later. I think we would be very hard pressed to find an evaluation that utterly missed for 10 years all of a sudden become relevant. Instead, something unforeseen probably transpired in those years to get the player to a level that may reflect the bottom line in the evaluation, but not the actual process in that evaluation to come up with the bottom line.
ReplyDeleteSo, two parts. One, Matt's league rules perspective, which has been the consensus way to measure this sing Victor Wang's work in 2006 or so. Two, ignoring those six seasons of team control rules, I am unsure how an evaluation would be relevant beyond those rules. I think in a practical sense that evaluation have a shelf life of about three or four years, which is one of the reasons why there is a considerable amount of noise.
The purpose of rating prospects is to identify who will be a better player, not who will be a better player in the first six years of his career. The value of a prospect TO A TEAM may very well be only in his first six years, but the value of a prospect AS A PLAYER covers his entire career. Thus, I maintain that evaluating the accuracy of prospect rankings, as opposed to determining the trade value of a prospect, requires more than an arbitrary six-year window.
ReplyDeleteI'm not talking about an R.A. Dickey here; I'm talking about the difference between a guy who delivers 4 WAR a year for years 1-6 and a guy who delivers 4 WAR a year for years 3-8. A ranking that rated the second guy a better prospect than the first is not necessarily inaccurate.
Obviously, RA Dickey was employed as an extreme example. That said, maybe you are onto something, but from my experience and from the experience of people in front offices who are hired to evaluate processes...this is how it is done across the board. Whatever issues limiting it to 6 years might have, it is considered irrelevant by people like the aforementioned Wang who is a baseball executive now. Perhaps, the whole train went off the tracks a decade ago and cannot see a better path, but I find your rationale to be unconvincing here.
ReplyDeleteAgain, an evaluation is not about the entire career of a player. I would find that assertion by any scout to be somewhat misguided. What a scout does is find value in the player as seen by the organization. This is done in near terms and in long term. If a scout somehow could divine value for a hiA player 8 years from now but not 3 years from now, that kind of evaluation is not going to be incredibly useful to an organization.
In a vacuum, your concept makes sense and you will find some studies back around 2002-2006 that tried to do that, but it makes much more functional sense to use near term value to assess the usefulness of an evaluation. 6 years control is arbitrary as an endpoint. Somewhere around there is likely the best endpoint, the MLB rules simply give that exact point more gravity. That said, focusing on the arbitrary nature of that endpoint will result in overlooking where and how value is derived from a player and how related an evaluation is to what actually the player becomes.
I should clarify that I'm not using a six-year window. It's the period that a player is under team control. A player like Tillman is going on his ninth year in the majors and is still under team control. Therefore, this year would be counted as a team-controlled year because he's still under team control. It's six years of team control which is quite different than six actual years.
ReplyDeleteSounds like you've found a study for yourself Joe. Determine the value of prospects over their first six team-controlled years and compare them to their entire careers (at least up to this point). Then you can see for yourself whether there's a large difference worth noting or whether there's only a minimal difference.
I think we're approaching this from different perspectives. I agree that from the standpoint of an executive or scout, looking at the relatively short-term perspective of "what's best for my team now", a short horizon of six years is appropriate. I'm not sure that the more-casual fan sees it that way.
ReplyDeleteAlso, just to reiterate what Matt and I have written, the horizon is player specific and ranges from 6-11 years. Most are about 8-9 years. These evaluations are about applied scouting and, roughly, what is best for my team 3-9 years out.
ReplyDeleteIn other words, I am unsure if scouts are considering what a player will be doing post-peak, which is what you will capture with the methodology you suggest.
ReplyDeleteAt that point, what exactly are we evaluating? Unintended performance?
Interesting but not super helpful if not compared to other prospect ranking(s). Are they pefromong better/worse than an BP? Keith Law? MLB.com etc.
ReplyDeleteBP is a poor comparison because the lists had vastly different ways of composing it every couple of years. I assume Law's rankings would perform similarly because he likes to make narrative picks. MLB.com with Callis, I assume, is now the gold standard.
DeleteIs a decrease in performance a valuable conclusion even without that comparison? Yes. It should make one think about what is different. Is it a change in talent with Callis departing or is it something unique to these players? Or something else.
It really depends upon the scope of what you want to know. Benchmarking certainly can be useful but it is not always a necessity.
Performing* damn not autocorrect!
ReplyDeleteAgreed that comparisons to other sources would be helpful. Unfortunately, the common methodology used requires at least ten years of rankings before we can do anything with the results. And also, other sources don't make it easy to collate their data. It's only possible at the moment to do a narrative type of article like this one instead of an analytic type of argument using data from 2010-2016.
ReplyDeleteI'm working on methodology to reduce the number of rankings necessary to grade players, but getting data from other sources will still be a challenge.