Baseball America Is Striking Out Ranking Pitching

Previously, I noted that Baseball America has volatile results when ranking pitching prospects. Baseball America had significant success from 2008-2011 while having other poor stretches earlier in its rankings days. Unfortunately, it appears their performance has regressed when ranking pitchers in the top ten.

From 2010-2016, Baseball America ranked 18 unique pitchers in the top ten of its rankings. The pitcher with the best performance so far is the deceased Jose Fernandez, although Julio Teheran and Gerrit Cole are likely to overtake him in future years. In addition, pitchers like Shelby Miller, Alex Reyes and Julio Urias have the potential to be great pitchers. Pitchers like Stephen Strasburg, Jeremy Hellickson, Yu Darvish and Arlodis Chapman provided decent value to their club while still being under team-control.

However, when we compare these results to the pitchers ranked in the top ten from 2000-2009, it is easy to see how this crop of pitchers falls short. There were six unique pitchers from 2000-2009 ranked in the top ten that contributed 25 or more wins of surplus value as a team controlled player. The only pitchers with remaining eligibility that have contributed more than ten wins of value are Julio Teheran and Gerrit Cole, both of which are unlikely to contribute even 20 wins of surplus value. If Jose Fernandez was able to stay healthy, then he probably would have reached the 25 wins of surplus value mark, but unfortunately that wasn’t meant to be. Baseball America will need to hope that its 2016 trio of starters, Julio Urias, Alex Reyes and Lucas Giolito, are more successful than the previous 15 starters. Yet, Giolito has already been traded by the Nationals suggesting they don’t think he’ll be successful.

Even more egregious, Baseball America was successful at ranking pitchers from 2010-2016 that were ultimately worth over 20 wins, but they failed to rank them as top ten pitchers. Madison Bumgarner and Chris Sale were both ranked by Baseball America and were worth over 25 wins. However, both of these starters were ranked between 11 and 25th in 2010 and 2011 instead of top 10. In addition, with one year of eligibility remaining, Jake Arrieta was ranked between 51st and 100th in 2010 and has contributed 19.2 wins of surplus value. It’s likely that Arrieta will end up contributing roughly 25 wins of surplus value. This shows that there was a trio of pitchers from 2010-2016 that are likely to break at least 20 wins of surplus value, if not 25, but Baseball America failed to rank them in the top ten.

In addition, there are a lot of up and coming pitchers that are showing promise to break 20 wins that weren’t ranked in the top 10. Matt Harvey has barely thrown for more than 500 innings, but has still contributed 11.75 wins of surplus value and has two years of eligibility remaining. If he can stay healthy (a big if), then he’ll likely contribute more than 20 wins of surplus value. He was ranked between 51 and 100 by Baseball America, which makes him a big miss. It seems like Chris Archer has been around forever, but he still has three years of team-controlled eligibility remaining and has contributed over 10 wins of surplus value. With a team friendly contract, he has a strong chance of contributing more than 20 wins of surplus value as a team-controlled pitcher. He was ranked between 26 and 50 twice and between 51 and 100 once. Zach Britton may be a closer, but has still contributed over 9 wins of excess value and has two years of team-controlled eligibility remaining. He has a strong chance of contributing more than 10 wins of excess value and being a better value than nearly every single pitcher that Baseball America ranked in the top ten. Other starters like Sonny Gray, Aaron Sanchez, Jake Odorizzi, Noah Syndergaard, Yordano Ventura, Kevin Gausman, Michael Fulmer and Steven Matz are showing potential to contribute a considerable amount of excess value but weren’t ranked in the top 10 by Baseball America. There have been pitchers with strong potential to be great, but weren’t ranked highly enough by Baseball America.

Even worse than this is the number of top pitchers that weren’t ranked by Baseball America from 2010-2016 but started their careers during that period. Jose Quintana and Corey Kluber have two years each of team control remaining and have contributed 18.86 and 17.2 surplus value wins. They’re likely to have contributed at least 20 surplus value wins and maybe even 25 before all is said and done. Tanner Roark, Jacob DeGrom and Kyle Hendricks have each been worth more than 10 wins of surplus value. Roark has three years of team control remaining and the other two pitchers have four. These three pitchers are likely to contribute more than 20 surplus value wins and could get to the 25 mark. None of the three were ranked. Iwakuma, Keuchel, Miley, Lynn and Miguel Gonzalez have all contributed more than 10 wins of surplus value and have at least one year of team control remaining. Keuchel and Iwakuma are likely to contribute roughly 15 wins of surplus value.

From 2010-2016, there have been 76 pitchers that have contributed at least 5 surplus value wins to their clubs as team controlled players. Only 37 of these pitchers were ranked from 2010-2016. The ranked and unranked pitchers have contributed roughly the same amount of value so far. It will start to get ugly soon however. Only 3 pitchers out of the top 10 ranked pitchers still have eligibility remaining, Arrieta with one year and Teheran/Harvey with two years. All ten of the top unranked pitchers have eligibility remaining, and 6 of the 10 have more than one year remaining. Once that happens, the unranked pitchers will likely be much more valuable than the ranked pitchers.

More to the point, the top ten pitchers from 2010-2016 so far have probably been Bumgarner, Sale, Quintana, Kluber, Roark, Arrieta, Fernandez, Strasburg, DeGrom and Teheran. Only three of these pitchers were ranked in the top ten and only Teheran has a chance to stay in the top ten in a few years. Four of the ten were not ranked from 2010-2016. None of the top six were ever ranked in the top ten by Baseball America either. Pitchers like Keuchel, Hendricks, Cole and Archer are nearly in the top ten already and have multiple years to break into the top ten while Fernandez and Strasburg are out of eligibility. It’s very possible that only one pitcher ranked in the top ten by Baseball America from 2010-2016 will end up being one of the top ten pitchers that had their rookie year during that period.

Given a poor six year stretch of failing to rank the top pitching prospects in the top ten, it is highly likely that people using Baseball America to rate prospects will notice a problem around 2020. If they can't predict the top pitchers, then their rankings will lose significant value.

For Top 100 Prospect Lists, F Is A Passing Grade

Executive Summary:

• Team controlled players contributed 36% of total wins from 1997-2006 and roughly 52% of wins from 2007-2015 indicated the increased value of a successful farm system.
• Team-controlled position players and team-controlled pitchers contribute roughly the same percentage of production.
• The top twenty-five team-controlled pitchers produced 60% of total production and the top twenty-five team controlled position players produced 50% of total production.
• Baseball America successfully identified fewer top ten team-controlled position players and team-controlled position players ranked 26^th to 50^th than they have in the past. 
• Ranked team-controlled position players produced 72% of production from 1997 to 2005 but only 61% from 2006 to 2015.
• Baseball America has volatile results when predicting pitching performance. They were especially successful from 2008-2011 but have largely failed to maintain that success.

There have been a number of articles over the years attempting to determine the value of ranked prospects.  These articles, with the exception of my own, have largely followed the same methodology: determine how much production prospects ranked  for given years at a given spot produced over six years. My work at Camden Depot on the subject differs because I determine the actual time that each prospect is under team control and use actual salaries instead of approximated salaries for each prospect.

The major problem with these methodologies is that it requires most players ranked in a given year to have completed their team controlled years so that their performance can be assessed. This means that it doesn’t allow the audience to determine an idea of prospects’ current worth, but rather only a retrospective view. It’s helpful to know how prospects performed in the mid-2000s, but it would be far better to know how they performed in 2015.  

Therefore, instead of judging prospects by when they were ranked, I decided to determine how much production team controlled players have provided in each year from 1996 to 2014 provided that they were still under team control. A prospect that was ranked #8 in 2011 and didn’t make it to the majors until the middle of 2013 would have his 2013 contributions calculated in 2013 and his 2014 contributions calculated in 2014. In 2020, when he used up his six years of service time, his contributions would not count towards this ranking because he would no longer be a team-controlled player even if he signed an extension. This allows the reader a more current view.

I use RA9_WAR to determine production, Lew Pollis’ documentation of the value of a win while valuing a 2014 win at $7.5M and a 2015 win at $8M, determine actual salaries via Cot’s and the Lehman Database and determine service time also via Cots where available and by my own algorithms and Baseball Reference when not available.  

The chart below shows how much production that ranked and unranked position and pitching prospects contributed each year from 1997 to 2015. While ranked position players have contributed the most production, the graph is reasonably volatile. There was a period from 2008 to 2011 where ranked pitching prospects were at their peak value. Before that, from 2003 to 2006, ranked pitching prospects produced less value than usual. Meanwhile, unranked position prospects did well in 2014 and 2015 and poorly in 2003 and 2004.  All in all, ranked position team-controlled players have contributed the most value, followed by ranked pitchers, unranked position players and unranked pitchers as one would have expected.

 

This next graph shows how much production team controlled players have produced when ranked, unranked and total by both year and whether they’re position players or pitchers. This graph measures production as a percent and divides the production produced by players in a given category by either all pitchers or position players.



The first thing that is that team controlled position and pitching players have produced a similar percentage of total pitching and position production. One problem when measuring pitching and position production is that position players can earn 570 WAR in a year while pitchers can only earn 430 WAR. This makes sense because position players catch pitches and therefore can frame them as well as field balls put into play, but it does mean that comparing position and pitching production can be like comparing apples and oranges.

The second thing is that team controlled players, both pitching and position, have become more valuable over time. From 1998-2006, team controlled players produced only 36% of total production while from 2007-2015, they’ve produced over 50% of total production. It has certainly become more important for teams to be able to successfully develop their top talent due to the value that younger, team controlled players provide.

This next graph shows the production that ranked players have provided divided by total production for both pitchers and position players by year. This graph shows that ranked position players aren’t producing as much value as they once did. For whatever reason, Baseball America seems to be doing a worse job adequately grading position players than they have done previously. It also shows that Baseball America has periods where they accurately grade pitching prospects and where they do less well.

The margin of error appears to be small for Baseball America and this is because most production is produced by a few top players. On average, the top ten team-controlled pitchers contribute 30.4% of total production while the next fifteen contributes another 30.6%.  Team controlled pitchers ranked below 100 contribute -37.9% of total production due to being primarily below replacement level. It should come as no surprise that the Gini coefficient for team-controlled pitchers is .911.

As a result, team controlled ranked pitchers were extremely successful in 2008-2011 because on average they successfully ranked 8 of the top 10 team-controlled pitching prospects instead of the average 6. They only hit on seven of the top ten in 2008, but they also ranked the top five in that year. In 2011, they only missed on the #4, #6 and #10 pitching prospect and this also inflated their results.  The chart below shows the percent of pitchers ranked whose production was ranked between 1-10, 11-25 and 26-50.

Likewise, the top ten team controlled position players contributed 25.2% of total production while the next fifteen contributed 26%. Team controlled position players ranked below 100 contributed -18% of production due to being below replacement level. The Gini coefficient is slightly smaller for team-controlled position players, but still high at .871.

As a result, Baseball Americas numbers look considerably worse because they were only able to rank 5 of the top 10 position prospects in 2014 and 2015 compared to their average of between 7 and 9. It doesn’t help that they’ve also been unable to rank more than 50% of position players that produced between the 26^th and 50^th total WAR for position players.

To sum up, team controlled players are becoming more valuable now than they have been in the past. They contributed 36% of total wins from 1997-2006 and roughly 52% of wins from 2007-2015. This indicates that youth is becoming more important.

Pitchers contribute only 430 WAR per season while position players contribute 570 WAR per season. Therefore, in absolute terms, team-controlled position players contribute more production than team-controlled pitchers. However, in relative terms, team-controlled position players and pitchers contribute roughly the same percentage of production, indicating that it’s important to develop both position players and pitching.

Relatively few team-controlled players contribute the vast majority of total team controlled production. The top ten team-controlled pitchers produced 30% of total production and the next fifteen produced another 30%. The top twenty five position players produced half of total production.

Baseball America isn’t doing as effective of a job ranking position players. They’ve predicted fewer top ten position players in recent years and have done a poor job ranking position players ranked 25^th through 50^th over the past two years. As a result, ranked team-controlled position players produced 72% of total production from 1997-2005 and only 61% from 2006-2015. They only produced roughly 50% in 2014-2015.

Baseball America’s performance is very volatile when it comes to pitching prospects. They were successful ranking pitching prospects from 2000-2003 and 2008-2011 and less successful during other periods. This is because missing on a few top pitching prospects can have drastic impacts. It isn’t clear whether Baseball America has gotten worse at predicting pitcher performance.

It seems that baseball prospects are becoming more and more valuable as players are expected to be productive at younger ages. This makes publications like Baseball America more important and means that reliable rankings are important to adequately assess the status of baseball teams.