02 February 2016

For Top 100 Prospect Lists, F Is A Passing Grade

Executive Summary:

  • Team controlled players contributed 36% of total wins from 1997-2006 and roughly 52% of wins from 2007-2015 indicated the increased value of a successful farm system.
  • Team-controlled position players and team-controlled pitchers contribute roughly the same percentage of production.
  • The top twenty-five team-controlled pitchers produced 60% of total production and the top twenty-five team controlled position players produced 50% of total production.
  • Baseball America successfully identified fewer top ten team-controlled position players and team-controlled position players ranked 26th to 50th than they have in the past. 
  • Ranked team-controlled position players produced 72% of production from 1997 to 2005 but only 61% from 2006 to 2015.
  • Baseball America has volatile results when predicting pitching performance. They were especially successful from 2008-2011 but have largely failed to maintain that success.

There have been a number of articles over the years attempting to determine the value of ranked prospects.  These articles, with the exception of my own, have largely followed the same methodology: determine how much production prospects ranked  for given years at a given spot produced over six years. My work at Camden Depot on the subject differs because I determine the actual time that each prospect is under team control and use actual salaries instead of approximated salaries for each prospect.

The major problem with these methodologies is that it requires most players ranked in a given year to have completed their team controlled years so that their performance can be assessed. This means that it doesn’t allow the audience to determine an idea of prospects’ current worth, but rather only a retrospective view. It’s helpful to know how prospects performed in the mid-2000s, but it would be far better to know how they performed in 2015.  

Therefore, instead of judging prospects by when they were ranked, I decided to determine how much production team controlled players have provided in each year from 1996 to 2014 provided that they were still under team control. A prospect that was ranked #8 in 2011 and didn’t make it to the majors until the middle of 2013 would have his 2013 contributions calculated in 2013 and his 2014 contributions calculated in 2014. In 2020, when he used up his six years of service time, his contributions would not count towards this ranking because he would no longer be a team-controlled player even if he signed an extension. This allows the reader a more current view.

I use RA9_WAR to determine production, Lew Pollis’ documentation of the value of a win while valuing a 2014 win at $7.5M and a 2015 win at $8M, determine actual salaries via Cot’s and the Lehman Database and determine service time also via Cots where available and by my own algorithms and Baseball Reference when not available.  

The chart below shows how much production that ranked and unranked position and pitching prospects contributed each year from 1997 to 2015. While ranked position players have contributed the most production, the graph is reasonably volatile. There was a period from 2008 to 2011 where ranked pitching prospects were at their peak value. Before that, from 2003 to 2006, ranked pitching prospects produced less value than usual. Meanwhile, unranked position prospects did well in 2014 and 2015 and poorly in 2003 and 2004.  All in all, ranked position team-controlled players have contributed the most value, followed by ranked pitchers, unranked position players and unranked pitchers as one would have expected.


This next graph shows how much production team controlled players have produced when ranked, unranked and total by both year and whether they’re position players or pitchers. This graph measures production as a percent and divides the production produced by players in a given category by either all pitchers or position players.

The first thing that is that team controlled position and pitching players have produced a similar percentage of total pitching and position production. One problem when measuring pitching and position production is that position players can earn 570 WAR in a year while pitchers can only earn 430 WAR. This makes sense because position players catch pitches and therefore can frame them as well as field balls put into play, but it does mean that comparing position and pitching production can be like comparing apples and oranges.

The second thing is that team controlled players, both pitching and position, have become more valuable over time. From 1998-2006, team controlled players produced only 36% of total production while from 2007-2015, they’ve produced over 50% of total production. It has certainly become more important for teams to be able to successfully develop their top talent due to the value that younger, team controlled players provide.

This next graph shows the production that ranked players have provided divided by total production for both pitchers and position players by year. This graph shows that ranked position players aren’t producing as much value as they once did. For whatever reason, Baseball America seems to be doing a worse job adequately grading position players than they have done previously. It also shows that Baseball America has periods where they accurately grade pitching prospects and where they do less well.

The margin of error appears to be small for Baseball America and this is because most production is produced by a few top players. On average, the top ten team-controlled pitchers contribute 30.4% of total production while the next fifteen contributes another 30.6%.  Team controlled pitchers ranked below 100 contribute -37.9% of total production due to being primarily below replacement level. It should come as no surprise that the Gini coefficient for team-controlled pitchers is .911.

As a result, team controlled ranked pitchers were extremely successful in 2008-2011 because on average they successfully ranked 8 of the top 10 team-controlled pitching prospects instead of the average 6. They only hit on seven of the top ten in 2008, but they also ranked the top five in that year. In 2011, they only missed on the #4, #6 and #10 pitching prospect and this also inflated their results.  The chart below shows the percent of pitchers ranked whose production was ranked between 1-10, 11-25 and 26-50.

Likewise, the top ten team controlled position players contributed 25.2% of total production while the next fifteen contributed 26%. Team controlled position players ranked below 100 contributed -18% of production due to being below replacement level. The Gini coefficient is slightly smaller for team-controlled position players, but still high at .871.

As a result, Baseball Americas numbers look considerably worse because they were only able to rank 5 of the top 10 position prospects in 2014 and 2015 compared to their average of between 7 and 9. It doesn’t help that they’ve also been unable to rank more than 50% of position players that produced between the 26th and 50th total WAR for position players.

To sum up, team controlled players are becoming more valuable now than they have been in the past. They contributed 36% of total wins from 1997-2006 and roughly 52% of wins from 2007-2015. This indicates that youth is becoming more important.

Pitchers contribute only 430 WAR per season while position players contribute 570 WAR per season. Therefore, in absolute terms, team-controlled position players contribute more production than team-controlled pitchers. However, in relative terms, team-controlled position players and pitchers contribute roughly the same percentage of production, indicating that it’s important to develop both position players and pitching.

Relatively few team-controlled players contribute the vast majority of total team controlled production. The top ten team-controlled pitchers produced 30% of total production and the next fifteen produced another 30%. The top twenty five position players produced half of total production.

Baseball America isn’t doing as effective of a job ranking position players. They’ve predicted fewer top ten position players in recent years and have done a poor job ranking position players ranked 25th through 50th over the past two years. As a result, ranked team-controlled position players produced 72% of total production from 1997-2005 and only 61% from 2006-2015. They only produced roughly 50% in 2014-2015.

Baseball America’s performance is very volatile when it comes to pitching prospects. They were successful ranking pitching prospects from 2000-2003 and 2008-2011 and less successful during other periods. This is because missing on a few top pitching prospects can have drastic impacts. It isn’t clear whether Baseball America has gotten worse at predicting pitcher performance.

It seems that baseball prospects are becoming more and more valuable as players are expected to be productive at younger ages. This makes publications like Baseball America more important and means that reliable rankings are important to adequately assess the status of baseball teams.


Roger said...

How much do you think players coming from Cuba and Korea at more advanced ages (i.e team controllable years at older ages) skews the accuracy of BA or skews your results to thinking that there are better performing younger players.

Matt Perez said...

That's a really good question.

It doesn't skew the accuracy of BA by much. It should be relatively easy for BA to rank the top advanced players that come from South America and Asia, and indeed they do so. If anything, it should improve their performance.

It also probably doesn't seriously skew my results thinking that there are better performing younger players. My methodology takes actual contracts into consideration, which given that it also doesn't take signing bonus into consideration, in essence penalizes those players that come over from Asia. If they earn $12M in 2015, then they receive a 1.5 WAR penalty due to their contract. In general, the most valuable players are those that are good and making minimum wage.

I'm also not sure a guy like Darvish should be treated differently than a guy like Donaldson. Well, maybe. Cespedes became an FA despite the fact that he only played four years and the same goes for Chen. But I think the main point of interest about young players is that it's hard to predict their performance as opposed to players that have been in the majors by awhile.

I suppose one could go either way on whether it's necessary to exclude these players from the analysis. What do you think? I tend to lean towards keeping them in but certainly can see the other side.

Roger said...

Keep them in. The study seems to me to be about "team control" and "cheap performance" and that's what teams are going for with the more experienced Asian and Caribbean players. The main question for me was how it might skew BA's performance which you indicate has been inconsistent. I thought all the ratings agencies stopped rating prospects at 25-26yo. That's what kinda got me thinking. The best part of this analysis is that it clearly shows the trend we are seeing for teams to value high, stable, predictable potential and cheap early years with team control. Who wouldn't want Machado (or Trout or Harper) for $500K or even $6M for that matter? As he plays, the risk of injury or decline increases - especially in the high cost FA years. The real wheeling-dealing comes with high potential untested prospects and FA-to-be in the 28-32yo range - also the core of the QO/compensation pick drama. That's where you really have to assess risk/reward and take the biggest chances.

Jon Shepherd said...

Scouting journals stop ranking prospects when they no longer qualify for Rookie of the Year status. That is a rather standard practice.

Anonymous said...

This is why it kills me when they say they don't want to give up the 14th pick....It would probably be another Billy Rowell, Adam Loewen, Matt Riley, Matt Blobgood, Brandon Snyder etc, etc etc!

Matt Perez said...

That's a fascinating take on it. Did you miss the part where I noted that young team-controlled players are producing more wins than before? Regardless of the likelihood of a specific prospect being successful, it's impossible to win without prospects.

Roger - Agreed that's the trend. I think teams would prefer dealing with veteran players but don't have much of a choice but depend on prospects at this point.