A Look At The Distribution Of Prospect Values

Over the past few years, there’s been a movement to grade prospects by not only overall rank and organization rank, but also by future value. Future Value is not very well defined, but per MLB is a 20-80 scale in which a ranking of 20-30 is well below average, 40 is below average, 50 is average, 60 is above average and 70-80 is well above average. A player with a 65 FV is someone who could develop into a future impact Major Leaguer, perhaps an All-Star-caliber standout. Fangraphs defines future value, using a similar grading model and provides a chart showing how future value can be defined using WAR.

In general, I’m not a fan of using such definitions as Top #2-3 player or #2/#3 starter because it’s subjective and can’t be easily determined and prefer using WAR values because that at least is easily definable and somewhat objective. WAR may undervalue a strong player that is hurt for a few years, but any clear definition will have its limitations.

In order to learn about future value, I decided to build a metric called actual value. Future value attempts to predict how much production a prospect will produce using WAR, and actual value attempts to measure how much production a prospect actually did produce. In essence, I defined actual value in such a way that future value attempts to predict actual value. 

To do this, I looked at players from 1990-2016 and measured each prospects’ production while they were under team control or until they had six or more years of service time. Once players have six or more years of service time, they are eligible for free agency. Then, I determined their rookie year and measured their average value per year or actual value using the following scale.

With this distribution of actual value illustrating how prospects performed, we can understand what a reasonable distribution for future value might look like for the future. I looked at the performance of prospects over two time periods, 1990-2016 and 2004-2013 defined by a prospects rookie year. Here were the valuations this process returned from 1990-2016.

Each year from 1990-2016, there were roughly 18.5 prospects that had an actual value per year of 2 WAR or more. This may seem surprising at first glance, but one should consider that from 2000-2017, there were roughly 130 offensive players and 90 pitchers that were worth more than 2 wins or above average. This includes players that were one year wonders like 2009 Jason Kubel (2.5 fWAR) or 2016 Michael A Taylor (3.2 fWAR). They may have been worth 2 fWAR in a single season, but weren’t above average players over an extended period of time. When looking at players that have staying power, there are significantly fewer than 220 at any given time – maybe around 150 total. If there are few average and above average players in the league, then there can be few prospects each year to replace them. If 20 prospects each year are above average, and there are 150 players total above average, then the replacement period would be about 7.5 years. This seems to suggest that 20 average prospects per year is within reason.

There were significantly more prospects each year that ended up being below average and the vast majority were either 25s or 30s. This makes a lot of sense because teams have injuries and need to call up their prospects to make it through the season. In addition, teams would prefer to give their prospects chances to be successful because they’re cheaper than free agents. However, the vast majority of prospects either fail or struggle to be better than replacement level.

The story is reasonably similar when looking at prospects that became rookies from 2004-2013. There were about 20 rookies per year that because average or better players in the majors. Another 22.5 per year were below average, but still worth over 1 WAR per season. The vast majority of rookies, or roughly 184 per year, ended up being worth less than 1 WAR per season or clearly below average. Both sets of data tell the same story: being successful in major league baseball is difficult.

If only twenty players per year have above average actual value, than prospects ranked outside of the top 100 should not be expected to have a high future value. After all, future value is trying to predict and should have similar behavior to actual value. Prospect lists are tricky because they grade all prospects and not just those that should be expected to enter the league each year. So, I would argue that it’s reasonable to presume that all prospects considered to be at least average should be in the top 100, and likely all prospects with a future value of 45. I would say that all prospects with a future value of 40 should be at least in the top 125 prospects and all prospects with a future value of 35 should be in the top 200 prospects. Players ranked worse than 200 should be seen as slightly better than replacement players at best and ideally will be either bench players, middle relief or minor league depth pieces.

Fangraphs provides a board with its top 850 prospects as ranked at the beginning of the season with both future value grades and estimated time of arrival. We can use this board to determine the time period that their prospect list measures as well as see whether their distribution seems reasonable.

If they still use the future value definitions defined by McDaniel, than their distribution is difficult to comprehend. In their defense, it appears that they consider their FV to be their peak level rather than their average level. In addition, they deserve some leeway for injuries. Still, it is highly unlikely that there were will be 54 prospects in the 2018 rookie class that end up being average or better or 110 prospects that are worth 1 fWAR or more. It seems that a 55 in the Fangraphs distribution is the same as a 50-55 in the historical distribution, a 50 in Fangraphs is the same as a 35-45 in the historical distribution, a 45 in Fangraphs is the same as a 30 in the historical distribution and a 40 in Fangraphs matching a 25 in the historical dataset. Perhaps this is why Fangraphs only gives valuations for players that are ranked up to 45?

Even prospects graded as a 30 have a surprisingly high amount of value using a WAR linear model. These players typically produce about .2 fWAR per year and last for about 4.5 years. Using our predicted arbitration rates would suggest that these players produce $9M in value (at $10M per win) and cost $2.5M for a total surplus value of $6.5M. This seems surprising at first glance, but consider that players like Caleb Joseph and Ryan Flaherty rate as having an actual value of 30. These players aren’t great, but Joseph is certainly a decent backup catcher option while Flaherty was a decent utility player. Despite the fact that these players are valuable, it is impossible to build an average team with players just like those two, explaining why teams are willing to trade them away for marginal improvements. Treating WAR as a linear model does have its limitations.

Players ranking 35, 40 and 45 in actual value have a preposterous amount of value using this metric. 35 actual value level players produce roughly 4.2 WAR of production over their six years of team control or produce $42M in value (at $10M per WAR) but only cost about $10.6M putting their value at $30M. 40 level players produce roughly 7.2 WAR of production over their six years of team control or $72M in value but only cost $15.6M putting their value at $56M. 45 level players produce roughly 10 WAR of production over their six years of team control, but cost roughly $24M putting their value at $76M. Presumably, Orioles fans would not have been happy if the Orioles received just one 45 level player (top 60-90 prospect) in return for Machado.

Part of the reason why these players have such high values is that non-arbitration player salaries are unfairly low. If a win is valued at $10M, and the salary for a non-arb player is $550k, then it is relatively easy for a non-arb player to be underpaid. A reasonable minimum salary of $1.5M would rectify the situation where near replacement players have significant value. Part of the reason is that having a slightly above replacement player play instead of a below replacement player can have significant value. The fact is that teams are willing to pay a premium to avoid having to use a player like David Hess for 65 innings.

Future value may be the attempt to determine actual value, but that doesn’t mean they are the same. There’s a significant amount of uncertainty when predicting future value, and the distribution of prospect performances suggest that it is far easier to overestimate a prospect than underestimate a prospect. Most prospects that make it to the majors aren’t successful, and all prospects that don’t make it to the majors aren’t successful by definition. This uncertainty will mean that a player with an actual value of 40 (as determined in hindsight) is far more valuable than a prospect with a future value of 40. In a player with an actual value of 40 is worth $56M, than a prospect with a future value of 40 is worth significantly less. This is one reason why prospects aren’t valued as actual value suggests. In contrast, measuring the value of top 100 prospects take failed prospects into account and therefore provide a reasonable baseline for value.

In addition, a team filled with 35 AV players that are under team control, may be fiscally successful, but will also be terrible. In some roles, a 30 or 35 AV player can be acceptable, but in other roles (starters) they’re not particularly ideal. Obviously, it’s important to be fiscally responsible, but teams are in the business of maximizing their win total and not their excess value total. Others believe that teams would rather win now rather than later, and therefore future wins should be discounted.

Going forward, looking at how many prospects historically fit into each category of actual value should help analysts build future lists using future value. It doesn’t make sense to predict that hundreds of prospects will have a future value of 45 if few players ultimately have an actual value of 45. Using historical data to accurately assess the value of prospects will lower these rankings and make it more difficult to excite casual fans about their teams’ future. Still, even if it’s bad press, it makes sense to admit the reality of prospect value. Having a better idea how prospects historically perform can help teams plan for their future, and ultimately fans don’t really want to see how their prospects develop but rather watch their team be successful at the major league level.

Baseball America Is Striking Out Ranking Pitching

Previously, I noted that Baseball America has volatile results when ranking pitching prospects. Baseball America had significant success from 2008-2011 while having other poor stretches earlier in its rankings days. Unfortunately, it appears their performance has regressed when ranking pitchers in the top ten.

From 2010-2016, Baseball America ranked 18 unique pitchers in the top ten of its rankings. The pitcher with the best performance so far is the deceased Jose Fernandez, although Julio Teheran and Gerrit Cole are likely to overtake him in future years. In addition, pitchers like Shelby Miller, Alex Reyes and Julio Urias have the potential to be great pitchers. Pitchers like Stephen Strasburg, Jeremy Hellickson, Yu Darvish and Arlodis Chapman provided decent value to their club while still being under team-control.

However, when we compare these results to the pitchers ranked in the top ten from 2000-2009, it is easy to see how this crop of pitchers falls short. There were six unique pitchers from 2000-2009 ranked in the top ten that contributed 25 or more wins of surplus value as a team controlled player. The only pitchers with remaining eligibility that have contributed more than ten wins of value are Julio Teheran and Gerrit Cole, both of which are unlikely to contribute even 20 wins of surplus value. If Jose Fernandez was able to stay healthy, then he probably would have reached the 25 wins of surplus value mark, but unfortunately that wasn’t meant to be. Baseball America will need to hope that its 2016 trio of starters, Julio Urias, Alex Reyes and Lucas Giolito, are more successful than the previous 15 starters. Yet, Giolito has already been traded by the Nationals suggesting they don’t think he’ll be successful.

Even more egregious, Baseball America was successful at ranking pitchers from 2010-2016 that were ultimately worth over 20 wins, but they failed to rank them as top ten pitchers. Madison Bumgarner and Chris Sale were both ranked by Baseball America and were worth over 25 wins. However, both of these starters were ranked between 11 and 25th in 2010 and 2011 instead of top 10. In addition, with one year of eligibility remaining, Jake Arrieta was ranked between 51st and 100th in 2010 and has contributed 19.2 wins of surplus value. It’s likely that Arrieta will end up contributing roughly 25 wins of surplus value. This shows that there was a trio of pitchers from 2010-2016 that are likely to break at least 20 wins of surplus value, if not 25, but Baseball America failed to rank them in the top ten.

In addition, there are a lot of up and coming pitchers that are showing promise to break 20 wins that weren’t ranked in the top 10. Matt Harvey has barely thrown for more than 500 innings, but has still contributed 11.75 wins of surplus value and has two years of eligibility remaining. If he can stay healthy (a big if), then he’ll likely contribute more than 20 wins of surplus value. He was ranked between 51 and 100 by Baseball America, which makes him a big miss. It seems like Chris Archer has been around forever, but he still has three years of team-controlled eligibility remaining and has contributed over 10 wins of surplus value. With a team friendly contract, he has a strong chance of contributing more than 20 wins of surplus value as a team-controlled pitcher. He was ranked between 26 and 50 twice and between 51 and 100 once. Zach Britton may be a closer, but has still contributed over 9 wins of excess value and has two years of team-controlled eligibility remaining. He has a strong chance of contributing more than 10 wins of excess value and being a better value than nearly every single pitcher that Baseball America ranked in the top ten. Other starters like Sonny Gray, Aaron Sanchez, Jake Odorizzi, Noah Syndergaard, Yordano Ventura, Kevin Gausman, Michael Fulmer and Steven Matz are showing potential to contribute a considerable amount of excess value but weren’t ranked in the top 10 by Baseball America. There have been pitchers with strong potential to be great, but weren’t ranked highly enough by Baseball America.

Even worse than this is the number of top pitchers that weren’t ranked by Baseball America from 2010-2016 but started their careers during that period. Jose Quintana and Corey Kluber have two years each of team control remaining and have contributed 18.86 and 17.2 surplus value wins. They’re likely to have contributed at least 20 surplus value wins and maybe even 25 before all is said and done. Tanner Roark, Jacob DeGrom and Kyle Hendricks have each been worth more than 10 wins of surplus value. Roark has three years of team control remaining and the other two pitchers have four. These three pitchers are likely to contribute more than 20 surplus value wins and could get to the 25 mark. None of the three were ranked. Iwakuma, Keuchel, Miley, Lynn and Miguel Gonzalez have all contributed more than 10 wins of surplus value and have at least one year of team control remaining. Keuchel and Iwakuma are likely to contribute roughly 15 wins of surplus value.

From 2010-2016, there have been 76 pitchers that have contributed at least 5 surplus value wins to their clubs as team controlled players. Only 37 of these pitchers were ranked from 2010-2016. The ranked and unranked pitchers have contributed roughly the same amount of value so far. It will start to get ugly soon however. Only 3 pitchers out of the top 10 ranked pitchers still have eligibility remaining, Arrieta with one year and Teheran/Harvey with two years. All ten of the top unranked pitchers have eligibility remaining, and 6 of the 10 have more than one year remaining. Once that happens, the unranked pitchers will likely be much more valuable than the ranked pitchers.

More to the point, the top ten pitchers from 2010-2016 so far have probably been Bumgarner, Sale, Quintana, Kluber, Roark, Arrieta, Fernandez, Strasburg, DeGrom and Teheran. Only three of these pitchers were ranked in the top ten and only Teheran has a chance to stay in the top ten in a few years. Four of the ten were not ranked from 2010-2016. None of the top six were ever ranked in the top ten by Baseball America either. Pitchers like Keuchel, Hendricks, Cole and Archer are nearly in the top ten already and have multiple years to break into the top ten while Fernandez and Strasburg are out of eligibility. It’s very possible that only one pitcher ranked in the top ten by Baseball America from 2010-2016 will end up being one of the top ten pitchers that had their rookie year during that period.

Given a poor six year stretch of failing to rank the top pitching prospects in the top ten, it is highly likely that people using Baseball America to rate prospects will notice a problem around 2020. If they can't predict the top pitchers, then their rankings will lose significant value.

Free Agents Will Break Your Heart

Everyone knows that prospects will break your heart. Many prospects touted as the next big thing have ultimately failed. But top free agents that are unsuccessful with their new team don’t seem to have the same stigma attached to them. It seems the common perception is that top free agents are more likely to be productive than the top prospects. Certainly, casual fans get excited when they see their team sign a top free agent and usually are willing to trade prospects for top players.

I wanted to see whether this perception is accurate so I looked at all prospects ranked in Baseball America’s Top Hundred list from 1998-2007 and all free agents that earned an annual salary of at least five million, have completed their contract, and whose contract information was either in MLBTR or ESPN’s transaction tracker. These free agents were primarily players that signed a contract starting from 2007 to 2014. A more detailed description of my methodology can be found here.

Position prospects ranked 1-10 produced an annual average of 2.08 fWAR while free agent position players earning $15 million annually produced 2.09 fWAR per year. In addition, 56% of the prospects and 60% of the free agents produced more than 1.5 fWAR per year. This indicates that these position prospects were about as successful as the top free agents.

Free agent position players that earned between ten and fifteen million and those that earned between five and ten million performed similarly. Position players earning between ten to fifteen million produced on average 1.24 fWAR per season while those earning between five to ten million produced 1.13 fWAR. The position players that earned between ten to fifteen million did have slightly higher success rates as 41% were successful and 22% were stars. In contrast, 34% of the position players that earned between five to million were successful and 19% were stars. But in general, there’s little difference between position players that earned ten to fifteen million and those that earned five to ten million. This suggests that the best strategy for teams is to go after top position free agents or to stick with the ones that are affordable.  

Free agent position players that earned between $5 and $15 million annually performed similarly to position prospects ranked 51-100. The prospects produced an annual average of 1.06 fWAR while the free agents produced an average of 1.17 fWAR. 32% of the prospects and 36% of the free agents produced at least 1.5 fWAR per year. They also performed similarly to position prospects ranked between 26 and 50. Position prospects ranked 26-50 produced 1.34 fWAR per year while 35% produced at least 1.5 fWAR per year. It is worth noting that position prospects ranked 26-50 from 2003 to 2007 averaged 1.5 fWAR per year while 46% produced at least 1.5 fWAR per year.

Position prospects ranked 11-25 produced 1.63 fWAR per year while nearly 47% produced at least 1.5 fWAR per year. These prospects are easily better than free agents earning between $5 and $15 million annually but not as good as the top position free agents.

The chart below shows position category results.

Pitching prospects ranked 1-10 weren’t as successful as the top free agent starting pitchers. These prospects produced 1.98 fWAR on average per year while the top free agent starters produced 2.74 fWAR per year. 61.11% of the prospects and 70% of the free agents produced at least 1.5 fWAR per year. However, pitching prospects ranked 1-10 from 2003-2007 performed similarly to the top free agent starting pitchers. These prospects produced 2.36 fWAR on average per year while 81% of these prospects ultimately produced at least 1.5 fWAR per year. Pitching prospects ranked between 1 and 10 from 2003 and 2007 may not have had the same ceiling as the top starters but they arguably had a higher floor. Still, it appears that the top free agent starters are better than the top pitching prospects.

Pitching prospects ranked 11-25 performed similarly to free agents starting pitchers that earned between $10 and $15 million annually. From 1998-2007, the pitching prospects produced an annual average of 1.58 fWAR and 43% produced at least 1.5 fWAR per year. However, from 2003-2007 the prospects produced an annual average of 2.18 fWAR and 58% produced at least 1.5 fWAR per year. Free agent starting pitchers earning between $10 and $15 million annually produced an average of 1.85 fWAR and 53% produced at least 1.5 fWAR per year.

Pitching prospects ranked 26-50 from 1998-2007 produced an annual average of 1.01 fWAR produced an annual average of 1.05 fWAR while 31% produced at least 1.5 fWAR per year. Those ranked between 26 and 50 from 2003 to 2007 produced an annual average of 1.05 fWAR while 35% produced at least 1.5 fWAR. Free agent starting pitchers that earned between $5 and $10 million annually produced an average of 1.04 fWAR while 30% produced at least 1.5 fWAR per year.

Pitching prospects ranked between 51 and 100 from 1998 to 2007 produced an annual average of .73 fWAR while 20% produced at least 1.5 fWAR per year. There was no difference between the 1998 to 2007 and the 2003 to 2007 results.

If one believes that Baseball America is getting better at predicting pitching prospect performance over time then it makes sense to use the 2003-2007 prospect results. In fact, it makes sense to presume that current prospects will be even better than those ranked from 2003-2007. Therefore, it suggests that pitching prospects ranked from 1-10 have similar production to free agent starters earning $15 million or more, pitching prospects ranked from 11-25 have similar production to those free agent starters earning $10 and $15 million and those pitching prospects ranked from 26-50 have similar production to free agent starters earning between $5 and $10 million.

If one believes that Baseball America simply had good luck when selecting top prospects from 2003-2007 then this suggests that pitching prospects ranked from 1-10 are similar but better than those free agents that earn between $10 and $15 million and prospects ranked from 26-50 are similar to free agents that earn between $5 and $10 million.  

The chart below shows pitching category results.

Ultimately, the following groups have similar production:

• Free Agents earning at least $15 million per year = Prospects Ranked 1-10
• Free Agent Starting Pitchers earning between $10 and $15 million annually = Pitching Prospects Ranked from 11-25
• Free Agent Starting Pitchers earning between $5 and $10 million annually = Pitching Prospects Ranked from 26-50
• Free Agent Position Players earning between $5 and $15 annually = Position Prospects Ranked from 51-100

Prospects ranked by Baseball America have become significantly more productive and valuable over the years. Today the best free agents have similar success rates and production to the best prospects. Maybe it is time to start saying that free agents will break your heart.

Team-Controlled Players Have Seen a Pay Cut

I understand that sometimes when I write sabermetric posts that they can be hard to follow. What I'm going to try to do in the future is write a paragraph or two discussing the relevant points at the start of the article.  I'm hoping that doing this will make them easier to follow and make clear what I think are some of the most important points to note.

Abstract: In this post, I intend to show that both the cost of a free agent and team-controlled win (defined as a player with fewer than six years of service time) have increased from 1996 to 2013. The cost of a free agent win is increasing more than 3.9% annually more than a prospect win from 2004 to 2013. This indicates that team-controlled players are being underpaid compared to free agents. It also means that any attempt to determine a discount value for keeping prospects in the minors needs to consider that the value of a prospect win increases over time by 3.9%.

A few months ago, Lew Pollis wrote an article discussing the historical cost of a win in free agency from 1996 to 2013. He determined how much money teams spent on free agents in a given year (regardless of whether they were signed that year or not), used Fangraphs WAR to determine their production, and then divided the two. It is possible to use his method to do the same thing with team-controlled players and see whether their value has changed over that 17-year time frame. This makes it possible to determine whether a team-controlled win has increased in value compared to a free agent win and therefore if prospects are becoming more valuable over time.

The amount of money spent on payrolls has increased significantly from 1996 to 2013. Teams spent $984 million on payroll in 1996 and $3.138 billion on payroll in 2013. Team-controlled players (players with less than six years of service time) earned $357 million in 1996 and $1.323 billion in 2013. Extended players (players with more than six years of service time but did not sign in free agency) earned $420 million in 1996 and just $504 million in 2013 while free agents (players that have more than six years of service time and were signed in free agency) earned $207 million in 1996 and $1.31 billion in 2013. The chart below shows how much money free agents, team-controlled players, and extended players earned from 1996-2013.

The amount of production that each group produces partly explains this trend. Free agents have produced roughly 200 WAR per season from 1996 to 2013. However, extended players produced 324 wins in 1996 but only 110 in 2013 while team-controlled players produced only 478 wins in 1996 but 703 wins in 2013. The chart below shows the number of wins produced by each group annually from 1996 to 2013.

The cost of a win for each group of players has increased over the sample but the cost of a free agent and extended player win is increasing at a considerably larger rate than the cost of a team-controlled player win. The cost of a free agent win has increased from $1.04 million in 1996 to $7.03 million in 2013. The cost of an extended player win increased from $1.3 million in 1996 to $4.55 million in 2013. The cost of a team-controlled player win increased from $750,000 in 1996 to $1.88 million in 2013. The chart below shows how this changed over time.

Not only is the actual amount for a free agent win considerably more than the cost of a team-controlled player win but the annual rate of increase for a free agent win is more than twice that of a team-controlled player win. Over the entire sample, the value of a future team-controlled win increases by 5.6% (1.11/1.05) each year while for the past 10 years it has increased by 3.9% each year. Either way, this indicates that teams would save on player costs by keeping their prospects in the minors for extra time.

It appears that team-controlled players earning under a million dollars and those earning over a million dollars are seeing their cost per win increase at a similar rate. From 2005 to 2013, team-controlled players in each of these categories saw an annual increase of roughly 4%. I used 2005 to 2013 because players earning under $1 million were remarkably ineffective in 2004 while those earning over $1 million were remarkably effective. 2004 appears to be a clear outlier and therefore shouldn’t be used as a baseline.  This chart shows the change over time.

In addition, prospects do not appear to be receiving significantly higher signing bonuses.  According to the Associated Press, teams spent $150 million on the draft in 2004 and $208 million in 2013, meaning that signing bonuses have increased by only 4.3% per year over that period. This is similar to the increase in salaries for team-controlled players during that time period and therefore doesn’t indicate that teams are paying more in signing bonuses while paying less in salaries.

This indicates that both the minimum salary is too low and that team-controlled players don’t receive large enough salary increases in arbitration. In order to keep pace with free agent salaries they should be earning $3.4 million per win instead of $2.2 million per win and therefore their salaries should be roughly 50% higher. This raise in salary should come partially via an increase in the minimum salary and partially via an increase in arbitration.

Over the 17-year period, it is clear that team-controlled players are becoming considerably more valuable compared to free agents. The amount of money that it takes to purchase a win in free agency is growing at a considerably faster rate than the amount it takes to purchase a win with a team-controlled player -- team-controlled players produce nearly 70% of wins while free agents only produce 20%.

This is probably good for the game because it allows for greater parity. As it becomes more and more expensive to get value in the free agent market, then large market teams get less benefit for having a higher payroll. But it also means that prospects and other team-controlled players receive unfair compensation for their efforts.