Death to TINSTAAPP: Updating McKinney's Work on Prospect Success

 Having some formatting problems, will resolve at some point.

As with Scott McKinney so long ago, I have once again been trumped in assessing the rankings of Baseball America.  I simply need to work faster or maybe become content that others will take care of things more quickly than I.  Anyway, one of the site's readers, Matt Perez, has put together an update on the original work by McKinney.  I am hopeful to say this article might be the first to provide data to sink the TINSTAAPP meme.  There are pitching prospects and they are becoming clearer.

---------
Death to TINSTAAPP: Updating McKinney's Work on Prospect Success
by Matt Perez

By popular demand: TINSTAAPP is the well known acronym used to express that There Is No Such Thing As A Pitching Prospect.

Addendum: Here is a link to Perez' follow up to this article. 

Two and a half years ago, Scott McKinney wrote a very popular and rather authoritative study discussing the value of prospects as ranked by Baseball America. Over the years we have seen more and more evidence of teams apparently becoming less amenable to include top prospects in trades.  A couple possibilities exist for why that grip became stronger, but one I would like to explore in this article is whether we are beginning to see a better evaluation process.  In other words, do teams have a better understanding on who will be successful and who won't.  Given that Baseball America develops their rankings from discussions with professional scouts, we can assume that they are a decent proxy for general knowledge in the industry.

First, it should be noted that in updating the information Scott presented that there have been changes. Fangraphs changed its definition of replacement level in March 2013. In Fangraphs’ article discussing the change (http://www.fangraphs.com/blogs/unifying-replacement-level/), they note that the net effect of this change is that players will get a little less WAR per season in their method (and a little more in B-R’s) than they used to. This means that some players who were successes in 2011 will end up being busts today due to these modifications to WAR.  This will result in a little wobbling of the data due to the change in the measuring stick.  However, the impact should be relatively minimal.

For this post, additional data was included so that the time span is increased from 1990 to 2006.

Methodology

I stated above that I used similar methodology to Scott but for convenience I will include it here with all changes.

In order to determine a top prospect, I used Baseball America’s top 100 lists from 1990 to 2006.  Many prospects showed up on multiple lists, but I counted each occurrence of the player because my goal is to measure the performance of rankings as opposed to the players. Each yearly ranking and the subgroups within that yearly ranking represent an evaluation.  In other words, if a player was ranked over four seasons as the 64th, 22nd, 4th, and 18th best player then his career is replicated for each of those rankings.  If he was worth 2 WAR, then in the first year the 64th player would have a 2 WAR value associated to that ranking.  The following year has 2 WAR associated with the 22nd ranking and so on and so on.

WAR was calculated based on the player’s average Wins Above Replacement (WAR) from Fangraphs.com over his cost controlled years. If a player totaled fewer than 100 plate appearances or 25 innings pitched in his first major league season, I omitted it from the calculation. If the player also failed to meet those minimums in his second season, I omitted that season as well. If a player failed to meet those minimums in his third season then I included that season in the calculation.

One of the more difficult tasks of analyzing the data was creating an operational definition of "success" and "failure" or what constituted a prospect "bust." Using the rule of thumb breakdown for WAR, I created the following groupings: 

Avg WAR                   Qualitative Assignment

Didn't Qualify	Didn't Make it	Bust
Under .50	Very poor
0.50-1.49	Below Average
1.5-2.49	Average		Success
2.5-3.49	Good
3.5-4.49	Very good	Superior
4.5 or higher	Great

I decided to add a new category for all of the prospects that didn’t have a season in the majors where they either pitched 25 or more innings or totaled over 100 plate appearances. While it’s true that those players can be labeled as “very poor," I believe that there is a valid distinction between players who at least made it to the majors for an extended period of time and those who did not.

A quick note about language: when the term "lower ranked prospects," is used in this post it means  prospects with a rank number that is lower (i.e., #10 is a lower ranked prospect than #20). And "higher ranked prospects" have a rank number which is greater.

Overall Success and Failure Rates for BA Top 100 Prospects 1990-2006

There are very few changes to the overall success and failure rate. The table below shows the success and failure for all top hundred prospects.

Quantitative Breakdown

	Didn't Play	Under 0.5	0.5-1.49	1.5-2.49	2.5-3.49	3.5-4.49	4.5+
Pitching	17.00%	40.43%	20.08%	11.91%	6.43%	3.48%	0.67%
Position	12.80%	32.74%	20.46%	13.12%	11.33%	4.09%	5.46%
Total	14.65%	36.12%	20.29%	12.59%	9.18%	3.82%	3.35%

Qualitative Breakdown

	Bust	Success	Superior
Pitching	77.51%	22.49%	10.58%
Position	66.00%	34.00%	20.88%
Total	71.06%	28.94%	16.35%

These are similar results to the ones seen in Scott’s study. Three out of every four pitching prospects fail and two out of every three hitting prospects fail. About 70% of all prospects fail. The difference between hitting and pitching prospects has decreased but is still reasonably large.

By parsing the prospect performance by rank, we can discern more about the rankings.

Prospect Success by Decile and Quintile

		Bust	Success	Superior
1-20	Total	51.76%	48.24%	30.88%
21-40	Total	71.47%	28.53%	17.06%
41-60	Total	76.47%	23.53%	13.53%
61-80	Total	77.35%	22.65%	11.76%
81-100	Total	78.24%	21.76%	8.53%
1-10	Total	48.24%	51.76%	32.35%
11-20	Total	55.29%	44.71%	29.41%
21-30	Total	71.18%	28.82%	18.24%
31-40	Total	71.76%	28.24%	15.88%
41-50	Total	74.71%	25.29%	11.18%
51-60	Total	78.24%	21.76%	15.88%
61-70	Total	79.41%	20.59%	9.41%
71-80	Total	75.29%	24.71%	14.12%
81-90	Total	81.18%	18.82%	7.65%
91-100	Total	75.29%	24.71%	9.41%

There are some some interesting changes illustrated in these charts in that prospects ranked 1-10 are successful about half of the time while those ranked 11-20 are successful about 45% of the time. Those ranked 21-40 are successful about 28% of the time while those ranked 41-100 are successful about 22% of the time.  The only difference between prospects ranked 41-100 is that those with a better ranking are more likely to have superior performance.

It should be noted that pitching and positional prospects do not seem to perform identically based on their ranking.

		Bust	Success	Superior
1-20	Pitching	60.00%	40.00%	23.20%
21-40	Pitching	76.47%	23.53%	9.56%
41-60	Pitching	83.04%	16.96%	8.77%
61-80	Pitching	80.13%	19.87%	7.69%
81-100	Pitching	83.65%	16.35%	6.29%
		Bust	Success	Superior
1-10	Pitching	56.14%	43.86%	24.56%
11-20	Pitching	63.24%	36.76%	22.06%
21-30	Pitching	75.68%	24.32%	14.86%
31-40	Pitching	77.42%	22.58%	3.23%
41-50	Pitching	81.25%	18.75%	7.50%
51-60	Pitching	84.62%	15.38%	9.89%
61-70	Pitching	82.28%	17.72%	8.86%
71-80	Pitching	77.92%	22.08%	6.49%
81-90	Pitching	83.53%	16.47%	7.06%
91-100	Pitching	83.78%	16.22%	5.41%
		Bust	Success	Superior
1-20	Position	46.98%	53.02%	35.35%
21-40	Position	68.14%	31.86%	22.06%
41-60	Position	69.82%	30.18%	18.34%
61-80	Position	75.00%	25.00%	15.22%
81-100	Position	73.48%	26.52%	10.50%
		Bust	Success	Superior
1-10	Position	44.25%	55.75%	36.28%
11-20	Position	50.00%	50.00%	34.31%
21-30	Position	67.71%	32.29%	20.83%
31-40	Position	68.52%	31.48%	23.15%
41-50	Position	68.89%	31.11%	14.44%
51-60	Position	70.89%	29.11%	22.78%
61-70	Position	76.92%	23.08%	9.89%
71-80	Position	73.12%	26.88%	20.43%
81-90	Position	78.82%	21.18%	8.24%
91-100	Position	68.75%	31.25%	12.50%

Looking at pitching, there’s now a difference between prospects ranked between 1-10 and those 11-20. Given that we should expect the prospects from 1-10 to be better than those ranked 11-20 this is a good development. Prospects ranked from 21-40 seem to be showing more success than those ranked 41-100. While about 1 out of 5 pitching prospects ranked from 1-20 have superior success, only 1 out of 7 prospects ranked 21-30 show superior success and about 1 out of 14 prospects ranked 31-100 show superior success. 

For position prospects there have been few changes. Prospects ranked 1-20 are still more likely to be successes than busts. Prospects from 21-60 are about successful at a 30% rate while those ranked 61-100 are successful at a 25% rate. The one new change is that the delineations are becoming clearer with more data.

Prospect Success by Time Period

I split the data into four groups: prospects from 1990-1993, 1994-1997, 1998-2002, and 2003-2006. It’s impossible to split 17 years of data into four even groups so I used one group of five years and three groups of four years. 

Period		Bust	Success	Superior
1990-1993	Total	73.75%	26.25%	13.75%
1994-1997	Total	70.25%	29.75%	15.25%
1998-2002	Total	71.60%	28.40%	16.20%
2003-2006	Total	68.50%	31.50%	20.25%

 

It appears that the percentage of busted prospects is decreasing while the percentage of successful and superior prospects is growing. The data is showing that Baseball America had its worst results from 1990-1993, showed some improvement from 1994-2002, and showed even more improvement from 2003-2006. However this isn't a perfect trend. If we look at the data from these years in more detail, we can see why this is the case.

		Unqualified	Below .5	.5-1.49	1.5-2.49	2.5-3.49	3.4-4.49	Above 4.5
1990-1993	Total	16.75%	39.50%	17.50%	12.50%	8.00%	2.25%	3.50%
1994-1997	Total	11.25%	36.50%	22.50%	14.50%	6.75%	3.50%	5.00%
1998-2002	Total	15.20%	37.40%	19.00%	12.20%	10.20%	4.40%	1.60%
2003-2006	Total	15.25%	30.75%	22.50%	11.25%	11.50%	5.00%	3.75%

From 1994-1997 there seems to be an extremely high amount of prospects having at least one season with either 25 innings pitched or 100 PAs. Given that the data in all other time frames is remarkably consistent it means that this was probably a fluke. This probably indicates that a small amount of prospects were hurt during that time frame and thus skewed the results. 

In any event, it does appear that there is a trend of improvement for scouting and/or development of these prospects. Given that, it would be interesting to see whether it’s coming from pitching prospects, position prospects, or both.

		Bust	Success	Superior
1990-1993	Pitching	84.52%	15.48%	6.55%
1994-1997	Pitching	77.78%	22.22%	6.43%
1998-2002	Pitching	76.26%	23.74%	12.33%
2003-2006	Pitching	72.49%	27.51%	15.87%

 

		Bust	Success	Superior
1990-1993	Position	65.95%	34.05%	18.97%
1994-1997	Position	64.63%	35.37%	21.83%
1998-2002	Position	67.97%	32.03%	19.22%
2003-2006	Position	64.93%	35.07%	24.17%

There has been no apparent change in the success rates of position players from 1990-2006. There possibly has been an increase in the number of superior prospects from 1990-1993 to 2003-2006 but it’s difficult to tell whether that’s an actual trend or merely noise.

When it comes to pitching we see a lot of improvement over the time period. The success rate has nearly doubled from 1990-1993 to 2003-2006. The superior rate more than doubled from 1990-1997 to 1998-2006. Simply put, pitching prospects have become far more valuable than they once were. We can see where the changes have occurred by looking at the data ordered by time period and rank.

By splitting the data into the same four time periods and quintiles we can determine if a particular ranked pitching or position prospect has improved.

	Rankings	Years	Bust	Success	Superior
Pitching	1-20	1990-1993	84.38%	15.63%	9.38%
Pitching	1-20	1994-1997	60.00%	40.00%	16.67%
Pitching	1-20	1998-2002	58.97%	41.03%	25.64%
Pitching	1-20	2003-2006	29.17%	70.83%	45.83%
Pitching	21-40	1990-1993	83.87%	16.13%	3.23%
Pitching	21-40	1994-1997	83.33%	16.67%	6.67%
Pitching	21-40	1998-2002	74.42%	25.58%	11.63%
Pitching	21-40	2003-2006	65.63%	34.38%	15.63%
Pitching	41-60	1990-1993	88.57%	11.43%	2.86%
Pitching	41-60	1994-1997	78.79%	21.21%	6.06%
Pitching	41-60	1998-2002	84.62%	15.38%	7.69%
Pitching	41-60	2003-2006	80.39%	19.61%	15.69%
Pitching	61-80	1990-1993	85.29%	14.71%	8.82%
Pitching	61-80	1994-1997	79.49%	20.51%	2.56%
Pitching	61-80	1998-2002	73.33%	26.67%	8.89%
Pitching	61-80	2003-2006	84.21%	15.79%	10.53%
Pitching	81-100	1990-1993	80.56%	19.44%	8.33%
Pitching	81-100	1994-1997	84.62%	15.38%	2.56%
Pitching	81-100	1998-2002	87.50%	12.50%	10.00%
Pitching	81-100	2003-2006	81.82%	18.18%	4.55%

	Rankings	Years	Bust	Success	Superior
Position	1-20	1990-1993	43.75%	56.25%	37.50%
Position	1-20	1994-1997	40.00%	60.00%	44.00%
Position	1-20	1998-2002	49.18%	50.82%	31.15%
Position	1-20	2003-2006	53.57%	46.43%	30.36%
Position	21-40	1990-1993	71.43%	28.57%	18.37%
Position	21-40	1994-1997	60.00%	40.00%	26.00%
Position	21-40	1998-2002	71.93%	28.07%	21.05%
Position	21-40	2003-2006	68.75%	31.25%	22.92%
Position	41-60	1990-1993	73.33%	26.67%	11.11%
Position	41-60	1994-1997	72.34%	27.66%	14.89%
Position	41-60	1998-2002	68.75%	31.25%	22.92%
Position	41-60	2003-2006	62.07%	37.93%	27.59%
Position	61-80	1990-1993	67.39%	32.61%	17.39%
Position	61-80	1994-1997	85.37%	14.63%	9.76%
Position	61-80	1998-2002	76.36%	23.64%	12.73%
Position	61-80	2003-2006	71.43%	28.57%	21.43%
Position	81-100	1990-1993	75.00%	25.00%	9.09%
Position	81-100	1994-1997	70.73%	29.27%	9.76%
Position	81-100	1998-2002	75.00%	25.00%	8.33%
Position	81-100	2003-2006	72.22%	27.78%	16.67%

This seems to indicate that there’s been little change in the success rates of position players over the period of 1990-2006 even when split into quintiles.  It also seems to indicate that pitching prospects ranked from 1-20 have shown significant improvement. In 1990-1993 they succeeded only 15% of the time but from 1994-2002 they succeeded 40% of the time and in 2003-2006 they succeeded more than 70% of the time. In addition, pitching prospects ranked 21-40 have also shown improvement. Their success rate has nearly doubled going from just 16% in 1990-1993 to 34% in 2003-2006. The percentage of superior prospects has increased nearly fivefold for pitching prospects ranked from 1-60 from 1990-1993 to 2003-2006. The percentage of superior prospects ranked 1-60 increased roughly 250% from 1990-1993 to 1994-2002. At the very least, it seems clear that pitching prospects in 1990-1993 had lower success rates than we’d expect today.

Conclusions

From 1990-2006 about 70% of Baseball America’s Top Hundred Prospects failed. However, it appears that this failure rate will decrease due to what appears to be an increase in success rates of pitching prospects.  Much of this improvement is due to better scouting performance of those prospects ranked in the top 30.

The performance of the evaluation of position prospects does not appear to be changing, but position prospects still appear to be far safer than pitching prospects.  The success rate for position prospects was more than twice that of pitching prospects from 1990-1993. By 2003-2006 position prospects only succeeded about 25% more often. As we get more data, year by year, we will begin to see how narrow this gap can become.  Mind you, the narrowing of the gap may be the result of many things ranging from better scouting, better development, or simply better medicine.