03 December 2013

Death to TINSTAAPP: Updating McKinney's Work on Prospect Success

 Having some formatting problems, will resolve at some point.

As with Scott McKinney so long ago, I have once again been trumped in assessing the rankings of Baseball America.  I simply need to work faster or maybe become content that others will take care of things more quickly than I.  Anyway, one of the site's readers, Matt Perez, has put together an update on the original work by McKinney.  I am hopeful to say this article might be the first to provide data to sink the TINSTAAPP meme.  There are pitching prospects and they are becoming clearer.

---------
Death to TINSTAAPP: Updating McKinney's Work on Prospect Success
by Matt Perez

By popular demand: TINSTAAPP is the well known acronym used to express that There Is No Such Thing As A Pitching Prospect.

Addendum: Here is a link to Perez' follow up to this article. 


Two and a half years ago, Scott McKinney wrote a very popular and rather authoritative study discussing the value of prospects as ranked by Baseball America. Over the years we have seen more and more evidence of teams apparently becoming less amenable to include top prospects in trades.  A couple possibilities exist for why that grip became stronger, but one I would like to explore in this article is whether we are beginning to see a better evaluation process.  In other words, do teams have a better understanding on who will be successful and who won't.  Given that Baseball America develops their rankings from discussions with professional scouts, we can assume that they are a decent proxy for general knowledge in the industry.

First, it should be noted that in updating the information Scott presented that there have been changes. Fangraphs changed its definition of replacement level in March 2013. In Fangraphs’ article discussing the change (http://www.fangraphs.com/blogs/unifying-replacement-level/), they note that the net effect of this change is that players will get a little less WAR per season in their method (and a little more in B-R’s) than they used to. This means that some players who were successes in 2011 will end up being busts today due to these modifications to WAR.  This will result in a little wobbling of the data due to the change in the measuring stick.  However, the impact should be relatively minimal.

For this post, additional data was included so that the time span is increased from 1990 to 2006.

Methodology
I stated above that I used similar methodology to Scott but for convenience I will include it here with all changes.

In order to determine a top prospect, I used Baseball America’s top 100 lists from 1990 to 2006.  Many prospects showed up on multiple lists, but I counted each occurrence of the player because my goal is to measure the performance of rankings as opposed to the players. Each yearly ranking and the subgroups within that yearly ranking represent an evaluation.  In other words, if a player was ranked over four seasons as the 64th, 22nd, 4th, and 18th best player then his career is replicated for each of those rankings.  If he was worth 2 WAR, then in the first year the 64th player would have a 2 WAR value associated to that ranking.  The following year has 2 WAR associated with the 22nd ranking and so on and so on.

WAR was calculated based on the player’s average Wins Above Replacement (WAR) from Fangraphs.com over his cost controlled years. If a player totaled fewer than 100 plate appearances or 25 innings pitched in his first major league season, I omitted it from the calculation. If the player also failed to meet those minimums in his second season, I omitted that season as well. If a player failed to meet those minimums in his third season then I included that season in the calculation.

One of the more difficult tasks of analyzing the data was creating an operational definition of "success" and "failure" or what constituted a prospect "bust." Using the rule of thumb breakdown for WAR, I created the following groupings: 


Avg WAR                   Qualitative Assignment
Didn't Qualify
Didn't Make it
 Bust

Under .50
Very poor


0.50-1.49
Below Average


1.5-2.49
Average

 Success
2.5-3.49
Good


3.5-4.49
Very good
Superior

4.5 or higher
Great




I decided to add a new category for all of the prospects that didn’t have a season in the majors where they either pitched 25 or more innings or totaled over 100 plate appearances. While it’s true that those players can be labeled as “very poor," I believe that there is a valid distinction between players who at least made it to the majors for an extended period of time and those who did not.

A quick note about language: when the term "lower ranked prospects," is used in this post it means  prospects with a rank number that is lower (i.e., #10 is a lower ranked prospect than #20). And "higher ranked prospects" have a rank number which is greater.

Overall Success and Failure Rates for BA Top 100 Prospects 1990-2006
There are very few changes to the overall success and failure rate. The table below shows the success and failure for all top hundred prospects.

Quantitative Breakdown



Didn't Play
Under 0.5
0.5-1.49
1.5-2.49
2.5-3.49
3.5-4.49
4.5+
Pitching
17.00%
40.43%
20.08%
11.91%
6.43%
3.48%
0.67%
Position
12.80%
32.74%
20.46%
13.12%
11.33%
4.09%
5.46%
Total
14.65%
36.12%
20.29%
12.59%
9.18%
3.82%
3.35%


Qualitative Breakdown

Bust
Success
Superior
Pitching
77.51%
22.49%
10.58%
Position
66.00%
34.00%
20.88%
Total
71.06%
28.94%
16.35%

These are similar results to the ones seen in Scott’s study. Three out of every four pitching prospects fail and two out of every three hitting prospects fail. About 70% of all prospects fail. The difference between hitting and pitching prospects has decreased but is still reasonably large.

By parsing the prospect performance by rank, we can discern more about the rankings.

Prospect Success by Decile and Quintile




Bust
Success
Superior
1-20
Total
51.76%
48.24%
30.88%
21-40
Total
71.47%
28.53%
17.06%
41-60
Total
76.47%
23.53%
13.53%
61-80
Total
77.35%
22.65%
11.76%
81-100
Total
78.24%
21.76%
8.53%





1-10
Total
48.24%
51.76%
32.35%
11-20
Total
55.29%
44.71%
29.41%
21-30
Total
71.18%
28.82%
18.24%
31-40
Total
71.76%
28.24%
15.88%
41-50
Total
74.71%
25.29%
11.18%
51-60
Total
78.24%
21.76%
15.88%
61-70
Total
79.41%
20.59%
9.41%
71-80
Total
75.29%
24.71%
14.12%
81-90
Total
81.18%
18.82%
7.65%
91-100
Total
75.29%
24.71%
9.41%

There are some some interesting changes illustrated in these charts in that prospects ranked 1-10 are successful about half of the time while those ranked 11-20 are successful about 45% of the time. Those ranked 21-40 are successful about 28% of the time while those ranked 41-100 are successful about 22% of the time.  The only difference between prospects ranked 41-100 is that those with a better ranking are more likely to have superior performance.

It should be noted that pitching and positional prospects do not seem to perform identically based on their ranking.





Bust
Success
Superior
1-20
Pitching
60.00%
40.00%
23.20%
21-40
Pitching
76.47%
23.53%
9.56%
41-60
Pitching
83.04%
16.96%
8.77%
61-80
Pitching
80.13%
19.87%
7.69%
81-100
Pitching
83.65%
16.35%
6.29%


Bust
Success
Superior
1-10
Pitching
56.14%
43.86%
24.56%
11-20
Pitching
63.24%
36.76%
22.06%
21-30
Pitching
75.68%
24.32%
14.86%
31-40
Pitching
77.42%
22.58%
3.23%
41-50
Pitching
81.25%
18.75%
7.50%
51-60
Pitching
84.62%
15.38%
9.89%
61-70
Pitching
82.28%
17.72%
8.86%
71-80
Pitching
77.92%
22.08%
6.49%
81-90
Pitching
83.53%
16.47%
7.06%
91-100
Pitching
83.78%
16.22%
5.41%


Bust
Success
Superior
1-20
Position
46.98%
53.02%
35.35%
21-40
Position
68.14%
31.86%
22.06%
41-60
Position
69.82%
30.18%
18.34%
61-80
Position
75.00%
25.00%
15.22%
81-100
Position
73.48%
26.52%
10.50%


Bust
Success
Superior
1-10
Position
44.25%
55.75%
36.28%
11-20
Position
50.00%
50.00%
34.31%
21-30
Position
67.71%
32.29%
20.83%
31-40
Position
68.52%
31.48%
23.15%
41-50
Position
68.89%
31.11%
14.44%
51-60
Position
70.89%
29.11%
22.78%
61-70
Position
76.92%
23.08%
9.89%
71-80
Position
73.12%
26.88%
20.43%
81-90
Position
78.82%
21.18%
8.24%
91-100
Position
68.75%
31.25%
12.50%

Looking at pitching, there’s now a difference between prospects ranked between 1-10 and those 11-20. Given that we should expect the prospects from 1-10 to be better than those ranked 11-20 this is a good development. Prospects ranked from 21-40 seem to be showing more success than those ranked 41-100. While about 1 out of 5 pitching prospects ranked from 1-20 have superior success, only 1 out of 7 prospects ranked 21-30 show superior success and about 1 out of 14 prospects ranked 31-100 show superior success. 

For position prospects there have been few changes. Prospects ranked 1-20 are still more likely to be successes than busts. Prospects from 21-60 are about successful at a 30% rate while those ranked 61-100 are successful at a 25% rate. The one new change is that the delineations are becoming clearer with more data.

Prospect Success by Time Period
I split the data into four groups: prospects from 1990-1993, 1994-1997, 1998-2002, and 2003-2006. It’s impossible to split 17 years of data into four even groups so I used one group of five years and three groups of four years. 


Period

Bust
Success
Superior
1990-1993
Total
73.75%
26.25%
13.75%
1994-1997
Total
70.25%
29.75%
15.25%
1998-2002
Total
71.60%
28.40%
16.20%
2003-2006
Total
68.50%
31.50%
20.25%
 
It appears that the percentage of busted prospects is decreasing while the percentage of successful and superior prospects is growing. The data is showing that Baseball America had its worst results from 1990-1993, showed some improvement from 1994-2002, and showed even more improvement from 2003-2006. However this isn't a perfect trend. If we look at the data from these years in more detail, we can see why this is the case.




Unqualified
Below .5
.5-1.49
1.5-2.49
2.5-3.49
3.4-4.49
Above 4.5
1990-1993
Total
16.75%
39.50%
17.50%
12.50%
8.00%
2.25%
3.50%
1994-1997
Total
11.25%
36.50%
22.50%
14.50%
6.75%
3.50%
5.00%
1998-2002
Total
15.20%
37.40%
19.00%
12.20%
10.20%
4.40%
1.60%
2003-2006
Total
15.25%
30.75%
22.50%
11.25%
11.50%
5.00%
3.75%

From 1994-1997 there seems to be an extremely high amount of prospects having at least one season with either 25 innings pitched or 100 PAs. Given that the data in all other time frames is remarkably consistent it means that this was probably a fluke. This probably indicates that a small amount of prospects were hurt during that time frame and thus skewed the results. 

In any event, it does appear that there is a trend of improvement for scouting and/or development of these prospects. Given that, it would be interesting to see whether it’s coming from pitching prospects, position prospects, or both.




Bust
Success
Superior
1990-1993
Pitching
84.52%
15.48%
6.55%
1994-1997
Pitching
77.78%
22.22%
6.43%
1998-2002
Pitching
76.26%
23.74%
12.33%
2003-2006
Pitching
72.49%
27.51%
15.87%
 


Bust
Success
Superior
1990-1993
Position
65.95%
34.05%
18.97%
1994-1997
Position
64.63%
35.37%
21.83%
1998-2002
Position
67.97%
32.03%
19.22%
2003-2006
Position
64.93%
35.07%
24.17%

There has been no apparent change in the success rates of position players from 1990-2006. There possibly has been an increase in the number of superior prospects from 1990-1993 to 2003-2006 but it’s difficult to tell whether that’s an actual trend or merely noise.

When it comes to pitching we see a lot of improvement over the time period. The success rate has nearly doubled from 1990-1993 to 2003-2006. The superior rate more than doubled from 1990-1997 to 1998-2006. Simply put, pitching prospects have become far more valuable than they once were. We can see where the changes have occurred by looking at the data ordered by time period and rank.

By splitting the data into the same four time periods and quintiles we can determine if a particular ranked pitching or position prospect has improved.



Rankings
Years
Bust
Success
Superior
Pitching
1-20
1990-1993
84.38%
15.63%
9.38%
Pitching
1-20
1994-1997
60.00%
40.00%
16.67%
Pitching
1-20
1998-2002
58.97%
41.03%
25.64%
Pitching
1-20
2003-2006
29.17%
70.83%
45.83%
Pitching
21-40
1990-1993
83.87%
16.13%
3.23%
Pitching
21-40
1994-1997
83.33%
16.67%
6.67%
Pitching
21-40
1998-2002
74.42%
25.58%
11.63%
Pitching
21-40
2003-2006
65.63%
34.38%
15.63%
Pitching
41-60
1990-1993
88.57%
11.43%
2.86%
Pitching
41-60
1994-1997
78.79%
21.21%
6.06%
Pitching
41-60
1998-2002
84.62%
15.38%
7.69%
Pitching
41-60
2003-2006
80.39%
19.61%
15.69%
Pitching
61-80
1990-1993
85.29%
14.71%
8.82%
Pitching
61-80
1994-1997
79.49%
20.51%
2.56%
Pitching
61-80
1998-2002
73.33%
26.67%
8.89%
Pitching
61-80
2003-2006
84.21%
15.79%
10.53%
Pitching
81-100
1990-1993
80.56%
19.44%
8.33%
Pitching
81-100
1994-1997
84.62%
15.38%
2.56%
Pitching
81-100
1998-2002
87.50%
12.50%
10.00%
Pitching
81-100
2003-2006
81.82%
18.18%
4.55%



Rankings
Years
Bust
Success
Superior
Position
1-20
1990-1993
43.75%
56.25%
37.50%
Position
1-20
1994-1997
40.00%
60.00%
44.00%
Position
1-20
1998-2002
49.18%
50.82%
31.15%
Position
1-20
2003-2006
53.57%
46.43%
30.36%
Position
21-40
1990-1993
71.43%
28.57%
18.37%
Position
21-40
1994-1997
60.00%
40.00%
26.00%
Position
21-40
1998-2002
71.93%
28.07%
21.05%
Position
21-40
2003-2006
68.75%
31.25%
22.92%
Position
41-60
1990-1993
73.33%
26.67%
11.11%
Position
41-60
1994-1997
72.34%
27.66%
14.89%
Position
41-60
1998-2002
68.75%
31.25%
22.92%
Position
41-60
2003-2006
62.07%
37.93%
27.59%
Position
61-80
1990-1993
67.39%
32.61%
17.39%
Position
61-80
1994-1997
85.37%
14.63%
9.76%
Position
61-80
1998-2002
76.36%
23.64%
12.73%
Position
61-80
2003-2006
71.43%
28.57%
21.43%
Position
81-100
1990-1993
75.00%
25.00%
9.09%
Position
81-100
1994-1997
70.73%
29.27%
9.76%
Position
81-100
1998-2002
75.00%
25.00%
8.33%
Position
81-100
2003-2006
72.22%
27.78%
16.67%

This seems to indicate that there’s been little change in the success rates of position players over the period of 1990-2006 even when split into quintiles.  It also seems to indicate that pitching prospects ranked from 1-20 have shown significant improvement. In 1990-1993 they succeeded only 15% of the time but from 1994-2002 they succeeded 40% of the time and in 2003-2006 they succeeded more than 70% of the time. In addition, pitching prospects ranked 21-40 have also shown improvement. Their success rate has nearly doubled going from just 16% in 1990-1993 to 34% in 2003-2006. The percentage of superior prospects has increased nearly fivefold for pitching prospects ranked from 1-60 from 1990-1993 to 2003-2006. The percentage of superior prospects ranked 1-60 increased roughly 250% from 1990-1993 to 1994-2002. At the very least, it seems clear that pitching prospects in 1990-1993 had lower success rates than we’d expect today.

Conclusions
From 1990-2006 about 70% of Baseball America’s Top Hundred Prospects failed. However, it appears that this failure rate will decrease due to what appears to be an increase in success rates of pitching prospects.  Much of this improvement is due to better scouting performance of those prospects ranked in the top 30.

The performance of the evaluation of position prospects does not appear to be changing, but position prospects still appear to be far safer than pitching prospects.  The success rate for position prospects was more than twice that of pitching prospects from 1990-1993. By 2003-2006 position prospects only succeeded about 25% more often. As we get more data, year by year, we will begin to see how narrow this gap can become.  Mind you, the narrowing of the gap may be the result of many things ranging from better scouting, better development, or simply better medicine.

12 comments:

James Feldman said...

Very interesting, but it seems to me that the assumption that pitcher scouting has improved is still up in the air, because the more obvious explanation of this is that teams have gotten better at protecting the health of pitching prospects/medicine has advanced and made more pitchers likely to come back from injury.

Anonymous said...

How relevant is the steroid era?

Jon Shepherd said...

@James - I think Matt addresses it well in the last sentence in that this is a piece that is about the performance of these rankings as opposed to discerning the reasons behind why performance is changing. We simply do not have the mechanistic data required to answer that questions.

From a conjecture standpoint, I think that teams are better developing their prospects. I think that is why in the past eight years we have seen an explosion in high velocity pitching. It is quite remarkable and a major reason why the run environment has decreased. Maybe medicine has something to do with that, maybe training regimes and workloads have been properly altered to deal with the modern stresses of pitching.

@anon - This will probably result in some blow back, but I think very little if anything has to do with the steroid era. Fangraphs, I believe, published updated aging curves that suggested that players hit their peak now at a much younger age than expected historically. That probably is an artifact of younger players being developed in a better way as opposed to any real performance alteration to the age curve. Hope that makes sense.

Dorasaga said...

It's possible that more prospects turned out as serviceable on the Major League level not because of clubs better at this, but 1. the competitiveness upgraded.

2. Steroid out, veterans no longer stays as long, forcing clubs to use more prospects on the M.L. level.

There has been no change of roster size between 1999 and 2012. With a fixed amount of slot for players to accrue WAR (constant), more veterans needed to fade out. Their slots to play were then opened to prospects.

Jon Shepherd said...

1. Competition may not play a major role. We have not seen similar trends when other expansion took place with teams or races. You would think that would happen to age curves then if it is happening now.

2. I don't think there really is any good evidence suggesting steroids improve the play of older players through health or strength improvement. If so...we would probably have seen this impact at other times, right? This current phenomena seems unique, so I try to find what is so unique about things.

Dorasaga said...

Well, there was a fangraphs article about older players in the Steroid Age (1990-2005, something like that) entering their decline phase LATER. Then as they went into decline, the curve for production was slower than it was and it has been since 2006.

One thing I'm trying to question is whether more veterans stayed in the league during earlier times, and slowed down the turn out rate for prospects, hence inflated the numbers you observed.

Jon Shepherd said...

I don't recall that article exactly, but wasn't that a very minimal shift? Now we are experiencing what appears to be a major shift in the other direction.

Regarding your hypothesis...Don't know. Likely. I imagine if developmental process is churning out high skilled players at an earlier age means that skilled but athletically flailing older players would find themselves out more quickly. You may also have a situation where well paid veterans are more likely to take absolute stances and won't accept being bench or AAA guys.

Dorasaga said...

Thanks. It's interesting what comes first. Your point is valid. It could be an organizational decision throughout the league. I'll need to locate the right data to figure this out.

BTW, I came to this article via ChicagoCubsOnline. I've also been following W.Y.Chen since his days with the Japanese Dragons. Interesting results the O's GM got. Wada didn't work out, and now given a second chance with the Cubs. Chen looks like another Ted Lilly (very similar career numbers at the same age). I once wrote about their career so neat and tight.

Dorasaga said...

Oh, here's the fangraphs article. The shift is significant:

http://www.fangraphs.com/blogs/hitters-no-longer-peak-only-decline/

Jon Shepherd said...

Dorasaga - Oh, I know that recent one. I don't think I took it to mean what you say the data means. You see it kind of with the RC+ data. This current crop really appears to be the one that is acting differently. That all said...the study is more or less a pilot study where we have further questions about other time frames as well as about sample size. You would think we would get some interesting backwelling with letting black people play, establishment of fences, and other things like that. I'd like to see that data before saying whether or not the apparent new curve is the norm or the "steroid" curve is the norm. I find a lot of the problem with reference to that era as the "steroid" era is that it already paints a narrative that too easily explains away a lot of things that it truly does not explain well. We could just as well call it the expansion+ballpark+conditioning+transitionalpitching era.

Matt Perez said...

@James - As shown in my follow-up article, Baseball America was correctly able to predict a much larger amount of the top fifty pitching prospects from 1998-2006 then they were from 1990-1997.

You may be correct that a reason for this is because teams have gotten better at protecting the health of their pitchers. This would make pitcher scouting an easier task because you'd have one less variable to work with.

"With a fixed amount of slot for players to accrue WAR (constant), more veterans needed to fade out. Their slots to play were then opened to prospects."

@Dorasaga: Are you talking about offensive prospects or pitching prospects? There was minimal change in the effects of offensive prospects. I'd expect a larger change if the steroid era was relevant.

Dorasaga said...

Both pitching and hitting shifted. A "minimal change" of 5-15% is significant.