Following Up Death to TINSTAAPP

by Matt Perez
Freelance for Camden Depot

In my previous post, I noted that the value of pitching prospects has improved over the past sixteen years. As a follow up, I thought it might be useful to look into some of the potential reasons for this and determine whether pitching prospects have improved or if Baseball America has gotten better at identifying top prospects. One approach to do this is by comparing rookies ranked by Baseball America to rookies that are not ranked by Baseball America and measure how the performance of each of these groups (ranked vs. unranked) stacks up over time.

This study, like the last, was built upon the methodology of Scott McKinney's work.  The population for this study included all rookies from 1990-2006 that had at least one season with either 100 PAs or 25 innings pitched. Total Wins Above Replacement (WAR) over their cost controlled years was calculated for each player. For these rookies, if in their first year they didn’t have 100 Plate Appearances or 25 Innings Pitched, then that season was omitted. If they did not meet one of those criteria in their second year also, then I omitted both seasons from any calculations. In order to determine the value of a given rookie, the following criteria was used:

Avg WAR Qualitative Assignment

Under .50	Very poor	Bust
0.50-1.49	Below Average
1.5-2.49	Average
2.5-3.49	Good		Success
3.5-4.49	Very good
4.5 or higher	Great	Superior

Each rookie that met the given criteria was split into groups by year (1990-1993, 1994-1997, 1998-2002 and 2003-2006), whether they were position or pitching prospects and whether they were ranked.  

Offensive prospects were the first ones considered.  The following table shows their success rate.

Year	Top 100	<.5	.5-1.49	1.5-2.49	2.5-3.49	3.5-4.49	>4.5	Success
1990-1993	N	197	38	10	3	3	0	6.4%
1990-1993	Y	41	25	19	16	4	6	40.5%
1994-1997	N	138	28	9	3	2	0	7.8%
1994-1997	Y	35	20	15	7	5	6	37.5%
1998-2002	N	167	46	17	4	1	0	9.4%
1998-2002	Y	62	24	19	16	2	4	32.3%
2003-2006	N	145	31	9	12	1	0	11.1%
2003-2006	Y	30	25	18	11	8	6	43.9%

This table shows results similar to those that one would expect. The majority of successful position rookies were ranked by Baseball America.  There was not a single period of time in which there were more successful unranked offensive prospects than ranked offensive prospects even though there are overwhelmingly more unranked players than ranked ones.  This shows an aptitude in Baseball America being able to identify notable prospects as opposed to the process being a crap shoot. Interestingly, the percentage of successful unranked prospects nearly doubled in 1998-2006 from the 1990-2007 numbers. 

One way to determine whether this is significant or not is by running a t-test on the data for offensive prospects that aren’t ranked during each period of time. A t-test is used to determine whether two sets of data are significantly different from each other by calculating a t-value. A t-value is the probability that the sets of data are not statistically different. A low t-value signifies that the data is statistically different while a high t-value signifies that any difference in the data is due to randomness. If one were to find a t-value of .05 than it would mean that there is a 5% chance of the differences between the two sets of data being random variations. This level of confidence is general accepted in most fields. Although you will see importance given to lower thresholds if certain types of data are well founded as being highly variable. For the purpose of this study, the methodology implemented will be more conservative and stick to the generally accepted .05 level to determine if two data sets are significantly different.

The result of the t-test indicates that for offensive prospects that the data in 1990-1993 and 2003-2006 is .10, which, according to our methodology, would not be considered statistically significant.  In other words, Baseball America performed as well with their rankings of offensive prospects for both time periods. However, there does appear to be a trend of improvement between these time periods, so it is worth monitoring in the future to see if it does become more different and, thus, significant.

The number of unranked prospects who are considered to have produced superior results (worth 2.5 WAR or more per season) increase to 13 in 2003-2006 from an average of 5-6 in 1990-2002. A t-test on the data shows that this is a significant change at the 95% level and therefore more superior baseball prospects are being unranked by Baseball America. This suggests that Baseball America is missing several superior baseball prospects in 2003-2006 than they had in the past.

We can look at pitching prospects using the same framework. Here is a table with the results:

Year Group	Top 100	<.5	.5-1.49	1.5-2.49	2.5-3.49	3.5-4.49	>4.5	Success
1990-1993	N	208	50	11	7	1	0	6.9%
1990-1993	Y	43	14	9	3	2	2	21.9%
1994-1997	N	186	39	11	2	1	0	5.9%
1994-1997	Y	45	22	12	3	2	0	20.2%
1998-2002	N	266	41	13	2	2	1	5.5%
1998-2002	Y	48	19	19	13	6	1	36.8%
2003-2006	N	215	50	6	2	0	1	3.3%
2003-2006	Y	41	21	11	10	5	1	30.3%

From 1998-2006 the percentage of successful ranked pitching prospects is nearly twice as high as it was from 1990-1997. T-tests on both the successful and superior ranked pitching prospects show that there is a significant difference between the 1990-1997 data and the 1998-2006 data at the 99% confidence level. Baseball America is finding a larger number of strong pitching prospects from 1998-2007 than they did in 1990-1997. Part of the reason is because the number of strong pitching prospects has increased.  From 1990-1997 there were only 66 total successful pitching prospects and 23 superior prospects and from 1998-2006 there were 93 successful pitching prospects and 43 superior. 

Unlike with position prospects, the percentage of successful unranked pitching prospects has fallen from 6% to 3%. A t-test shows that the difference between the 1990-1993 data and 2003-2006 data is significant at the 90% significance level. This possibly suggests that despite the increase of successful pitching prospects the amount of unranked successful pitching prospects is decreasing.

While a larger percentage of ranked position prospects from 1998-2006 were success than ranked pitching prospects, the difference between the two has decreased. This is due to the improvement in pitching prospects. 

In order to determine whether Baseball America has gotten better or worse at predicting which prospects will be the best rookies, we can evaluate the top fifty rookies in a given period and see whether they were ranked. If Baseball America is able to rank a larger percentage of the top rookies than this would indicate that their rankings are more accurate and therefore more meaningful. The following table measures offensive players.

Year	Top 100	Top Fifty Players
1990-1993	N	12
1990-1993	Y	38
1994-1997	N	17
1994-1997	Y	33
1998-2002	N	17
1998-2002	Y	33
2003-2006	N	14
2003-2006	Y	36

These results indicate that a similar amount of top fifty rookies were correctly predicted by Baseball America from 1990-2006 with the best period coming in 1990-1993.

Here is the table for pitching prospects.

Year Group	Top 100	Top Fifty Players
1990-1993	N	31
1990-1993	Y	19
1994-1997	N	23
1994-1997	Y	27
1998-2002	N	15
1998-2002	Y	35
2003-2006	N	19
2003-2006	Y	31

These results show that Baseball America better identified top fifty pitching players who were rookies from 1998-2006 than they did from 1990-1997. Indeed, in 1990-1993 they identified less than half of the top fifty pitchers. One of the reasons why the value of a Baseball America top pitching prospect was limited is because they were unable to identify the top pitchers. Now that Baseball America is more accurate, we can see the actual value of a pitching prospect.

In addition, the value of a top fifty pitcher has increased from 1998-2006 from the 1990-1997 numbers.  This table shows the total value as well as the average value for pitching prospects.

Year Group	Top Fifty Total Value	Top Fifty Average Value
1990-1993	614.3	2.05
1994-1997	558.5	1.86
1998-2002	796.3	2.65
2003-2006	698.0	2.33

The difference between pitchers who were rookies in 1998-2006 from those who were rookies in 1990-1997 is significant at the 99% confidence level signified that their value increased from 1998-2006. That stated, these rookies are still less valuable than offensive rookies.  The values for offensive rookies can be found here.

Year	Top Fifty Total Value	Top Fifty Average Value
1990-1993	943.4	3.14
1994-1997	845.1	2.82
1998-2002	846.7	2.82
2003-2006	963.7	3.21

In conclusion, both the value of a pitching rookie has increased over time and Baseball America has improved its methods to better determine which pitching prospects will become the best rookies and young stars. While these factors do not make pitching prospects as valuable as hitting prospects, it does lessen the gap and indicate that a top pitching prospect is far more valuable than once thought.

It also seems that Baseball America was more accurate determining the best offensive prospects in 1990-2002 then it was from 2003-2006. Certainly, Baseball America has predicted fewer superior prospects than they once did. While the change appears to be minor at the moment, it will be interesting to see if this trend continues in the future.