24 December 2013

More on Jim Johnson and Elite Closers Status

A couple months back, I wrote about making an argument that Jim Johnson is effectively an elite closer.  This gave rise to a few thoughts from Matt Perez.


Thoughts on Jim Johnson and Elite Closers
by Matt Perez

A couple months ago, Jon wrote a post discussing whether JJ Johnson is an elite closer by looking at JJs conversion rate of save and hold opportunities. While blown holds and saves do get a lot of notice, I was wondering whether they were related to run prevention. Do pitchers with a significantly lower ERA have a higher conversion rate than those pitchers with a significantly higher ERA? 

In order to test this, I looked at individual seasons for all relievers with both a qualifying amount of innings and also ten or more saves plus holds as well as all relievers with eighty or more saves plus holds total from 2003-2013. For each of these relievers, I determined their Save plus Hold conversion rate (in order to save time, I’ll use the abbreviation Sv_Hld_Pct for the rest of this post) as well as their ERA and FIP . Then I determined their correlation coefficient

A correlation coefficient is a statistic that determines how strongly specific metrics resemble each other.  In this case, the metrics are ERA and Sv_Hld_Pct. The possible values range from -1 to 1. The closer the value is to -1 or to 1, the higher the likelihood that the two metrics resemble each other. The definitions for when a correlation is strong, weak or non-existent depends on what one is studying and can be subjective rather than objective. However,  a general rule of thumb is that an r with an absolute value below .4 or above -.4 is weak, between .4-.7 or -.4 to -.7 is moderate and higher than .7 or lower than -.7 is strong. 

Listed below is a table with the results.

Individual Seasons
Metric                    r
With ERA:            - .3657
With FIP               -.2715

Career Numbers
Metric                  r
With ERA            -.5038
With FIP             -.4431

These results indicate that there is only a weak correlation between Sv_Hld_Pct and ERA/FIP for individual seasons and a moderate correlation between Sv_Hld_Pct and ERA/FIP for a full career. It would seem that it wouldn’t make sense to judge a closer based on his Sv_Hld_Pct.

It is possible to split up pitchers into two categories; pitchers that have more saves than holds and pitchers with more holds than saves. Determining the correlation between ERA/FIP and the above categories save or conversion rate will determine whether one category has a significant correlation. Listed below is a table with the results.

Individual Season
Metric                  r for More Saves       r for More Holds
With ERA            -.6059                           -.2644                           
With FIP             -.4798                           -.1780

Career Numbers
Metric                r for More Saves     r for More Holds
With ERA          - .6837                        -.3945
With FIP            -.6306                         -.3229

There are drastically different results for these two categories. ERA and FIP have a moderate to high correlation with save conversion percentage but only a low to moderate correlation with hold conversion percentage. 

There are a few reasons why this could be the case. Closers are typically brought into the game at the start of the ninth. In contrast, a setup man is more likely to be called into a game in the middle of an inning. Suppose a reliever comes into the game in the middle of an inning when there is an offensive player already on base. If that reliever allows that runner to score then he wouldn’t be charged the run but he may blow the lead and thus get a blown save. This is because runs are charged to the pitcher that allows an opponent to reach base and not to the pitcher that allows an opponent to score.
However blown saves are charged to the pitcher that allows a runner to score but not to reach base. If a reliever allows many inherited runners to score then he could in theory have a low Sv_Hld_Rate as well as a low ERA.

However, I think the main reason is how closers and setup men are used. The average closer gets either a save, a hold or a blown save 57% of the time. The average setup man gets a save, hold or a blown save 33% of the time. It would make sense that the average closers’ Sv_Hld_Rate would be more likely to resemble his ERA then it would for the average setup man because the average closer simply comes into the game a lot more frequently in one of those situations. 

In order to properly judge Jim Johnson’s save conversion rate we need to compare him to other closers. I looked at all qualified relievers from 2003-2013 who had at least two complete seasons where they had ten saves and had at least three times as many saves than holds.  Any pitcher that fulfilled these conditions was at minimum the closer of a team for most of the year.  There are a total of 57 closers who fulfilled these conditions. Then, I compared how these relievers did in their first year to how they did in their second year. 

It turns out that the median reliever who fulfilled all of the above conditions converted 89.1% of their save attempts in their first year but only 86.3% in their second year. Likewise, ERA goes from 2.54 in their first year to 3.19 in their second year while FIP goes from 3.01 to 3.45. If we look at relievers who have a 90% conversion rate or higher in their first year, we see a drop in their conversion rate from 91.3% in their first year to 88.8% in their second year.  Their ERA drops from 2.50 in their first year to 3.22 in their second year and their FIP drops from 3.08 in their first year to 3.51 in their second year. These differences are significantly high and suggest we could have expected Jim Johnson to be significantly worse in his second season than in his first season. 

Now that we know this, suppose we look at all qualified relievers from 2003-2013 who had all the same qualifications as above except had three complete seasons instead of two. There are a total of 30 closers who have fulfilled these conditions.

It turns out that the median closer who closes for three seasons has a conversion rate of 90%, an ERA of 2.45 and an FIP of 2.79 in his first year, a conversion rate of 89.4%, an ERA of 2.69 and an FIP of 2.78 in his second year and a conversion rate of 88.1%, an ERA of 2.77 and an FIP of 3.23 in his third year. While these differences aren’t significant, they do indicate that the average closer continues to degrade in performance the longer he closes.

The problem is that it doesn’t matter whether a closer was elite his first year. The question is whether he can repeat his performance in future years. Unfortunately, this indicates that even closers like JJ who were successful in their first year become increasingly likely to fail in future years. Just because a closer has one successful year doesn’t mean he’ll be successful in later years.


Unknown said...

This is very interesting research. Like all good research studies, this provokes many questions:
(1) How would save/hold/blown save-per-game change if you included wins and losses? Set-up men generally appear in tie games. Should relievers get a "hold" if they enter a tie game and keep it tied?
(2) Because closers become closers generally only if they're pitching well, how much of the decline in performance is regression to norm?
(3) How many games and innings are involved?

I am not posing these questions as criticisms of this solid work. I hope that further research can shed more light on these questions.

Matt P said...

Thanks for the kind words.

A reliever can both get a loss and a blown save in a single outing. If I counted both then it would probably be double counting.

If closers do regress to the norm in year two then this indicates that having experience as a closer isn't a good thing. If proven closers were better than other closers we would expect them to get better as they gain experience. This isn't the case.

If this is correct then it would make sense to use your hottest reliever as a closer instead of using one reliever. This isn't necessarily feasible.

I don't know how many games and innings are involved off the top of my head. Are you interested in sums or numbers for each pitcher?

Unknown said...

I don't worship at what I call "the cult of the save." I don't believe that closing games, or more specifically preserving leads of three runs or fewer in the ninth inning, is a distinct skill from pitching effectively at other times.

My question about the innings and games is an attempt to determine if the number of innings is so small that luck can overwhelm the pitcher's skill. My bias is toward that it does, but I don't know.

Matt P said...

Actually, I don't have games and innings saved. I do know that we're talking about roughly 2,300 total Saves, Holds and Blown Saves per year for all 57 closers(so about 40 per year) that fulfill all the conditions in the article. This nearly definitely means they all pitched full seasons.

FIP and conversion rate were significant at the t=.02 level of significance for closers that close for two years.

These are pretty large differences and I would be inclined to say that luck by itself isn't a good answer. But it could be that "proven closers" are given more chances than "unproven closers".