Your Best Reliever Sometimes Should Not Close Games

On April 3, Buck Showalter sat down a tired Mike Wright Jr. at the end of the fifth inning.  His night ended at 82 pitches, which was nine more than his last game in Spring Training.  It felt like an early hook and that maybe he could stretch to a couple batters more to provide some cover for a bullpen that has been used heavily the previous couple days.

At the conclusion of the fifth, Wright had covered 20 batters and was about to enter the heart of the lineup for the third time.  In his former incarnations as a starting pitcher, the third time through was typically rough.  Well, all times through were rough, but it got worse that third time.  In his last season as a starter (2016), opponents would hit him in the low 800 OPS range, but that would jump to over 1000 OPS that third time through.  He walked a lot of batters, struck out few, and was hit hard.  The idea since then has been a transition over to a sinker/slider pitcher would help him get deeper into games by changing looks and keeping the ball on the ground.

That was not happening that night, Wright has trouble placing his sinker with it often hitting the glove a foot or more out of the zone.  This led him to rely more on his four seam fastball and slider/curve offerings.  The expectation was that this pairing got him through the order those first two times and he was bound to suffer that third time out there even though his pitch count left some room for another batter or two.  It was an ideal time for him to go.

Added to this, the Orioles had regained the lead and were up 4-3 going into the bottom of the sixth inning.  So traditionally, the idea would be to send out Wright again and see what happens.  As noted above, we should know better than that.  Slightly less traditionally, the idea would be that the sixth inning falls outside of the seventh-eighth-ninth holy reliever Save time and the proper reliever to send in would be your fourth righty or second lefty non-closer into the breach to stop the opposition.  However, we know better than that now.  We actually have known better for a while.

In 2002, Tom Tango (who is now MLBAM's "data architect") came up with the concept of leverage index.  You see, we have a whole lot of information about baseball over the years.  We know what has happened in certain scenarios and whether a team won or loss beyond that scenario.  By looking at all of those situations, you can develop historical-based probabilities on whether a team wins or not.  By knowing the inning, outs, baserunners and locations, and score differential, you get an average probability of whether a club may win or lose.

By taking that data and looking at how the probability can change, we end up at what is called a Leverage Index.  In general, these conditions are easy to see.  When a club is up seven runs in the ninth, the leverage index would be low because it does not really matter what happens with a single batter.  The differential is too great.  In a tighter game where the difference is a run, you can see how much more important that event is.

I won't go deeply into how it is all calculated, but I will provide a link to Leverage Index in certain scenarios and note the following:

Very High Leverage ... greater than 3.0
High Leverage ... 1.5 to 3.0
Medium Leverage ... 0.8 to 1.5
Low Leverage ... less than 0.8

The Orioles were facing a 1.6 Leverage Index in the bottom of the sixth inning, which would indicate that the club has entered a period where it is important to fight for the win as a single plate appearance could result in a rather poor outcome.  A modern bullpen when called upon would see this as an important moment to put in a rather solid reliever.  It is a role that Buck Showalter has often assigned to Mychal Givens.

On that night, it did not work out.  Givens was taken deep and the Orioles found themselves in a hole.  Buck, having a short bullpen, then tabbed Pedro Araujo in what would be considered a low leverage position (0.6).  Things quickly went south and the Orioles hung on with one run down and the bases loaded, two outs, in the bottom of the seventh.

At this point, there is a decision to be made.  The bullpen has one rested reliever left, Nestor Cortes, Jr.  The Leverage Index at this point is 2.4, which means the outcome of this event will likely have considerable impact on the game moving forward.  Do you go with the rested arm or do you send Darren O'Day out there for the second night in a row?  More so, do you have the foresight of this predicament to get O'Day up in the bullpen when Araujo immediately looks bad as he pitches to his first batter?  Do you get O'Day up while knowing that maybe Araujo gets lucky and settles down?  Do you try to sneak into the eighth or ninth inning where maybe the Leverage Index is higher for you to utilize O'Day?

It is a tough question to answer.  Yes, putting Givens in seems pretty easy and straightforward.  Putting in Araujo also seems like a no-brainer.  From there it got messy.

But let us go back to general bullpen usage.  Below I took a few excerpts out of the Leverage Index values link to earlier in the article.  What we see are generally preferrable times to bring in a very good reliever (italics) and your best reliever (bold) based on being the visitor and starting off the bottom of the inning.

Visitor
Bases Empty	No Outs
Bottom of	Up 2	Up 1	Tie	Down 1	Down 2
1st	0.9	0.9	0.9	0.8	0.6
2nd	1	1	0.9	0.8	0.6
3rd	1	1.1	1	0.8	0.6
4th	1.1	1.2	1.1	0.9	0.6
5th	1.3	1.3	1.2	0.9	0.6
6th	1.4	1.6	1.3	0.9	0.6
7th	1.6	1.9	1.5	0.8	0.4
8th	1.8	2.5	1.8	0.6	0.3
9th	2	3.6	2.3

With this initial setup, you can see how being up a run can make situations more important than they might seem and suggest a different usage.  Strong middle relievers should be considered in the sixth inning, like Givens.  Likewise, closers should be considered for entry into a close game in the ninth, which is something that Buck Showalter will live with on his non-use of Zach Britton in the Orioles only 2016 playoff game.

It is important to remember that these are initial conditions.  As baserunners appear and out accumulate, these numbers can go up or down.  For instance, the bottom of the third with bases loaded and down a run or two is considered a very high leverage scenario (3.1).  Ideally, you would put a closer in that situation, but the way bullpens are constructed it would be difficult to justify killing the bullpen for a game that is still pretty young.  So the number is not all, but it does provide a decent understanding of important scenarios and how maybe it is important to use high end relievers early or use your best pitcher in non-save scenarios.

What Leveraged ERA Says About The Orioles Bullpen

On April 16th, the Orioles bullpen gave up four runs in three innings. The consequences were drastic – the Orioles Pythagorean expectation dropped from .524 to .504 (as of 5/3/17), costing them .4 of a theoretical win. This game accounted for four of the forty runs allowed by the bullpen this year, inflating their bullpen ERA by .42 runs. And also, anyone who bet on the Orioles to win that day by ten runs or more ended up losing their wager. But the Orioles, who boasted leads of 6-0 and 11-1 ended up winning easily by a score of 11-4.

On a more serious note, this game illustrates the problem with bullpen ERA. The typical way to determine bullpen ERA treats runs allowed in a close game equally to runs allowed in a blowout. But this doesn’t make sense because runs allowed in a close game are far more valuable than those allowed in a blowout and teams make sure that their best pitchers (at least to some extent) pitch in more close games than blowouts.

Consider the following example. For the sake of simplicity, suppose two teams have a bullpen each with only two relievers. For both of these teams, one reliever is the closer who pitches in close games and one is a long man that pitches in blowouts. If one team has a closer and long man that each have a 3 ERA, while the other team has a closer with an ERA of 0 and a long man with an ERA of 6, then both bullpens have the same bullpen ERA. And yet, it’s pretty obvious to everyone that the team with the closer with an ERA of 0 will have a better bullpen than the other club.

At the same time, it’s impossible to just ignore the performance of relievers in blowouts. On April 28th, the Orioles bullpen gave up nine runs in three and a third innings, blowing a game in which they held  leads of 9-1 and 11-4. The inability of Vidal Nuno to get through an inning successfully helped put the Orioles in a save situation which they were unable to convert. Runs allowed in close games are more valuable than those allowed in blowouts, but all runs still have an impact in a game
.
The way to deal with this problem is to use something called leverage index. Leverage index measures which situations are tenser than others and assigns a value to them. The stat is normalized so that on average the leverage is 1.00. In tense situations, the leverage is higher than 1.00 (up to about 10) and in low-tension situations, the leverage is between 0 and 1.0. A metric called gmLI, the leverage index when the pitcher entered the game, can be used to determine which relievers are used in the tensest moments and can quantify the difference in importance between the innings thrown by two different relievers. Once we know about this metric, determining a bullpen’s leveraged ERA is reasonably simple. It’s simply (ERA*Innings Pitched*gmLI)/ (Total Innings Pitched By Bullpen* Bullpen Averaged Leverage).

I was thinking about this when I saw Joe’s article yesterday. Joe noted that the Orioles bullpen “finds itself out of the top 10 in all the statistics in which they had previously been dominant.” With a bullpen ERA of 4.12 so far this season (as of Tuesday), the Orioles’ bullpen hasn’t been impressive. But how does their leveraged bullpen ERA compare with their regular bullpen ERA? Here’s how the calculations look.

The Orioles rank 14th in regular bullpen ERA, but 9th in leveraged bullpen ERA. Their numbers aren’t as good as the elite teams like the Indians, White Sox, Red Sox, Yankees, and Cubs, but they’re definitely in that second tier of clubs. The reason for their improvement can be shown by this second chart.

This chart shows the difference between regular bullpen ERA and leveraged bullpen ERA for all teams in the league. The Orioles, with nearly a one run difference, rank third. The average difference is .46 runs, but for the Orioles its .99 runs. This makes sense because the back of the Orioles bullpen is terrible, but the top relievers have been pretty good.

It’s a similar story when it comes to FIP. The Orioles bullpen FIP is 4.06 and ranks 14th but their leveraged bullpen FIP is 3.49 and good for 12th. The delta between their bullpen FIP and leveraged bullpen FIP is .57 – the tenth highest in the majors and considerably higher than the .35 average.

The reason for the difference between the Orioles leveraged bullpen ERA and their bullpen ERA is because the guys at the end of the bullpen have been poor. Nuno, Drake and Crighton have combined for a 7.27 ERA in 17.33 innings – roughly 20% of the innings that the Orioles bullpen has thrown, but have an average gmLI of about .3. Meanwhile, the heart of the Orioles’ bullpen so far: Britton, Givens, Hart and Brach have an average ERA of 1.58 in 45.66 innings with an average gmLI of 1.78. These four pitchers are performing.

As Joe pointed out, O’Day is definitely a concern with a 5.56 ERA so far this season. He’s been used in crucial situations as his 1.85 gmLI illustrates. His FIP is 3.63, largely due to his avoiding giving up home runs so far this season. If nothing else, O’Day is allowing LHB to hit only .118/.211/.118 this season and hasn’t given up an extra base to anyone yet.

However, Baseball Savant thinks that O’Day has gotten somewhat lucky with pitches into play this season. He’s given up only a .275 wOBA in those situations, but should have allowed a .331 wOBA. This could be bad news for his ERA. Still, either he’ll start pitching better against right handed batters, will be used primarily against left handed batters or will be used in less tense situations. The Orioles have enough good relievers that they can afford to use O’Day as a middle reliever if need be.

For bullpens, not all runs allowed should be considered equal because managers can choose when to use their best pitchers. Therefore, managers can use their worst pitchers in blowouts and their best pitchers in close games. This means that determining bullpen ERA requires a leverage component since a run allowed in a blowout isn’t as damaging as a run allowed in a one run game. This is why it’s a good sign that the Orioles bullpen has a better leveraged bullpen ERA than a normal bullpen ERA. Even if the sixth and seventh reliever in the pen struggles, they can still be sure to use them only in the least crucial situations. That’s why the Orioles’ bullpen looks better than bullpen ERA suggests and why Orioles fans should be cautiously optimistic about the bullpen.

Do the Orioles Have a Bullpen Problem Brewing?

The bullpen has, since 2012, been a constant for the Orioles. From 2012-2016, it ranked second in fWAR, third in ERA, tenth in FIP, and first in wins, and fifth in innings pitched. Those are obviously stellar numbers, and the consistency of the pen is, without question, one of the biggest reasons that the Orioles posted the most wins in the American League during that time period.

This season, however, the bullpen has been much more pedestrian. While it once again leads baseball in victories with 8, the pen finds itself out of the top 10 in all the statistics in which they had previously been dominant. To be fair, much of this decline is directly related to last weekend's series against the Yankees. In 16 innings, the bullpen gave up 14 earned runs and potentially gave away two crucial early season victories. As a result, Vidal Nuno and Logan Verrett were sent down, and the O's once again turned to Mike Wright as a fresh arm.

Some of this can surely be attributed to Zach Britton missing two weeks with an arm injury, but Britton himself looked pretty shaky in April. After an historic 2016, he has struggled out of the gate, posting an unsightly 1.857 WHIP and striking out two fewer batters per 9 innings than he did last season during the month of April. Brad Brach filled in admirably as closer, but even he had a meltdown on Friday night in a game in which the Orioles blew leads of 9-1 and 11-4. Britton came off the disabled list and pitched in a non-save situation against the Red Sox last night and did not look good again, giving up a walk and a hit (as well as getting the most bizarre triple play you'll ever see).

Now, it's not like the pen has been bad, but there are potentially a few warning signs. The bullpen is collectively walking over 4 hitters per 9, nearly a full walk per 9 more than 2012-2016. Of greater concern, though, has to be the two year decline of Darren O'Day. After 4 brilliant seasons that earned him a $31 million deal, O'Day has dealt with injuries and struggled to a 4.25 ERA, 4.31 FIP, and 1.323 WHIP in his last 42 innings pitched.

Certainly, the Orioles have other good options, though Mychal Givens and Donnie Hart also struggled last night against the Red Sox, but the bullpen without a dominant O'Day has less room for error than in previous seasons. The encouraging signs for the O's are that the pen is still elite at limiting hard contact and stranding baserunners, which at this point seems to be an innate skill for all Orioles pitchers, and the fact that they are very stingy with the home run ball. It seems likely that Britton will return to being an elite pitcher (assuming he is healthy) and that the recent implosions will not become the norm. Hopefully.

Appreciating Buck Showalter's Bullpen Usage

In this piece praising the Orioles' and Cardinals' bullpens, David Schoenfield looked at some of the differences between how O's manager Buck Showalter and Cardinals' manager Mike Matheny handle their respective relief corps:

Orioles manager Buck Showalter is a little unique in his handling of his pen. They lead the majors in relief appearances of more than one inning with 106. The Cardinals have just 58 such appearances, although both Britton and Rosenthal have gone more than three outs six times. What Showalter prefers to avoid is using a reliever on two consecutive days, let alone three. Tommy Hunter, now with the Cubs, is the only Orioles reliever to appear in three straight days this year, which he did once. Showalter has used a reliever on consecutive days just 44 times; Cardinals manager Mike Matheny has used a reliever on three or more consecutive days 26 times and two or more 84 times, leading to some concern that he'll burn out the pen by October.

Showalter is frequently complimented as being an excellent bullpen handler, and the above demonstrates some of the reasons why. He's fine with relievers working a little longer, but he doesn't want them to get overworked by pitching too often on consecutive days.

Using Baseball-Reference.com's invaluable Play Index, we can extend the search terms a bit. Since the 2011 season (Showalter took over in 2010, but you can't start the search in the latter part of that season), the Orioles are second lowest in using relievers in consecutive games. They're also second lowest in ERA in the times they do opt to use relievers on zero days' rest. In terms of using relievers after one day of rest, they're in the middle (18th) during that span (and again are second lowest in ERA). After two days of rest? The O's are 11th (and fifth in ERA). Talented relievers deserve plenty of credit, but Showalter's handling is also a big part of the success.

The O's again boast one of the best bullpens in the majors. They rank fourth in ERA (2.68), t-fifth in FIP (3.29), fourth in K/9 (9.05), 18th in BB/9 (3.27), sixth in HR/9 (0.71), sixth in GB% (47.6%), and fifth in WPA (3.61).

Why Fans Should Be Excited About the 2014 Orioles Bullpen

Last week, Matt Kremnitzer wrote an article about the Orioles bullpen in which he compared the 2012 bullpen to the 2014 bullpen. In this article he included a chart comparing certain statistics of the 2012 and 2014 bullpen and noted the results have been reasonably similar even if the 2012 bullpen has been better. 

I was thinking about this article and how it measured bullpen performance. Bullpen performance is typically measured by taking the results for each reliever and weighting them by the number of innings pitched by that reliever. Two relievers that each threw sixty innings would have the same amount of weight when determining reliever performance. A reliever that throws sixty innings would have twice the weight as one that threw thirty innings.

The problem with this method is that it presumes that all relievers in the bullpen should be treated equally. I'm not sure I would do that because it’s only logical that the performance of a closer is more important than the performance of a long man. The closer usually comes into games in the ninth inning when the game is close while the long man comes into the game considerably earlier when the game isn’t close. If the long man gives up a run then little harm is done but if the closer allows a run then it could lead to a blown save and a loss. Teams would rather have a closer with a 0 ERA than a long man with a 0 ERA and it only makes sense to reflect that in bullpen performance. All relievers aren’t equal and it’s necessary to find some method of ensuring that the most important relievers have the most value in these calculations.

Tom Tango developed a statistic called leverage index (LI). This statistic measures how important a particular situation is in a baseball game depending on the inning, score, outs, and number of players on base. It is possible to use this statistic to determine which relievers have been used in the most important situations and therefore which relievers are most valuable. This allows us to quantify the difference between a closer and a long man. The leverage index most commonly used is one that measures a player’s average LI for all game events known as pLI. In fact, Fangraphs uses this leverage index when determining pitching WAR. Another leverage index is called gmLI. This index measures a pitchers average LI when he enters the game. I personally believe that gmLI is a better metric than pLI and will therefore use gmLI in this article. 

Fangraphs provides the gmLI for each reliever. It is possible to use this statistic to weight bullpen performance by the importance of each pitcher as well as by innings thrown which should result in a more accurate measure of bullpen performance than considering each pitcher to be of equal performance regardless of role. We can use this statistic to see how the Orioles 2014 bullpen compares to other teams bullpens. In addition, I’m only using relievers on an active roster because Orioles’ fans are excited about their current relievers. Orioles’ and other teams’ fans couldn’t care less about guys like Evan Meek.  

This table shows the number of pitchers used by each bullpen, the bullpen’s ERA without using leverage, the bullpen’s ERA using leverage and the difference between the two.

Team	Pitchers	Non-Leveraged ERA	Leverage ERA	Difference
Rays	10	3.12	3.14	-0.02
Mets	11	3.01	3.02	-0.01
Mariners	11	2.49	2.49	0.00
Tigers	18	4.28	4.24	0.04
Marlins	14	3.07	3.02	0.05
Giants	10	2.65	2.60	0.05
Twins	11	3.33	3.25	0.07
Blue Jays	16	3.75	3.66	0.09
Orioles	12	3.07	2.99	0.09
Rockies	15	4.84	4.74	0.10
Athletics	12	2.72	2.62	0.10
Indians	14	2.84	2.72	0.12
Astros	16	4.85	4.73	0.12
Nationals	12	2.81	2.69	0.13
Yankees	16	3.33	3.19	0.13
Braves	14	3.19	3.06	0.14
White Sox	16	4.28	4.14	0.14
Diamondbacks	15	3.75	3.60	0.15
Padres	12	2.39	2.22	0.18
Rangers	23	3.96	3.76	0.20
Cardinals	14	3.68	3.48	0.20
Cubs	15	3.45	3.24	0.21
Brewers	15	3.56	3.31	0.24
Angels	20	2.96	2.70	0.26
Pirates	14	3.40	3.10	0.29
Reds	12	3.96	3.58	0.38
Red Sox	14	3.27	2.82	0.45
Dodgers	13	3.75	3.27	0.48
Phillies	14	3.67	3.19	0.48
Royals	15	3.18	2.38	0.79

Using this method shows the strength of the Royals bullpen. The Royals have only three relievers that have pitched over ten innings with a gmLI over 1.1. All three of those relievers have an ERA under 2. The backend of the Royals bullpen has been dominant and as a result their bullpen ERA using leverage is considerably lower than their bullpen ERA not considering leverage. Likewise, the Phillies have had Papelbon and Adams pitch most of their high leverage innings and both of them have low ERAs. This method makes them look better than they would otherwise.

This does little to explain why people should excited about the Orioles bullpen. While our leveraged ERA is 2.99 this is only good for tenth in the majors. The difference between our leveraged bullpen ERA and non-leveraged bullpen ERA is miniscule and one of the lowest in the majors. In addition, the 2012 bullpen is still better then the 2014 bullpen even when considering leverage.

Fortunately, this next table may answer this question. This table consists of the team name and the average leverage of the bullpen and shows that the Orioles bullpen has the second highest game leverage.

Team	Average Leverage
Rangers	0.997
Blue Jays	1.010
Dodgers	1.019
Padres	1.023
Giants	1.050
Nationals	1.071
Twins	1.077
Astros	1.087
Royals	1.090
Athletics	1.106
Brewers	1.115
Tigers	1.117
Indians	1.120
Rockies	1.121
Cubs	1.156
Mets	1.165
Diamondbacks	1.181
Pirates	1.184
Mariners	1.191
Phillies	1.193
Angels	1.194
Reds	1.224
Marlins	1.229
Rays	1.231
Cardinals	1.252
White Sox	1.270
Red Sox	1.286
Braves	1.312
Orioles	1.340
Yankees	1.387

Orioles fans are excited about their bullpen because it has done well in high pressure situations. As I write this post, O'Day has just struck out Abreu and Garcia with runners at first and second to maintain a one run lead while Britton had a one-two-three ninth to pick up his 27th save despite having little margin for error with just a one run lead. The Orioles have needed their bullpen to come through in the clutch this season and it has met the challenge. As a result, the Orioles are in excellent position to clinch the division and fans are showing their bullpen the love.