What Leveraged ERA Says About The Orioles Bullpen

On April 16th, the Orioles bullpen gave up four runs in three innings. The consequences were drastic – the Orioles Pythagorean expectation dropped from .524 to .504 (as of 5/3/17), costing them .4 of a theoretical win. This game accounted for four of the forty runs allowed by the bullpen this year, inflating their bullpen ERA by .42 runs. And also, anyone who bet on the Orioles to win that day by ten runs or more ended up losing their wager. But the Orioles, who boasted leads of 6-0 and 11-1 ended up winning easily by a score of 11-4.

On a more serious note, this game illustrates the problem with bullpen ERA. The typical way to determine bullpen ERA treats runs allowed in a close game equally to runs allowed in a blowout. But this doesn’t make sense because runs allowed in a close game are far more valuable than those allowed in a blowout and teams make sure that their best pitchers (at least to some extent) pitch in more close games than blowouts.

Consider the following example. For the sake of simplicity, suppose two teams have a bullpen each with only two relievers. For both of these teams, one reliever is the closer who pitches in close games and one is a long man that pitches in blowouts. If one team has a closer and long man that each have a 3 ERA, while the other team has a closer with an ERA of 0 and a long man with an ERA of 6, then both bullpens have the same bullpen ERA. And yet, it’s pretty obvious to everyone that the team with the closer with an ERA of 0 will have a better bullpen than the other club.

At the same time, it’s impossible to just ignore the performance of relievers in blowouts. On April 28th, the Orioles bullpen gave up nine runs in three and a third innings, blowing a game in which they held  leads of 9-1 and 11-4. The inability of Vidal Nuno to get through an inning successfully helped put the Orioles in a save situation which they were unable to convert. Runs allowed in close games are more valuable than those allowed in blowouts, but all runs still have an impact in a game
.
The way to deal with this problem is to use something called leverage index. Leverage index measures which situations are tenser than others and assigns a value to them. The stat is normalized so that on average the leverage is 1.00. In tense situations, the leverage is higher than 1.00 (up to about 10) and in low-tension situations, the leverage is between 0 and 1.0. A metric called gmLI, the leverage index when the pitcher entered the game, can be used to determine which relievers are used in the tensest moments and can quantify the difference in importance between the innings thrown by two different relievers. Once we know about this metric, determining a bullpen’s leveraged ERA is reasonably simple. It’s simply (ERA*Innings Pitched*gmLI)/ (Total Innings Pitched By Bullpen* Bullpen Averaged Leverage).

I was thinking about this when I saw Joe’s article yesterday. Joe noted that the Orioles bullpen “finds itself out of the top 10 in all the statistics in which they had previously been dominant.” With a bullpen ERA of 4.12 so far this season (as of Tuesday), the Orioles’ bullpen hasn’t been impressive. But how does their leveraged bullpen ERA compare with their regular bullpen ERA? Here’s how the calculations look.

The Orioles rank 14th in regular bullpen ERA, but 9th in leveraged bullpen ERA. Their numbers aren’t as good as the elite teams like the Indians, White Sox, Red Sox, Yankees, and Cubs, but they’re definitely in that second tier of clubs. The reason for their improvement can be shown by this second chart.

This chart shows the difference between regular bullpen ERA and leveraged bullpen ERA for all teams in the league. The Orioles, with nearly a one run difference, rank third. The average difference is .46 runs, but for the Orioles its .99 runs. This makes sense because the back of the Orioles bullpen is terrible, but the top relievers have been pretty good.

It’s a similar story when it comes to FIP. The Orioles bullpen FIP is 4.06 and ranks 14th but their leveraged bullpen FIP is 3.49 and good for 12th. The delta between their bullpen FIP and leveraged bullpen FIP is .57 – the tenth highest in the majors and considerably higher than the .35 average.

The reason for the difference between the Orioles leveraged bullpen ERA and their bullpen ERA is because the guys at the end of the bullpen have been poor. Nuno, Drake and Crighton have combined for a 7.27 ERA in 17.33 innings – roughly 20% of the innings that the Orioles bullpen has thrown, but have an average gmLI of about .3. Meanwhile, the heart of the Orioles’ bullpen so far: Britton, Givens, Hart and Brach have an average ERA of 1.58 in 45.66 innings with an average gmLI of 1.78. These four pitchers are performing.

As Joe pointed out, O’Day is definitely a concern with a 5.56 ERA so far this season. He’s been used in crucial situations as his 1.85 gmLI illustrates. His FIP is 3.63, largely due to his avoiding giving up home runs so far this season. If nothing else, O’Day is allowing LHB to hit only .118/.211/.118 this season and hasn’t given up an extra base to anyone yet.

However, Baseball Savant thinks that O’Day has gotten somewhat lucky with pitches into play this season. He’s given up only a .275 wOBA in those situations, but should have allowed a .331 wOBA. This could be bad news for his ERA. Still, either he’ll start pitching better against right handed batters, will be used primarily against left handed batters or will be used in less tense situations. The Orioles have enough good relievers that they can afford to use O’Day as a middle reliever if need be.

For bullpens, not all runs allowed should be considered equal because managers can choose when to use their best pitchers. Therefore, managers can use their worst pitchers in blowouts and their best pitchers in close games. This means that determining bullpen ERA requires a leverage component since a run allowed in a blowout isn’t as damaging as a run allowed in a one run game. This is why it’s a good sign that the Orioles bullpen has a better leveraged bullpen ERA than a normal bullpen ERA. Even if the sixth and seventh reliever in the pen struggles, they can still be sure to use them only in the least crucial situations. That’s why the Orioles’ bullpen looks better than bullpen ERA suggests and why Orioles fans should be cautiously optimistic about the bullpen.

What WPA Teaches About The 2015 Orioles

Just five games ago, it seemed like the Orioles had finally turned the corner and were set to make a run to at least secure a wild-card spot. The Orioles lost each of those next five games and are now at .500. Orioles’ fans are feeling despondent while Manny Machado apparently boiled over in the locker room after the latest Orioles heartbreaker. Why have the Orioles struggled this year?

In 2014, the Orioles scored 705 runs while hitting 211 home runs and are on pace to score 715 runs and hit 212 home runs this year which means that their offense hasn't gotten worse. The pitching hasn’t performed as well as the Orioles allowed 505 runs through August (3.74 R/G) in 2014 and have so far allowed 498 runs this year (4.01 R/G).  Allowing an extra quarter of a run per game shouldn’t be the difference between winning 96 games and flirting with .500.

Looking at the Orioles’ Win Probability Added (WPA) and Clutch statistics from 2012-2015 can help explain why the Orioles have struggled this year. For those unfamiliar, WPA determines the change in the likelihood that a team will win from one plate appearance to the next and credits or debits the player based on how much their action increased their team’s odds of winning. The Clutch statistic measures how well a player has performed in high leverage situations.

The first chart, which measures the Orioles’ offense from 2012-2015, shows that the Orioles have scored a standard amount of runs. The problem is that they haven’t been in the right situations and is why the Orioles are projected to have a negative clutch score for the first time in the sample. In contrast, the Orioles’ offense was very clutch in 2013 and reasonably clutch in 2012 and 2014. This is also why the Orioles’ offensive WPA is on pace to be worth 2 wins while it was worth over 6 wins in 2013 and 2014.

While the Orioles are on pace to score 715 runs in 2015, the Orioles’ wRC suggests that they should only be scoring 687. This is lower than what the Orioles scored from 2012 through 2014 and could indicate that Orioles’ fans feel like their offense isn’t as good as it has been in past years because they've scored more runs than expected. 

The next chart shows that Orioles’ starters have a better ERA in 2015 than they did in 2012 or 2013 but have a considerably worse win-loss record and have a lower WPA. The data also suggests that the Orioles’ starters have been only average in the clutch as opposed to 2014 where they came up big in crucial situations. Unlike in previous years where the offense was able to offset the starting pitching weaknesses, it appears that the Orioles have been unable to recover from their starting pitching performances.

The third chart shows that the Orioles 2015 bullpen has had the best ERA out of the four bullpens when using either standard ERA or leveraged ERA (a metric that I use which grades relievers based on their innings pitched and game leverage to ensure that a closer is given more credit than a middle reliever for bullpen performance). Unsurprisingly, the solid 2015 bullpen has a high 59% win rate (equivalent to a 96-66 team) which is the best of any Orioles club over that period with the sole exception of the extremely clutch 2012 bullpen that had a 74.4% win rate (equivalent to a 121-41 team). Yet while this bullpen has been extremely effective, it hasn’t resulted in producing wins according to WPA compared to those other bullpens.

 

The reason for this can be explained partly by game leverage and partly by comparing saves, holds and blown saves to previous years. In 2015, the Orioles bullpen average game leverage has been 1.05. Their average game leverage ranged from 1.15 to 1.30 from 2012 to 2014. This means that they haven’t been able to pitch in as crucial of situations as they have in the past. This explains why the Orioles’ bullpen is on pace to have only 42 saves and 48 holds in 2015 despite having mid-50 saves from 2012-2014 and between 67 and 97 holds from 2012 to 2014. It would appear that the Orioles bullpen hasn’t had as many opportunities to pitch with a lead as they have previously and consequently saves, holds and blown saves have all decreased.

It should come as no surprise that the Orioles are 17-22 in one run games and 10-15 in two run games because those are the ones most dependent on luck or clutch performance. If the Orioles were to have won as many one and two run games as they’ve won games decided by three or more runs, then they would have another ten wins and would likely be in a fight with the Blue Jays to determine the AL East winner.  Their problem is that they simply haven’t been clutch.

In previous years, the Orioles have been successful by getting a lead to their bullpen and letting their bullpen save the day. In 2013, the Orioles ultimately fell short because the bullpen couldn’t hold onto enough leads and win enough games. This year the bullpen has done well when given a lead (except for this past week) but it just hasn’t happened often enough. The story that the data seems to be telling is that the bullpen has been placed in a situation where they’re trailing by a small amount instead of leading by a small amount and the offense just can’t quite get that hit that would give the bullpen a lead. There’s only so much a bullpen can do in those situations.

The Orioles are still .500 and have over a month to play. Their team is really built to win in September and if the Os do go on a run and take the second wild card spot then no one will care what happened earlier this year. But if the Orioles don’t end up making the playoffs, then people will look at these close games and wonder what could have been with just one more solid offensive player or slightly better pitching from the starting rotation.