On a more serious note, this game illustrates the problem with bullpen ERA. The typical way to determine bullpen ERA treats runs allowed in a close game equally to runs allowed in a blowout. But this doesn’t make sense because runs allowed in a close game are far more valuable than those allowed in a blowout and teams make sure that their best pitchers (at least to some extent) pitch in more close games than blowouts.
Consider the following example. For the sake of simplicity, suppose two teams have a bullpen each with only two relievers. For both of these teams, one reliever is the closer who pitches in close games and one is a long man that pitches in blowouts. If one team has a closer and long man that each have a 3 ERA, while the other team has a closer with an ERA of 0 and a long man with an ERA of 6, then both bullpens have the same bullpen ERA. And yet, it’s pretty obvious to everyone that the team with the closer with an ERA of 0 will have a better bullpen than the other club.
At the same time, it’s impossible to just ignore the performance of relievers in blowouts. On April 28th, the Orioles bullpen gave up nine runs in three and a third innings, blowing a game in which they held leads of 9-1 and 11-4. The inability of Vidal Nuno to get through an inning successfully helped put the Orioles in a save situation which they were unable to convert. Runs allowed in close games are more valuable than those allowed in blowouts, but all runs still have an impact in a game
.
The way to deal with this problem is to use something called leverage index. Leverage index measures which situations are tenser than others and assigns a value to them. The stat is normalized so that on average the leverage is 1.00. In tense situations, the leverage is higher than 1.00 (up to about 10) and in low-tension situations, the leverage is between 0 and 1.0. A metric called gmLI, the leverage index when the pitcher entered the game, can be used to determine which relievers are used in the tensest moments and can quantify the difference in importance between the innings thrown by two different relievers. Once we know about this metric, determining a bullpen’s leveraged ERA is reasonably simple. It’s simply (ERA*Innings Pitched*gmLI)/ (Total Innings Pitched By Bullpen* Bullpen Averaged Leverage).
I was thinking about this when I saw Joe’s article yesterday. Joe noted that the Orioles bullpen “finds itself out of the top 10 in all the statistics in which they had previously been dominant.” With a bullpen ERA of 4.12 so far this season (as of Tuesday), the Orioles’ bullpen hasn’t been impressive. But how does their leveraged bullpen ERA compare with their regular bullpen ERA? Here’s how the calculations look.
The Orioles rank 14th in regular bullpen ERA, but 9th in leveraged bullpen ERA. Their numbers aren’t as good as the elite teams like the Indians, White Sox, Red Sox, Yankees, and Cubs, but they’re definitely in that second tier of clubs. The reason for their improvement can be shown by this second chart.
This chart shows the difference between regular bullpen ERA and leveraged bullpen ERA for all teams in the league. The Orioles, with nearly a one run difference, rank third. The average difference is .46 runs, but for the Orioles its .99 runs. This makes sense because the back of the Orioles bullpen is terrible, but the top relievers have been pretty good.
It’s a similar story when it comes to FIP. The Orioles bullpen FIP is 4.06 and ranks 14th but their leveraged bullpen FIP is 3.49 and good for 12th. The delta between their bullpen FIP and leveraged bullpen FIP is .57 – the tenth highest in the majors and considerably higher than the .35 average.
The reason for the difference between the Orioles leveraged bullpen ERA and their bullpen ERA is because the guys at the end of the bullpen have been poor. Nuno, Drake and Crighton have combined for a 7.27 ERA in 17.33 innings – roughly 20% of the innings that the Orioles bullpen has thrown, but have an average gmLI of about .3. Meanwhile, the heart of the Orioles’ bullpen so far: Britton, Givens, Hart and Brach have an average ERA of 1.58 in 45.66 innings with an average gmLI of 1.78. These four pitchers are performing.
As Joe pointed out, O’Day is definitely a concern with a 5.56 ERA so far this season. He’s been used in crucial situations as his 1.85 gmLI illustrates. His FIP is 3.63, largely due to his avoiding giving up home runs so far this season. If nothing else, O’Day is allowing LHB to hit only .118/.211/.118 this season and hasn’t given up an extra base to anyone yet.
However, Baseball Savant thinks that O’Day has gotten somewhat lucky with pitches into play this season. He’s given up only a .275 wOBA in those situations, but should have allowed a .331 wOBA. This could be bad news for his ERA. Still, either he’ll start pitching better against right handed batters, will be used primarily against left handed batters or will be used in less tense situations. The Orioles have enough good relievers that they can afford to use O’Day as a middle reliever if need be.
For bullpens, not all runs allowed should be considered equal because managers can choose when to use their best pitchers. Therefore, managers can use their worst pitchers in blowouts and their best pitchers in close games. This means that determining bullpen ERA requires a leverage component since a run allowed in a blowout isn’t as damaging as a run allowed in a one run game. This is why it’s a good sign that the Orioles bullpen has a better leveraged bullpen ERA than a normal bullpen ERA. Even if the sixth and seventh reliever in the pen struggles, they can still be sure to use them only in the least crucial situations. That’s why the Orioles’ bullpen looks better than bullpen ERA suggests and why Orioles fans should be cautiously optimistic about the bullpen.
You know that O'Day's numbers are inflated by one really bad outing early on. Buck is continuing to use him in highly leveraged situations. He was even used as the putative closer two days after Brach's implosion (ironically he failed and Brach was perfect that day). We can't expect anyone to be perfect even if we want to with Britton, but O'Day needs to be included with the relievers that are doing well even if his numbers don't support it. Which brings up a whole other question: how do you measure relievers when they pitch one inning and they have 9 straight zeros and one inning where they give up 5 runs? This was pretty much the Jim Johnson disease.
ReplyDeleteO'Day's numbers are terrible against right handed batters so far. He's been unable to strike them out and has a terribly high walk rate. They haven't hit him hard yet, but you've got to figure they'll get an extra base hit soon at this rate.
ReplyDeleteBut it's early after all and the Os are in a situation where they have enough relief pitchers doing well that they can afford to hide O'Day. If he does better, like you think he will, then we won't remember any of this. And in that case, he can take the seventh and eighth innings along with Brach while pushing Givens into a sixth and seventh inning role.
You measure relievers the way you suggested by using WPA (Win Probability Added). Basically, it tracks the changes in win probability based on the games current state and credits or debits pitchers based on whether they make it more likely that their team will win. The Orioles bullpen ranks fifth in this stat.
It would make sense to measure leverage bullpen ERA on a per game basis rather than a per pitcher basis. That way, you're not punishing a team if their closer gives up three runs in a laugher. But that data isn't readily available.
I think what this article shows well is that you have WPA issues in large when comparing starters and relievers. The nature of their usage distorts those numbers, but using it with bullpen utilization can open up more obvious application.
DeleteThat ugly Washington number really backs my feeling that they might be willing to include Robles in a trade package for Britton. On the other hand, recent performance undermines my theory that the bullpen has enough depth to succeed without him.
ReplyDeleteQuite. Relievers need to be analyzed differently than hitters and starting pitchers. WPA perhaps isn't the best way to categorize those differences. For one thing, relievers would have a higher WPA than starters because they have an easier role. For another, even if they were used randomly, they'd still pitch in more tense situations on average than starters.
ReplyDeleteBut starters should be expected (definitely if you use gmLI like I do, to only a large extent if you use a different leverage index like most of the baseball world) to have a similar leverage index to each other. Hence, bullpen chaining and all of that jazz.
The thought of the Orioles and Nationals making a trade seems highly unlikely. Of course, it's not likely that the Orioles will trade Britton while they're still competitive. But for sure someone will be in a position to make a killing trading a few relievers to the Nationals.