24 May 2016

Looking Back At Fangraphs' Projections

After a fourth of the season is in the books, the Orioles are 26-16 and have the second highest winning percentage in the majors. Naturally, this vindicates Fangraphs which projected that the Orioles would win the AL East this year. Wait, what’s that you say? Oh yeah, they projected the Orioles to be in last place and not first. On the bright side, at least they didn’t project the Orioles to win 72 or 73 games. I guess the obvious question to ask is what happened.

For starters, it’s worth noting that the Orioles aren’t their largest miss. When comparing their actual winning percentage to their projected winning percentage, Fangraphs has been worse when projecting the Twins (off by 21.9%), Phillies (off by 17.3%), Braves (off by 14.1%) and Astros (off by 15.9%). Meanwhile, Fangraphs projected the Orioles to have a 49.4% winning percentage and the Orioles actually have a 61.9% winning percentage. On average, Fangraphs is off by 6.77% or roughly 11 wins per team over a full season. If someone had projected that each team would go .500, then they’d be off by 7.73% or 12.5 wins per team over a full season. So far, Fangraphs has been more accurate than just presuming that each team would go .500, but not by very much. It is still early in the season though, but at the current moment, well.

As for the Orioles specifically, Fangraphs projected that the Orioles would score 4.64 runs per game and the Orioles have actually scored 4.55. The Orioles are on pace to score 15 runs fewer than Fangraphs projected. This is reasonably close and suggests that Fangraphs overestimated the Orioles offense. These results illustrate that the problem with their projection wasn’t runs scored. Rather, their problem was with runs allowed. Fangraphs projected that the Orioles would allow 4.69 runs per game. The Orioles have actually allowed 3.98 runs per game. The Orioles are on pace to allow 116 runs less than their March projections and 83 runs less than their current projections. If the Orioles can keep that up, then they’ll prove the computers wrong.

The reason for the discrepancy obviously comes on the pitching/defense side of things. But is it all of the pitching? Fangraphs originally projected that the Orioles starting pitching would pitch 935 innings, with a 4.38 ERA and thus give up 455 earned runs. So far, they’re actually doing pretty well with this prediction. The Orioles starting pitching is on pace to throw 914 innings with a 4.44 ERA and to give up 451 earned runs. The Orioles starting rotation would be on pace to allow 461 earned runs if it threw 935 innings. It’s pretty clear that Fangraphs has nailed the Orioles’ starting pitching so far.

The problem is that Fangraphs also projected that the Orioles bullpen would pitch 523 innings, with a 3.76 ERA and give up 218 runs. So far, the bullpen is on pace to pitch 521 innings, but with a 2.67 ERA and thus give up 154 runs. In addition, Fangraphs projected the Orioles to allow 86 unearned runs. So far, the Orioles have allowed 10 and are on pace to allow just 39. Back in March, I wrote an article discussing how Fangraphs projected standings is likely flawed due to how it accounts for unearned runs. It seems like that flaw has come back to bite them.

This flaw has a surprisingly large impact. Using a t-test, Fangraphs' projected runs allowed on the team level is statistically different then the actual results so far (t>.9781). But Fangraphs' projected runs allowed and actual runs allowed by the starting rotation isn't statistically different (t>.1410). This is also the case for the bullpen (t>.2019) suggesting that they're doing a reasonably good job projecting starter and reliever performance. But a t-test comparing the amount of total unearned runs allowed by team to the projected number of unearned runs allowed by team is statistically different at the <.0001 level. Their inability to predict unearned runs significantly weakens the value of their runs allowed projections. Fangraphs' results in this regard are so poor that the Orioles aren't the most egregious case. The Royals, Indians and Rays are each on pace to outperform their projection by over 50 runs. They were projected on average to give up 77 unearned runs and are on pace to allow 21 each. That can't be good.

Using the Wayback Machine, since Fangraphs doesn’t save its original depth charts, it’s possible to review Fangraphs’ projections on March 11th, 2016. So far, a number of crucial pitchers in the bullpen have outperformed Fangraphs’ assumptions. For example, Fangraphs thought that Zach Britton would be good, and projected him to allow 19 earned runs while throwing 65 innings. Britton is on pace to allow 13 runs while throwing 76.66 innings. But Brach is the real lynchpin. Fangraphs projected Brach to give up 21.75 runs over 55 innings. Brach is on pace to give up 12.7 runs over 98.5 innings. That’s a 26 run swing right there, presuming he throws 98 innings in relief. Givens is on pace to give up 13 fewer runs than projected. McFarland, Bundy and Matusz are the only underperforming relievers and they’ve thrown 37 of the bullpens’ 135 innings. After those three, the reliever with the worst results is Darren O’Day with his 2.76 ERA.

Furthermore, the Orioles four elite relievers are throwing 60% of all the innings thrown by the bullpen despite being projected to throw 44% of the innings. Bullpens have better ERAs when their best pitchers throw more innings. That, combined with unexpectedly strong performances from Worley (0 ER in 14 innings as a reliever) and Wilson (1 ER in 8 innings as a reliever) has meant that the Orioles’ bullpen has vastly over performed.

On the defensive side, the Orioles may rank poorly in Fangraphs’ Def stat (20th in MLB), but they only allowed 18 errors in their first 42 games. The Orioles allow .55 unearned runs per error which is worse than league average. But they have the fifth lowest error per game ratio in MLB and is why the Orioles have given up so few unearned runs. On a simplistic level, if the Orioles continue to give up few errors, they’ll allow few unearned runs. On a more complex level, does this mean that the Orioles defense is better than it appears? The Orioles fielding weakness is corner outfield defense, which UZR may not be able to measure properly. Their BABIP is league average, but it would take an analysis to determine the type of batted-ball contact that their pitching has allowed. Further, Inside Edge noted in a blog post that the Orioles rank tenth in defensive giveaways.

The Orioles have outperformed Fangraphs expectations so far due to their excellent bullpen and their ability to avoid unearned runs. They have been slightly lucky, but it makes sense that a team with a strong bullpen will get lucky. Going forward, perhaps this suggests that the Orioles shouldn’t be focusing on adding starting pitching, but perhaps adding a good reliever to ensure the bullpen can continue performing at its current level. It’s pretty clear that the Orioles’ plan is to hope that their offense will maintain its current standard and that their bullpen can continue dominating.

No comments: