Orioles in First, Which is What Our System Projected

Alleged security camera recording of an alleged drunken,
shirtless alleged Steve alleged Melewski allegedly
destroying alleged computer equipment at an
alleged Best Buy on allegedly York Road, allegedly.

Back in March, mouths were foaming about how the projection models rated the Baltimore Orioles.  Steve Melewski was alleged going to local Best Buys, drinking a six pack in the parking lot, and then going inside to jeer at salespeople in the tablet section.  It was a dark time in Baltimore, but a rather proliferous one for message boards and writers who needed inspiration for fan friendly columns.

Truth be told, the projection models were not exactly in unison.  While Baseball Prospectus' PECOTA had the Orioles at 72 wins, ZiPS and FanGraphs pegged the club at 81 and 82 wins, respectively.  Some took those rosier projection harshly as well without acknowledging that the first place clubs were sitting with 84 wins in those projections.  In other words, two model approaches saw the AL East largely flattened while PECOTA was off on its own doing its own thing, which is what more recent iterations of PECOTA has been known to do as it has been more aggressively updated in the past few years.

At the Depot, I usually put together projections, but time was rushed and I simply was more focused on my CRAP draft system than updating my BUCK patch model to the ZiPS system.  In my gut, the club felt like a fourth place team around 82 wins in a pretty competitive AL East.  Other clubs looked better like Toronto and Boston, but the difference was not all that great.

In the first few weeks of the season, the Orioles won and won often, but typically not all that pretty.  The rotation was a mess, but the offense was humming and the club got some good fortune.  It meant that the data analyst cautious approach continued to note that the Orioles looked uneven and ready to fail.  Keith Law noted that without the winning streak at the beginning of the year, the club had played .500 ball.  Yes, we all know that removing a winning streak makes a team look worse and that there is typically no good reason to ignore a week and a half of games.  Anyway, an allegedly drunken Steve Melewski allegedly ran wild in several more local Best Buys.

In June, I decided to finally go back and figure out what my Buck patch model would have projected in the AL East.  I updated the system to consider the past three seasons each manager managed and each General Manager general managed.  I looked at what ZiPS thought they would do and then what they wound up doing in order to get an idea of the impact of those positions on the success of a baseball club.  All of this is based on my previous work, which I still have trouble believing.  For those without much experience, I simply projected them to not deviate from ZiPS in those seasons.

AL East	adjCD	RankCD	Rank
Baltimore	85	1	1
New York	83	2	4
Toronto	82	3	2
Boston	79	4	3
Tampa Bay	77	5	5
AL Central
Royals	92	1	2
Cleveland	88	2	1
Chicago	79	4	4
Detroit	77	5	3
Twins	80	3	5
AL West
Houston	88	1	2
Texas	84	2	1
LAAA	81	3	5
Mariners	80	4	3
Oakland	76	5	4

The patch model elevated the Orioles over the opposition as it considers the work by Showalter and Duquette to be exceptional with respect to expectations.  Over 2/3 of the projected ranks are off no more than one from where the clubs currently sit.  The biggest misses are the Twins who have been simply gruesome in part due to a heavy reliance on Minor League data for their projections.  The other big miss is the Rangers who have been blowing up other clubs this year even though they are a bit saddled with an aging and sleep deprived Prince Fielder.

All in all, it seems the data analysis approach has worked remarkably well for most of us doing this.

Looking Back At Fangraphs' Projections

After a fourth of the season is in the books, the Orioles are 26-16 and have the second highest winning percentage in the majors. Naturally, this vindicates Fangraphs which projected that the Orioles would win the AL East this year. Wait, what’s that you say? Oh yeah, they projected the Orioles to be in last place and not first. On the bright side, at least they didn’t project the Orioles to win 72 or 73 games. I guess the obvious question to ask is what happened.

For starters, it’s worth noting that the Orioles aren’t their largest miss. When comparing their actual winning percentage to their projected winning percentage, Fangraphs has been worse when projecting the Twins (off by 21.9%), Phillies (off by 17.3%), Braves (off by 14.1%) and Astros (off by 15.9%). Meanwhile, Fangraphs projected the Orioles to have a 49.4% winning percentage and the Orioles actually have a 61.9% winning percentage. On average, Fangraphs is off by 6.77% or roughly 11 wins per team over a full season. If someone had projected that each team would go .500, then they’d be off by 7.73% or 12.5 wins per team over a full season. So far, Fangraphs has been more accurate than just presuming that each team would go .500, but not by very much. It is still early in the season though, but at the current moment, well.

As for the Orioles specifically, Fangraphs projected that the Orioles would score 4.64 runs per game and the Orioles have actually scored 4.55. The Orioles are on pace to score 15 runs fewer than Fangraphs projected. This is reasonably close and suggests that Fangraphs overestimated the Orioles offense. These results illustrate that the problem with their projection wasn’t runs scored. Rather, their problem was with runs allowed. Fangraphs projected that the Orioles would allow 4.69 runs per game. The Orioles have actually allowed 3.98 runs per game. The Orioles are on pace to allow 116 runs less than their March projections and 83 runs less than their current projections. If the Orioles can keep that up, then they’ll prove the computers wrong.

The reason for the discrepancy obviously comes on the pitching/defense side of things. But is it all of the pitching? Fangraphs originally projected that the Orioles starting pitching would pitch 935 innings, with a 4.38 ERA and thus give up 455 earned runs. So far, they’re actually doing pretty well with this prediction. The Orioles starting pitching is on pace to throw 914 innings with a 4.44 ERA and to give up 451 earned runs. The Orioles starting rotation would be on pace to allow 461 earned runs if it threw 935 innings. It’s pretty clear that Fangraphs has nailed the Orioles’ starting pitching so far.

The problem is that Fangraphs also projected that the Orioles bullpen would pitch 523 innings, with a 3.76 ERA and give up 218 runs. So far, the bullpen is on pace to pitch 521 innings, but with a 2.67 ERA and thus give up 154 runs. In addition, Fangraphs projected the Orioles to allow 86 unearned runs. So far, the Orioles have allowed 10 and are on pace to allow just 39. Back in March, I wrote an article discussing how Fangraphs projected standings is likely flawed due to how it accounts for unearned runs. It seems like that flaw has come back to bite them.

This flaw has a surprisingly large impact. Using a t-test, Fangraphs' projected runs allowed on the team level is statistically different then the actual results so far (t>.9781). But Fangraphs' projected runs allowed and actual runs allowed by the starting rotation isn't statistically different (t>.1410). This is also the case for the bullpen (t>.2019) suggesting that they're doing a reasonably good job projecting starter and reliever performance. But a t-test comparing the amount of total unearned runs allowed by team to the projected number of unearned runs allowed by team is statistically different at the <.0001 level. Their inability to predict unearned runs significantly weakens the value of their runs allowed projections. Fangraphs' results in this regard are so poor that the Orioles aren't the most egregious case. The Royals, Indians and Rays are each on pace to outperform their projection by over 50 runs. They were projected on average to give up 77 unearned runs and are on pace to allow 21 each. That can't be good.

Using the Wayback Machine, since Fangraphs doesn’t save its original depth charts, it’s possible to review Fangraphs’ projections on March 11th, 2016. So far, a number of crucial pitchers in the bullpen have outperformed Fangraphs’ assumptions. For example, Fangraphs thought that Zach Britton would be good, and projected him to allow 19 earned runs while throwing 65 innings. Britton is on pace to allow 13 runs while throwing 76.66 innings. But Brach is the real lynchpin. Fangraphs projected Brach to give up 21.75 runs over 55 innings. Brach is on pace to give up 12.7 runs over 98.5 innings. That’s a 26 run swing right there, presuming he throws 98 innings in relief. Givens is on pace to give up 13 fewer runs than projected. McFarland, Bundy and Matusz are the only underperforming relievers and they’ve thrown 37 of the bullpens’ 135 innings. After those three, the reliever with the worst results is Darren O’Day with his 2.76 ERA.

Furthermore, the Orioles four elite relievers are throwing 60% of all the innings thrown by the bullpen despite being projected to throw 44% of the innings. Bullpens have better ERAs when their best pitchers throw more innings. That, combined with unexpectedly strong performances from Worley (0 ER in 14 innings as a reliever) and Wilson (1 ER in 8 innings as a reliever) has meant that the Orioles’ bullpen has vastly over performed.

On the defensive side, the Orioles may rank poorly in Fangraphs’ Def stat (20th in MLB), but they only allowed 18 errors in their first 42 games. The Orioles allow .55 unearned runs per error which is worse than league average. But they have the fifth lowest error per game ratio in MLB and is why the Orioles have given up so few unearned runs. On a simplistic level, if the Orioles continue to give up few errors, they’ll allow few unearned runs. On a more complex level, does this mean that the Orioles defense is better than it appears? The Orioles fielding weakness is corner outfield defense, which UZR may not be able to measure properly. Their BABIP is league average, but it would take an analysis to determine the type of batted-ball contact that their pitching has allowed. Further, Inside Edge noted in a blog post that the Orioles rank tenth in defensive giveaways.

The Orioles have outperformed Fangraphs expectations so far due to their excellent bullpen and their ability to avoid unearned runs. They have been slightly lucky, but it makes sense that a team with a strong bullpen will get lucky. Going forward, perhaps this suggests that the Orioles shouldn’t be focusing on adding starting pitching, but perhaps adding a good reliever to ensure the bullpen can continue performing at its current level. It’s pretty clear that the Orioles’ plan is to hope that their offense will maintain its current standard and that their bullpen can continue dominating.

Two Problems With Fangraphs' Projected Standings

A few months ago, Fangraphs published its projected win totals for 2016. They weren’t very complimentary of the Orioles as they projected them to win just 80 games and be the worst team in the AL East. The Royals, the 2015 World Series Champions, are projected to win only 77 games this season despite minimal changes. I thought it would be worth looking into why Fangraphs thought that the Orioles might struggle.

Fangraphs projects the Orioles to score 4.64 runs per game (752 total) while allowing 4.69 (760 total). If the Orioles did score 752 runs, then that would be the most they’ve scored since 2008. It's clear that the projections predict that the Orioles’ pitching is their weak point.

Fangraphs WAR depth charts tell a similar story.  They project the Orioles’ to have the eighth highest offensive WAR totals, but tied for the sixth lowest pitching WAR totals. However, what’s interesting about the Fangraphs WAR depth charts is that they project a total of 1085.8 WAR. Given that there can only be 1000 WAR in a season, they have an error somewhere. Neil Weinberg suggested “the reason for this is because no one has gotten hurt yet, so the depth charts are oversampling PA/IP from good players”. If this is the case, then Fangraphs is being overly optimistic projecting health from established players. The Orioles are depending on a number of star players and have limited quality depth available, so they are more vulnerable to injuries than other clubs. Such a flaw would overrate the Orioles’ ability.

Fangraphs doesn’t think much of the Orioles starting pitching. It is ranked third worst in WAR and are projected to have the second lowest ERA. They also project the Orioles’ starters to throw the second lowest amount of innings in the majors, even fewer than teams like the Reds and Phillies. The Orioles rotation performed poorly in 2015, and lost Wei-Yin Chen in free agency so such an event would be unfortunate but also plausible.

Fangraphs is higher on the Orioles’ bullpen. It is ranked sixth in WAR, and 20th in ERA. The Orioles’ rotation isn’t expected to throw many innings and therefore the bullpen will have to take over the slack. As a result, the Orioles top relievers of Britton, O’Day, Brach and Givens are projected to throw only 230 of the 523 innings thrown by the bullpen. The more innings thrown by long relievers and non-elite arms, the more runs that the Orioles’ bullpen will allow to opposing teams. It’s very possible for the Orioles to have elite relievers in the bullpen and still have a poor bullpen ERA.

It makes sense to presume that the Orioles pitching will struggle due to the weakness of the Orioles rotation, but not that they would be the second worst in the majors especially given their decent bullpen. I decided to create a table showing the total runs allowed by each teams’ pitching, and the amount of earned runs allowed by starters and relievers. In addition, by subtracting total runs from earned runs, I was able to derive the total amount of unearned runs allowed by each team.

I found that the Orioles rotation was projected to allow roughly 455 earned runs or the fourth most in the majors. Their bullpen was projected to allow 218 earned runs or the seventh most in the majors and all told the Orioles were expected to allow 674 earned runs good for the fifth most in the majors. It wouldn’t be particularly surprising if the Orioles allowed that many earned runs. The Orioles allowed 642 in 2012, 678 in 2013, 558 in 2014 and 646 in 2015 so allowing 674 in 2016 would be within reason even if I’d expect 650. The chart looks like this.

The reason why the Orioles do so poorly is that they are projected to allow 86 unearned runs or tie for worst in the majors with the Blue Jays. This is surprising as the Orioles allowed 31 in 2013, 35 in 2014 and 47 in 2015. 86 unearned runs is more then the Orioles allowed in 2014 and 2015 combined. No team has allowed 86 unearned runs or more in the past three seasons and these projections project that each team will allow an average of 75 unearned runs this year versus roughly 50 from 2013-2015.

The numbers look even more bizarre when looking at the five teams projected to allow the most unearned runs in 2016. The Royals are actually tied for third with 84 despite having an elite defense and the other two clubs are the Diamondbacks and Reds. All five of these clubs are projected to have top eight defenses in 2016 as measured by Fangraphs Field Metric and would probably be expected to have some of the lowest unearned run totals in the majors.

Neil did argue that fielding isn’t equal to unearned runs. This is completely true, but the field metric does have a correlation with unearned runs. The correlation between the fielding metric and unearned runs was -.436 in 2013, -.491 in 2014, -.489 in 2015. This is the expected result as it suggests there's a moderate correlation between having good fielders and limiting unearned runs. One shouldn't expect a strong correlation because factors such as sequencing impact the amount of unearned runs a team allows and is independent of defense. Likewise, teams with good range will have good fielding scores but may have many errors.

For 2016, the projected correlation is +.487. This suggests that there’s the same amount of certainty between 2013-2015 and 2016, but Fangraphs is projecting that good fielding teams will allow more unearned runs. This is counter to historical data and basic common sense. Given the relationship between the field metric and unearned runs, it would seem likely that there is a faulty addition or subtraction operation in their projection model.

Using historical data and 2016 projected defense, I built two models that project how many unearned runs teams should be expected to allow in 2016. I found that teams like the Orioles and Royals should be expected to give up forty fewer unearned runs than currently projected while teams like the Pirates and Padres should probably allow roughly 60 unearned runs. If so, when taking into account the lower run environment, teams like the Royals should be expected to win an extra two games, while the Padres should be another two games worse. Two wins may appear to have only a minor impact, but the difference between the third-best team and third-worst team in the majors is only 17 wins and another six wins would change the Royals from being in the cellar to being in first place. Small changes have a larger impact then one may think.

The Fangraphs model has 1080 WAR instead of 1000 and projects each team will allow 75 unearned runs rather than only 50. They may want to consider looking into these problems and fixing them.