Why You Can't Just Look at WAR to Determine a Player's Ability

The other day, I got into an argument about Rick Porcello. One person made that argument that if you believe in fWAR, Porcello has been good. He’s been worth 8.4 fWAR over the past 3.5 years or about roughly 2.4 fWAR per year, primarily due to a strong FIP and the ability to pitch a lot of innings. If one win costs $7.5 million then paying $20 million per year is a slight but not huge overpay. Writers at Fangraphs have also argued that Porcello is underrated, that he’s developed nicely into a 3-win player, that moving to Boston will make him better, that he deserved a huge payday, and that paying $20 million per year is reasonable. Paul Swydan, an author for Fangraphs, wrote an article in the Boston Globe suggesting that Porcello is the 13th-best pitcher in baseball.

On the other hand, I made the argument that Porcello is a slightly better version of Bud Norris. Let me explain why I made that argument and why just looking at WAR to decide pitchers' value isn’t always the best idea.

This first table compares Norris and Porcello’s performances from 2012-2015.

Porcello has a number of advantages. He’s been healthier so therefore he’s thrown more innings, but he also throws more innings per start. His win-loss record is slightly above .500 while Norris’s was slightly below .500 and they have roughly the same ERA. The main difference is that Porcello has a FIP that’s 0.4 runs lower than Norris and that’s why he has a considerably higher fWAR than Norris but a similar RA9_WAR.

This second table compares Norris and Porcello’s performance from 2012-2015 with the bases empty, runners on base, and runners in scoring position.

Porcello does a good job pitching with no one on base. He has a decent strikeout rate and more importantly an excellent walk rate. He gives up a standard home run rate, but it ends up resulting in fewer home runs than average due to a low fly ball rate. When no one is on base, Porcello is an ace. Meanwhile, Norris does a poor job in those situations. He gives up a lot of walks and has a horrific FIP of 4.68.

The problems start for Porcello when runners are on base. His K-BB% drops from 15% when the bases are empty, to 5.2% when a runner is on base, to 2.8% when a runner is in scoring position. The amount of fly balls that he gives up stays the same, but he also allows more home runs due to a higher HR/FB%. His HR/FB% is higher than average for reasons that will become clear later in the post. Unsurprisingly, his FIP goes from 3.25 with the bases empty, to 4.58 with a runner on base, to 4.8 with runners in scoring position.

Meanwhile, Norris improves when men are on base. His K-BB% jumps from 9.6% to 14.4% and his HR/9 rate drops from 1.3 with the bases empty, to 1 with a runner on base, to 0.85 with runners in scoring position. Unsurprisingly, Norris has a better FIP when pitching with men on base than when pitching without men on base.

The bottom line is that Norris becomes more effective when runners are on base while Porcello is less effective. The problem with that is that ERA measures what actually happens so by definition, it takes Porcello collapsing with runners on base into account. After all, that causes him to allow more runs which counts against his ERA. FIP doesn't have a way of differentiating between how Porcello does with men on base and with the bases empty. The formula presumes that a pitcher will perform the same with runners on base than with the bases empty and therefore doesn’t take into account the fact that Porcello does a terrible job pitching with the bases empty. It seems reasonable that this flaw means that in this case, ERA is a more effective estimator than FIP. At the very least, it indicates that FIP is a bad estimator to determine Porcello's performance. Honestly, if any of the two pitchers has had bad luck it’s probably Bud Norris, as one would expect him to have a lower ERA than his FIP which isn't the case.

Furthermore, Porcello’s performance in this regard has been reasonably consistent. This is what he’s done from 2012-2015.

His performance has been pretty much consistent. It's true that he did better with the bases empty in 2013 than he has in previous years. He has also performed slightly worse with the bases empty in 2015. Likewise, when men are on base the numbers are also reasonably consistent. His 2015 FIP is a bit worse due to an elevated HR/FB% but his 2015 xFIP is in line with normal figures.

The only case where there’s a significant change is in 2014 when runners are in scoring position. In those situations, his FIP was 3.9 while his average FIP from 2012-2015 was 4.8. But the reason why his FIP was so good in 2014 in those situations was because of a 4.8% HR/FB rate and not because he was able to fix his poor K-BB%. A 4.8% HR/FB rate is not sustainable and indeed his xFIP for 2014 with RISP is similar to his 2012-2015 average.

Basically, the data show that Porcello hasn’t had a good K-BB rate with men on base in any year from 2012 to 2015 and that his success in 2014 was due to avoiding home runs with men on base. That's not a strategy for success.

This next table is created with data from ESPN's Stats and Information portal and further shows how Porcello has done from 2012-2015 with men on base.

It tells pretty much the same story. I'm including it because it has statistics like OPS and wOBA that may be more useful to the user, It also shows how a deflated BABIP also contributed to Porcello’s success in 2014. Looking at Porcello’s performance in 2015, we can pretty safely say that his good fortune didn’t continue. A pitcher doesn't often have a .684 OPS with an 11.60 K% and a 9.00 BB%.

One might wonder why Porcello was able to give up fewer home runs in 2014 than he did in other seasons. This next table, using data provided by ESPN Stats and Information, shows how many fly balls Porcello allowed with RISP from 2012-2015.

Porcello's fly balls weren’t hit as hard in 2014 with RISP as they were in 2013 and 2015 but they were hit as hard as they were in 2012. That could be seen as a good sign, but the problem is that Porcello only allowed 40 in those situations in 2014. This is an awfully small sample and in light of his 2015 results, it appears that it was just fortunate chance. This is especially supported by the fact that his fly balls were hit roughly just as hard with men on base in 2014 as they were in 2012 and 2013. He's been pounded pretty badly in 2015.

The next question is why does Porcello struggle to get strikeouts when runners are in scoring position? This is easily answered by looking at the results of his pitches over the period using data from ESPN Stats and Information. Here’s a chart.

Porcello throws more strikes when the bases are empty than when there are runners in scoring position while also allowing fewer balls being put into play. This results in him having a higher percent of called strikes when the bases are empty than when runners are in scoring position as well as also allowing more foul balls. It would seem that batters are better able to predict where his pitches will go when batters are in RISP chances than not. All in all, more strikes and fewer balls put into play results in more strikeouts and fewer walks when no one is on base.

This next table shows how batters perform against Porcello’s pitches.

Porcello appears able to throw his fastball for strikes and can use it to get strikeouts. The problem is that batters absolutely annihilate them when they put them into play. Batters hit the pitch so hard in fact, that it probably is a bad idea to throw it. In addition, batters also crush his curve/slider when they put those pitches into play. Those pitches appear to be slightly successful when no one is on base but result in absolute disaster when men are on base. The bottom line is that he only has an effective sinker and changeup. It turns out that there's a major difference between being able to throw five pitches and being able to throw five pitches well.

Naturally, the Red Sox have adjusted to this fact by changing what pitches he throws. This next table shows the percentages of each pitch he throws each year.

For some reason, the Red Sox have decided that Porcello should throw his fastball more often and that he should throw his changeup and sinker less often. Or they’ve decided he should throw his worst pitch rather than his best pitches. I have no idea why they'd resort to this strategy but it turns out that having a pitcher throw his worst pitches more often results in him having a worse year than average.

This suggests that his results in 2015 have been earned and that they aren't representative of how he could perform used properly. It also makes one wonder whether what the Red Sox are planning and whether they can use him properly.

If one just looks at WAR, FI,P and health, then Porcello appears to be a good pitcher. He’d almost definitely be considered above average if not a solid No. 2. Given that he's been healthy, it would seem reasonable to give him a large contract based on his prior performance despite his poor ERA.

However, if one takes a more in-depth look at his stats, it quickly becomes clear that he’s terrible when men are on base or in scoring position and was successful in 2014 solely due to luck with home runs. It seems he doesn’t have a viable fastball, is unable to throw strikes in the clutch, and that his ERA is probably a better predictor of his true ability than his FIP. This probably means that he’s a No. 5 starter and his true ability is limited. I suppose he may be better than Bud Norris but certainly not worth $20 million per year.

That’s exactly why one can’t just look at WAR to gauge ability. While WAR is helpful, it’s solely a number that summarizes a pitcher's performance without providing much insight into why a pitcher performs the way he does. Sometimes that insight makes it clear that a pitcher isn’t as good as one would otherwise think.