03 April 2014

Is Defense Overrated?

Fangraphs recently released some Inside Edge fielding data to the public. As stated in the article, Inside Edge scouts watch every play and determine how difficult it is to field on the following scale:

  • Impossible (0%)
  • Remote (1-10%)
  • Unlikely (10-40%)
  • About Even (40-60%)
  • Likely (60-90%)
  • Almost Certain / Certain (90-100%)

Unlike zone-based applications like DRS and UZR, Inside Edge defensive system results are determined primarily by scouts. This data gives insight to how scouts grade defense and determine similarities and differences between zone-based and scout-based systems.  

The Inside Edge fielding data available to the public states how many chances and the success rate for each fielder in every category listed above. For example in 2013, Nick Markakis had 112 impossible chances of which he converted zero, five remote chances of which he converted zero, five unlikely chances of which he converted two, six about even chances of which he converted all six, eighteen likely chances of which he converted fifteen and 289 almost certain chances of which he converted 288.
I used this data to create a metric I think of as Catches over Average (COA). It is derived by downloading all Inside Edge fielding data for each position and determining the average success rate for each of the six categories described above. Next subtract the average conversion rates for each category from the players performance and multiply by the number of chances. Finally, sum the results for each category. Many concepts incorporated in this metric were originally discussed here.

This metric has a number of shortcomings. It presumes that each chance in a category is of equal difficulty. It doesn’t measure how successful an infielder is at turning a double play. It doesn’t measure how successful an outfielder is at ensuring singles don’t turn into doubles or outfielder arm strength. Nor does it measure how well a catcher frames pitches or prevents passed balls or throws out potential base stealers. Inside Edge collects this data but doesn’t share it with the public. Despite these shortcomings this metric still can be used to compare Inside Edge fielding data results to UZR and DRS.

This table shows how some of the Orioles main players performed defensively in 2012 and 2013.

There are many similarities between COA and UZR or DRS. Matt Wieters, Manny Machado and JJ Hardy are considered excellent defenders while Ryan Flaherty and Chris Davis are considered above average. Each indicates that Mark Reynolds and Wilson Betemit are unable to play third base very well while Adam Jones is a poor defender in center field. The main difference is that Nick Markakis is a good defender according to COA and a bad one according to UZR or DRS.

In 2012, our outfield defense ranked 16th in the majors with a -1.73 COA while our infield defense ranked 7th with a 9.13 COA. In 2013, our outfield defense ranked 26th in the majors with a -2.40 COA while our infield defense ranked 2nd with a 30.9 COA. Having a full season of Manny Machado at third base instead of using Wilson Betemit and Mark Reynolds had a significant impact on our infield defense. This statistic indicates that the Orioles have excellent infield defense and mediocre outfield defense which is the common consensus.

The difference between the best and worst infield defense was 48 COA in 2012 and 52 COA in 2013 while the difference between the best and worst outfield defense was 30.5 COA in 2012 and 20 COA in 2013. These numbers seem low when compared to UZR especially when one considers that a catch doesn’t equal a run. In order to do a full comparison it is necessary to determine how many catches equal a run.

Tom Tango states that each catch is worth .8 runs. This implies that the amount of runs saved can be determined by multiplying the value of a catch by a players COA. I’ll refer to this stat as Defensive Runs over Average or DROA. *

 It was brought to my attention that Tom Tango has discussed the value of saving a play. Originally, I quoted Michael Lichtman who stated that a typical outfield hit is worth .83 runs and simply estimated the value of an infield hit. I believe that estimate was incorrect.  

It is possible to compare DROA to UZR. The table below shows the ranges for UZR and COA for each position.

Position COA Range Value Of Catch DROA Range UZR Range UZR/DROA Range
1B 16.56 0.80 13.25 30.90 2.33
2B 25.94 0.80 20.75 31.70 1.52
3B 26.01 0.80 20.80 48.00 2.31
SS 25.22 0.80 20.18 45.20 2.24
CF 14.60 0.80 11.68 42.50 3.64
LF 12.15 0.80 9.72 31.30 3.22
RF 13.31 0.80 10.65 47.10 4.42

The range for UZR is about 2.25 times larger than DROA for infielders (except second base) and 3.75 times larger than DROA for outfielders. Instead of a top defender like Machado being worth 3 wins defensively this stat indicates that he’s worth one win. This implies that defense is less valuable than one may have thought based on UZR. **

** Due to the new data, I've updated the ranges quoted above.

The good folks at Retrosheet attempt to document every baseball game played. They share their data with the public provided that the following disclaimer is used:

Information used here was obtained free of charge from and is copyrighted by Retrosheet. Interested parties may contact Retrosheet at www.retrosheet.org

I downloaded the data from the 2012 and 2013 seasons from their site and determined how many balls in play allowed by each team. I split the balls in play into outs and non-outs (including hits and bases reached on error). Then I did the same thing with the Inside Edge Data. Below are the results.

The numbers of outs for balls in play are different in each dataset by a minimal amount while there are more than twice as many balls in play in the Retrosheet dataset for each team than in the Inside Edge dataset. The average team had a BABIP of .298 using the Retrosheet data and a .188 using the Inside Edge data. The Inside Edge data is omitting a large number of hits.

UZR and DRS split up the whole field into zones. Each ball in play has to fall into a zone which is the responsibility of at least one but potentially more fielders. If a ball isn’t caught then it is someone’s fault and it impacts their UZR/DRS rating. The Inside Edge data seems to indicate that part of the field is no fielders’ responsibility. Other parts of the field may be a fielders’ responsibility but an uncaught ball hit there has no impact on his defensive rating if it’s considered an impossible chance like most non-out balls in play. Balls hit into play that do have an impact of a fielders rating are usually ones that every fielder can field with nearly perfect accuracy. As a result, there is less variation in defensive ratings and an elite player according to UZR may be worth 3 wins defensively while an elite player according to DROA is worth 1 win defensively.

It is possible to determine whether UZR or COA is more likely to predict future performance by doing a correlation analysis. I found that UZR had a correlation of .393 from 2012 and 2013 while COA had a correlation of only .249. This suggests that UZR is more likely to predict future performance though neither is particularly accurate. I also found that while range runs above average is the best factor to predict UZR that error runs above average is the best factor to predict COA which supports the above paragraph. This explains why players similar to Nick Markakis that have limited range but make few errors have much higher defensive ratings using Inside Edge data rather than UZR while players like David Lough that have excellent range but make errors have much higher defensive ratings based on UZR rather than Inside Edge.

The average AL team allowed 696 runs in 2013. According to UZR, the Royals defense (best in the AL) was worth 68 runs. This means that a team with average pitchers and the Royals defense would have allowed the fourth fewest runs in the American League. Using DROA, the Orioles defense (best in the AL) was worth 16 runs. A team with average pitchers and the Orioles defense would have given up the ninth largest amount of runs or still been about average. The amount of importance that UZR places on defense as opposed to Inside Edge is huge.

If UZR is correct in its rating of defense then focusing on defense should be a cost-effective way to improve performance. If Inside Edge is correct then excellent defense players offer smaller contributions than UZR and WAR would suggest. I believe that the Inside Edge data suggests that defense is overrated.


Jon Shepherd said...

Interesting article.

I would like to make a note though. Inside Edge does claim to have the "best" data because they use ex-baseball player "scouts". DRS and UZR use data collected from Baseball Info Solutions. Much of this data is collected by people who you could arguably refer to as a "scout". I would not, but you could. Our own Joe Reisel is one of those datacasters. That said, I am not sure that Inside Edge's hiring is remarkably different from BIS's. Perhaps I am wrong, but I would hesitate to make any conclusions on that without fully understanding their methodology as well as the quality of their hiring.

Matt Perez said...

I believe that you're right that Inside Edge's hiring isn't remarkably different than BIS. I think both use veteran scouts and ex-players.

My understanding is that BIS scouts are being used to determine in which zone a ball landed whereas Inside Edge scouts are told to grade the data based on their expertise. While BIS may be using scouts, I think that Inside Edge gives their scouts a larger say in the process.

Jon Shepherd said...

Not only the zone the ball falls into, but also the flight path of that ball and whether it was an exceptional play. Inside Edge does seem to give more control to their scouts.

That said, I am not sure if having more say is useful.

Unknown said...

As a BIS contractor, I don't know how far I can go in detailing their methods, so I won't go there at all. I will say that I am neither a scout nor an ex-player.

Jon Shepherd said...

Joe, I would also suggest that many of Inside Edge's are not either. However, I have no direct information on that.