The following graphic represents the BIS data from 2002-2005 that is used as the basis for statistics such as UZR and +/-. The three plots represent all batted balls for those seasons, broken down by batted ball type. The graphic displays a major flaw in the current defensive metrics that are based on batted ball data. Can you spot the issue (I will post the answer in the comments)?
[image title="data charts" size="full" id="23292" align="center" linkto="full" ]
If you enjoyed this article, please consider sharing it!
12 Responses to Spot The Problem With This Graphic
Leave a Reply
TIQ IQ
The issue is that batted balls are unlikely to be distributed exactly based upon the position of fielders. You would expect a more uniform distribution of balls across the entire field. The graphic suggests that the scorers are exhibiting range bias, where they are attributing balls to zones not based on whether the ball actually went to that zone, but basing the zone on the fielder. This means that if the outfielder goes into another zone to field a ball, the scorer still may be scoring the ball in the fielder’s initial zone. For metrics that are purporting to measure range, this is a big, big problem.
Good call, Moshe.
I always thought it was a little coincidental that Jeter always had a terrible UZR and then it immediately goes up when they change managers and thus changing exactly where he would have been positioned.
Positioning is huge for a short stop anyway but especially for a SS losing his range because you age you start to understand more and more where the ball should be by knowing what pitch is being thrown, this is the main reason that Jeter had a “better defensive year in 2009 because was positioning himself better but at this point in his career if he positions wrong in the least he doesn’t have the ability to make it up and get to the ball.
Yeah, the fact that no defensive metric takes initial positioning into account when calculating range forces me to not take any of them too seriously.
They may be tangentially correct, but their error bars should be sky-high, at least 10 runs in either direction. Makes the information not very useful.
Moshe, well done. I think that you are also showing a fundamental reason that stats can’t be taken as fact until there is more empirical evidence, i.e. confusing cause/effect.
This has always been the problem with UZR. But I still think it’s useful in the extremes, players rated very good or very bad tend to be correct. It will catch the stiff who’s not getting to anything, and the speedster who runs down balls way out of his zone. It’s everyone in the middle that I don’t believe.
Thanks for the intellectual honesty and rigor in pointing out one of the major flaws in the “advanced defensive metrics” and thus by extension WAR which uses these metrics in attempting to spit out a number which purports to calculate absolutely and comparatively, the “true value” of players throughout MLB for a single season and throughout their careers. Unfortunately, in their haste to promote their point of view, “sabermetricians” have in many instances put forth metrics which neither pass the eyeball test- Texeira was a below average defensive first baseman in 2009 according to UZR to illustrate a flawed finding which most who read this post can understand – nor the intellectual reality test when examined closely. Much like new drugs which have to pass years of FDA scrutiny before their initial claims are supported and they are available to the general public, so too should “advanced metrics” be subject to objective questioning and debate. Unfortunately, many “sabermetricians” become very defensive, pardon the pun, when one challenges the validity of their latest toy. Thanks for taking the time and effort to illustrate one of the fundamental flaws of the “advanced defensive metrics” and why they are often of limited, if not less, value.
Thanks classicsteve. Advanced statistics, perhaps especially the defensive statistics, are undergoing continual reassessment. This is a process of innovation. There will be error as a result of continuous trial. Defensive statistics will hopefully get better, but in the meantime, how do you assess Teixeira’s value at first base? By the eyeball test? If you have any results, please share them. Until then, please don’t scorn people trying to advance understanding of the sport we all love. Add to the conversation if you can. I look forward to an informative response.
when you nit wits get it right let us know..until then shut up, you suck
Sorry Damian for the time involved in getting back to you. I was away and didn’t check this site for a couple of days.No need to be so defensive. I wasn’t scorning- your word- those who have put forth the newer defensive metrics. However, I was saying that often these metrics are flawed in their methodology and often used as a basis to authoritatively measure things which frankly they don’t. For example, as I am sure you are aware, there are currently two different measures of WAR being put forth- Fangraphs WAR and Baseball-reference WAR- which purport to measure the same thing, a player’s overall contribution during a particular season and by extension his career. However, these two metrics often diverge drastically not only in their absolute measure- for example Player A can have a fWAR of 7.0 and bWAR of 5.5- but in their comparative ratings among two players- for example Player A can have an fWAR of 7.0 and bWAR 5.5 and Player B can have an fWAR of 6.0 and a bWAR of 6.5. Because of this one has to question the legitimacy of both numbers and by extension whether a player’s value as determined by WAR- assuming $4.5 per win- exceeds or is below his contract or his career WAR should merit consideration for the HOF to use two examples which WAR is incorrectly used for in my opinion..
While I respect that those who develop these metrics are trying to advance the understanding of baseball it does mean that I have to blindly accept their efficacy. Obviously, defense goes beyond primitive gross fielding percentage. I do believe one with a reasonably trained eye who observes enough games can determine that Keith Hernandez was a better defensive first baseman than Jason Giambi and Carl Crawford blows Marcus Thames away in left. However, as we move away from the extremes and go closer to the mean these determinations are more difficult to make but I don’t believe that UZR gets us there for the reasons pointed out by Moshe initially. One solution I had proposed , at least in terms of determining Gold Glove winners, would be to put together for the voters a tape which included all the plays in which players are involved in for a particular season from Game 1 through Game 162.. After looking at this, one would have a sense of the range , sure handedness and arm strength of the individuals involved. Of course there would be difficulty making this information available available to the public but I do believe it is possible..
In closing, I am all for statistics which better measure all aspects of a players performance- offense, defense, arm strength, base running. However, to validly critique an innovation is not to scorn the interest of the persons involved or the effort expended; rather, it is to suggest that the metric may not accurately measure that which it purports to.
E-5 Says:
when you nit wits get it right let us know..until then shut up, you suck