Manufactured Runs: How Do You Solve a Problem Like Derek Jeter?

One of the things about being a baseball analyst as a writer is that I have the luxury of taking the broad view. If I am right about most players, I’m doing a pretty good job. If I miss on a player here or there…I don’t enjoy it, and I try and learn from it, but it’s not devastating.

A general manager, on the other hand, is responsible for about 25 players or so—more if you include everyone who could conceivably play on a team in one season, fewer if you limit it to players who will end up having (or missing) enough playing time to have a real impact. Missing on one player can, in fact, be devastating.

All of which is to say—I don’t envy Brian Cashman right now. He has a lot of tough choices to make about Derek Jeter—and if he slips up, Yankees fans aren’t likely to be forgiving. Even if he does the right thing, Yankees fans may not be understanding.

While none of us shares Cashman’s unique burden, the outright refusal of MLB to play even a single game between now and March means a lot of us will satisfy our desire for baseball by following the Hot Stove League, and Jeter’s contract negotiation figures to be the star attraction. So, let’s ask, how much is Jeter worth?

Figuring out how much Jeter’s bat has been worth is relatively trivial; it’s also pretty uncontroversial. Here’s runs above replacement player for Jeter over the past five seasons from us, Baseball Reference, and FanGraphs, excluding defense:

Everyone seems to be using subtly different definitions of replacement-level offense, but past that there seems to be solid agreement between the three as to what Jeter has been worth with the bat.

The question the Yankees have to answer, at least in terms of batting, is this: was 2010 simply an aberration for an otherwise exceptional hitter, or was 2009 an Indian summer masking Jeter’s age taking its toll? What our projections can tell us is the most likely answer (or, more accurately, the answer with the least presumed error), but there’s still uncertainty around that forecast, and that uncertainty compounds the more years you tack onto the contract.

(The Yankees also need to come to grips with where in the lineup Jeter’s bat belongs. Barring a substantial bounce-back, it’s hard to make the case that the Yankees are best served with the Captain leading off every game).

But still, looking simply at Jeter’s bat and his position, he seems like an average to above-average player over the next few seasons, and the Yankees seem to lack decent alternatives (either internally or available through free agency). It seems like a simple decision to retain Jeter in pinstripes, doesn’t it?

The trouble is fielding.

Let’s look at several common fielding stats, all expressed in terms of runs saved compared to the average player at the position—Defensive Runs Saved as published by Baseball Info Solutions, Ultimate Zone Rating (based on the same BIS data as DRS), Sean Smith’s TotalZone, BP’s current implementation of FRAA, and our forthcoming overhaul of FRAA. Looking at years for which all of those metrics are available:

Someone relying on UZR or TotalZone might reasonably conclude that Jeter’s defense was subpar, but unlikely to deter the Yankees from wanting him to return. Looking at DRS and FRAA1, one might be more skeptical.

New FRAA, however, casts more substantial doubts on the question.

I’ve discussed what new FRAA does before, but a refresher is probably in order. Simply put, we count how many plays a player made, as well as expected plays for the average player at that position based upon a pitcher’s estimated ground-ball tendencies and the handedness of the batter. There are also adjustments for park and the base-out situations; depending on whether there are runners on base, as well as the number of outs, the shortstop may position himself differently, and we account for that in the average baselines.

The other metrics use other data to come to their estimate of expected outs—in the cases of UZR and DRS, it’s batted-ball and hit location data measured by BIS video scouts. In the cases of TZ and FRAA, it’s data collected by press box stringers working for MLB’s Gameday product.

(TotalZone and FRAA1 both incorporate some batted-ball data from MLB’s Gameday product from 2005 on; this means that what we’re calling TZ and FRAA1 aren’t exactly the same thing depending on season. GuyM has explained why range bias seems to exist in pre-hit location metrics, which is probably beyond the scope of this discussion).

So let’s examine nFRAA for all seasons Jeter played in the majors:

Over the course of his career, Jeter has made nearly 500 fewer plays than we would expect a shortstop to have made. MOE_PM represents the margin of error around our estimate of an average shortstop’s plays. What I want to re-emphasize is that the margin of error doesn’t scale linearly—the margin of error for three seasons is smaller than the margin of error for each of those seasons added together.

Looking only at 2003-2010, the cumulative MOE (expressed in runs, not plays) for those seasons is 41.209. In other words, 68 percent of shortstops with that number of chances will have their “actual” value within 41 runs of the value estimated by FRAA—that’s one standard deviation. Assuming a normal distribution, 99.73 percent of players will have their actual value within three standard deviations of the estimate.

The estimates of Jeter’s defense provided by UZR and TZ are about 3.7 standard deviations away from what nFRAA says. It is staggeringly unlikely that Jeter would end up with a batted-ball distribution that cost him so many opportunities based upon random chance alone. DRS is just under two SDs away (roughly, the 95 percent confidence interval)—still very, very unlikely.

If not random chance, then what else might it be? Is there something else that could be so consistent across Jeter’s career? We’ve already controlled for park factors, ground-ball tendencies of the pitchers, and the handedness of opposing hitters. He’s played in nearly 40 ballparks (including two home parks), behind nearly 80 starting pitchers, with two managers (one of whom was still catching for the Yankees when he started playing). Since Jeter became the full-time shortstop for the Yankees in 1996, the team has had six different starting second basemen and five different starting third basemen (not to mention the numerous backups to each). The single-greatest constant to Derek Jeter’s career has been, well, Derek Jeter.

The appropriate response is that the probabilities assume we’ve plucked Jeter at random; of course we haven’t. But we haven’t cherry-picked Jeter because he’s an isolated case. He’s an example of a systemic problem with fielding metrics based on observational data—range bias.

It’s a fairly straightforward matter to determine whether a play was made or not—primarily, you figure out if the batter was safe or out after putting the ball in play. It’s nearly as simple to figure out who made the play; typically the idea of “first touch” is used—who was the first fielder to contact the ball? So "plays made" is mainly an assertion of fact. Expected outs, derived from observation of where the ball was hit and how it travelled there, is more difficult. Here’s a still frame grabbed from a highlight of Jeter making a play:

What I want to emphasize is that, other than Alex Rodriguez, the only real reference point to where the ball is when it reaches Jeter is Jeter himself. Because of the way baseball telecasts are shot, this isn’t an isolated incident—this is how baseball looks on TV. And companies like BIS get the same video feeds the rest of us get.

So a player’s range seems to influence his expected outs. How can we tell this? The first data point we have is the spread of observed fielding performance—the spread of observed performance in metrics like UZR and DRS is much, much smaller than that of metrics like nFRAA (or Tom Tango’s With Or Without You system, which is similarly down on Jeter’s fielding ability).

The other thing we see is that a player’s expected outs as a share of team balls in play (or “balls in zone,” a proxy measure for expected outs) persists from season to season, even when looking only at players who switch teams. In other words, the identity of the player seems to change the recorded batted-ball distribution (We have no mechanism that would allow us to explain how a fielder could dramatically change the actual distribution of batted balls; it seems much easier for him to be impacting the estimates).

All of the available evidence seems to suggest that Jeter is a worse fielder than most defensive metrics indicate, perhaps on the order of 20 to 30 runs below the average shortstop. This makes it possible that Jeter, in 2010, was performing at roughly the same level as a typical replacement—in other words, his ability to hit like something resembling an average shortstop doesn’t offset his inability to field like one. And while Jeter’s bat may improve next year from a disappointing 2010, over the next three years it’s a fair bet that his hitting will continue to erode. That’s what happens to baseball players as they get closer to 40.

No, I don’t envy Brian Cashman at all.