Pebble Hunting: PECOTA Hates the Royals, Part II

Sometime today, or thereabouts, we expect to release all 2016 PECOTA projections. As part of that, there is a Royals team projection. And as part of that, there will be rolling of eyes, because, surprise, it is not a very optimistic projection. So in anticipation of the full PECOTA release…

Well, cuss.

This is the projected win total that nobody wanted. I didn’t want it. I’ve rooted for a lot of baseball outcomes, but this is the first time I’ve rooted for a baseball-projection outcome. It’d be a lot easier for me to spend the next two months answering questions about a Royals-Win-89 forecast than this one. We’ll get into that.

You didn’t want it. You want to believe in these projections. You know that there are mysteries about baseball that we and our little human brains can’t grasp, and you want to believe that those mysteries can be solved by the power of science and math. If we as a species can’t project Wade Davis, how will we as a species ever fix global warming or colonize Mars? We’ll get into that.

Our staff didn’t want it. It’s not just that when we polled our writers for their own Royals predictions—before PECOTA had been run—not one of them went as low as 76. It’s not even that not one of them went lower than 80, or that only one of 27 responses was lower than 85, or that the plurality response was 90, or that the average was 88. It’s this: When I asked a follow-up question a few days later—“If I told you PECOTA projects them to win 76 games this year, does your answer change?”—the response was overwhelmingly “nah.” In fact… counting them out… 73 percent of staff said it didn’t change their answer at all. As one put it: “No, because the projections just seem to not like the Royals.” This seems awfully close to a crisis of confidence. We’ll get into that.

The only people who wanted this projection are probably the Royals themselves, who are chilling on the fourth level of that Gandhi quote. Shredded projections are the confetti in their World Series parades.

If I were going to make the case against projection systems—without doing any research, just a common sense objection—it would be that the formula that misses on a player (or team) one year is in no position to project what they’ll do the next. This isn’t meant as a blanket “bah humbug” to the idea of using events of the past to forecast the future—everybody does that, just in less systematic ways than a system does. But much of the information that goes into a 2014 projection is carried over into a 2015 projection, and then into a 2016 projection. To give a clear, hyperbolic example: Imagine a player was playing without eyes in 2011, 2012, 2013 and 2014. He gets an eye transplant before 2015. PECOTA, with its multi-year memory, is going to a) miss very badly on that guy in 2015, and b) continue to miss until the blind years are out of its memory. Real world instances of this phenomenon surely abound, out of our sight but embedded in the PECOTA “misses.” Long story short: If bad intel got into PECOTA’s 2015 Royals projections, and PECOTA is mostly the same in 2016, and the Royals are mostly the same in 2016, why take the 2016 projections seriously?

Good question!

The Royals, coming into 2015, had outperformed PECOTA by 21 games in the previous three seasons. Only three teams (Baltimore, Pittsburgh, Oakland) had outdone PECOTA by more. That the Royals then overperformed yet again seems suggestive. That Pittsburgh (one of the teams in the parentheses one line up) was PECOTA’s second-biggest “miss” last year, winning 18 more games than projected, seems suggestive. That the A’s were PECOTA’s third-biggest “miss” would seem suggestive, except that the A’s won 15 fewer games than projected. Fewer. The team PECOTA had been consistently lowest on turned into the team that PECOTA was wrongfully highest on.

If we look at teams’ performances vs. PECOTA over the three year period entering 2015, and the same teams’ performances vs. PECOTA in 2015, there’s a correlation of about .30. That's weak, but still: If PECOTA was low on a team from 2012-2014, it was somewhat more likely to be low on the same team in 2015. But if we extend this to previous years, the correlation dissolves:

• Correlation between 2011-2013 "miss" and 2014 "miss": .07.
• 2010-12 to 2013: -.07
• 2009-11 to 2012: -.06
• 2008-10 to 2011: -.28

So, over a half-decade, there is no tendency for PECOTA to err consistently on the same teams. (Doing the same exercise year-by-year—so comparing a team’s performance vs. PECOTA over a single year to the same measure the following year—shows only slightly better, with a year-to-year correlation under .1.) So, PECOTA either doesn’t “miss”—but, rather, only looks like it misses when the least likely (but still acknowledged possible) outcomes happen—or PECOTA does “miss” but is quite adept at learning from its mistakes the next year.

Which is not to say that PECOTA got the Royals right this year. It’s just to challenge the preemptive claim that PECOTA definitely got them wrong, on account of a Royals blind spot. That’d be a very reasonable hypothesis, but it doesn’t bear out.

So to the Wade Davis problem. I did an autopsy on last year’s 72-win projection (which, hey, nobody gives PECOTA’s later runs credit for bumping it to 73 by the start of the season) for JABO last summer. You can read the whole thing, but here’s the intro and the bullets:

There are three foundations to a projection system: predicting who will play, predicting how well they will play, and converting those individual performances into team wins. Every projection "misses," in the sense that none predict with 100 percent accuracy, but we can see, in five critical projections, how the Royals especially diverted from their team-wide outlook.

Those five:

1. Wade Davis’ short history of dominance made him just about impossible to project algorithmically
2. Lorenzo Cain’s unique baseball background caused his late bloom sneak up on us
3. Edinson Volquez’s peripherals made a good case for him to be a mediocre pitcher, as he had been for most of his career. He then outperformed his peripherals. (He actually improved his peripherals in the final weeks of the season, after I wrote that piece.)
4. Ben Zobrist (and, sort of, Johnny Cueto) were added five months after PECOTA signed its work, and contributed to the Royals’ final record.
5. The Royals were super clutch.

Those five (as specific things, and as in-a-nutshell embodiments of the Royals as a whole) actually get you pretty close to the entire gap between projected and actual. There were other things going well—Chris Young and Mike Moustakas and Ryan Madson and so on—but there are always things going well for every team, just as there are things (Greg Holland, Alex Rios, Jeremy Guthrie) going terribly. The five bullets above are the combination of factors—massive breakout performance, good luck, literally unforeseeable roster changes, and (arguably) a projections blindspot—that led to a 23-win whiff.

We made some changes to the PECOTA this year. We’re using cFIP components instead of FRA to project pitchers, which everybody around here is extremely happy about. We’re using our new catcher stats. Outfield assists have been incorporated into FRAA. We’ve shortened PECOTA’s memory for pitchers and are weighting the years differently. We’ve got team defense incorporated into pitchers’ ERAs. These aren’t small changes—well, the outfield assists probably is, but they’re otherwise substantial—and we expect they’ll make PECOTA better and more accurate.

But they won’t help a ton with the bulletpoints. If the Royals are superclutch again, we won’t see it coming. If they trade for two stars in July, it’ll be a surprise. If pitchers outperform their peripherals—even accounting for the team’s defense, now part of their projected ERAs—they’ll outperform the projections.

Cain does get twice as good a projection as he had last year, and if it still reflects heavy regression—well, he’s 30, and it’s not that unreasonable for PECOTA to expect Cain to follow up a Michael-Brantley-in-2014 season with a Michael-Brantley-in-2015 season. Wade Davis—dang, I wish we could get Wade Davis right. This year is better—nearly a full run lower than last year’s projected ERA—but I bet most people would still take the under on a 2.64 ERA from him. Last year, Craig Kimbrel’s projected ERA was 1.33, so it has been possible for a reliever to overcome the small samples and natural volatility of the role and convince PECOTA to put a capital-E Elite projection on him. But just two outstanding years, with "lousy starter" still on his recent resume, aren’t enough to get PECOTA there on Davis. Of course, Kimbrel’s ERA last year was 2.58, so maybe I shouldn’t root for that to change.

So how am I going to defend this when I’m asked about it by radio hosts over the next two months?

First, I’ll note that I had the Royals winning 86 in that poll-the-staff exercise. I’ll tell them that our staff generally feels a lot more optimistic about the Royals’ chances than PECOTA does. I’ll say that most systems like the Royals’ defense more than ours does, and that most observers praise the Royals for things (like a great clubhouse, great advance scouting) that PECOTA doesn’t account for (though, if they’re consistent and valuable factors, they should be showing up in players’ stats and thus influencing their projections; whatever.) I'll allow that the Royals have a pretty mediocre starting rotation, that the bullpen after Wade Davis isn't all technically all that special, and that it's a fun lineup (that provides great defense) but there are three or four holes in it. I might mention that the Royals' deviant projection last year wasn't especially unusual; some team or another is shockingly high or low every year, but a combination of factors set the Royals projection up to be spotlighted when it went wayward. I’ll say that PECOTA follows a series of guidelines that work pretty well in the aggregate, but that there are unknowns that are just as unknown to it as they are to any of us. I’ll say that PECOTA can’t do much more than take an incredibly unpredictable sport and make it a little bit more predictable. It’s not fate. Obviously.

But I’ll also note that, unlike the 73 percent of my staff, seeing PECOTA’s 76 did change my answer. How could it not? It’s absurd to think that I, a guy with a depth chart and a bunch of untested hunches, have any real way of knowing with any precision how many games any team is going to win next year. My answer to a question like this should be extremely malleable; it changed when I saw my staff’s answers to the poll, because they as a group know better than I do, and it changed when I saw PECOTA’s answer, because as a well-designed and constantly evolving system it simply knows more than I do. Maybe those changes end up cancelling each other out, but they definitely change.

And then I’ll probably take the easy cop-out and point out that FanGraphs only projects 79, and Clay Davenport 74. Safety in, er, numbers.

What I probably won’t say, because it’s a complicated thought and I’m not sure how to express it, is that I’m still not totally convinced that PECOTA was exactly wrong last year, or is wrong this year. It doesn’t look great, to be sure, and at this point it’d be a lot easier if PECOTA would just apologize and come join us all over on this side of 81. Clearly, when you're an algorithm that exists to predict the future, the outcome is the grade. But the lesson I’d like to think Nate Silver wanted us all to take from PECOTA—with its percentile projections for players; with its application in playoff odds—is that there are a lot of ways for a season to play out, and we only get to see one of them. I’m happy for BP to take some abuse for last year’s whiff, don’t get me wrong. But isn't it a lot more fun to think of the Royals as the miracle team that defied a bunch of smart projections, rather than to think of the projections as just being dumb?