Moonshot: Finding PECOTA’s Limits

Image credit: USA Today Sports

The annual spring release of PECOTA‘s projections almost always incites a controversy and this year has been no exception. The most aggrieved fanbase for 2019 belongs to the Chicago Cubs, who PECOTA predicts for a modest 79 wins. More importantly to the 2016-2017 NL Central winners, the projection system puts them in the last place in the division. With the Cubs’ fans, players, coaches, and even president of baseball operations throwing shade at Baseball Prospectus, it’s worth considering what’s going on under PECOTA’s hood, and why the prediction that drives the most scorn—the last-place finish—is also the most difficult part of a projection system’s job.

All projection systems have limits. Anyone who tells you that they have consistently predicted a team’s finish to within one win, as Theo Epstein did in criticizing PECOTA, is either exaggerating or very lucky. There is so much uncertainty in how a baseball season will go, from not knowing the strength of a pitcher’s elbow ligaments to whether a gust of wind will bat down a long fly ball in a pivotal September game. A single injury to an important player is enough to put a team’s playoff hopes to bed. A wicked new pitch might be enough to turn a sub-replacement-level fifth starter into an ace, vaulting an iffy rotation into a good one.

These are possibilities PECOTA can account for probabilistically, but not foretell with certainty. The best any algorithm can do is average over all the many thousands of possible ways a season could unfold, conditional on the knowledge it does have. What PECOTA knows is how each player did in the preceding seasons, as well as how similar players over baseball history have performed. With that information—and that alone—PECOTA generates a prediction about the player’s next year.

Once PECOTA gets a read on each individual player, projecting a team is simply a matter of combining the total contributions of the roster. (This sentence sweeps some of the complexity of assigning playing time under the rug, but that’s a topic for another day.) Conveniently, each player’s contributions can be measured in runs, and by summing the total runs saved and generated, it’s possible to build a prediction for how good a team will be.

Based solely upon the Cubs’ predicted runs scored and runs allowed, PECOTA would peg them as an average team—scoring roughly as many runs as they allow. There’s plenty of uncertainty around that 79-win total. Internal Baseball Prospectus confidence intervals suggest that the Cubs could end up from 73 to 85 wins, even if they are only an average team.

But PECOTA doesn’t stop at assessing team strength—it also gives us a measure of where the team will finish within its division. That’s an exponentially harder problem. Most teams vacillate between 65-105 wins in a typical year, which gives PECOTA one of about 40 reasonable choices for how a team will do. Most of these choices will be absurd for any given team; the Orioles won’t win 105 games, the Yankees won’t win 70.

By contrast, there are 120 possible configurations of finishes in a five-team division (and many more than that in baseball as a whole). Even to predict whether a single team will finish higher than another team in the division requires PECOTA to make twice as many guesses—one for the first team’s win total, and another for the second team. Throw in the fact that teams within the division also play outside the division, and the problem rapidly becomes exponentially more complex than guessing any single team’s true strength.

In a tightly-compacted division, with each team within 10 wins of each other, all configurations are reasonable guesses, unlike the Orioles winning 105 games. A conservative estimate therefore puts the difficulty of predicting division finishes at about three times harder than how a single team will do (120 possible outcomes compared to 40).

The Cubs suffer from playing in what may turn out to be the hardest division in baseball this season. The up-and-coming Brewers, perennially contending Cardinals, surprising Pirates, and new-look Reds form a murderer’s row of competition in the NL Central. In a normal division, the Cubs might be projected for a couple of extra wins and a third-place finish. In the NL Central, that same team might only be good enough for last place.

It’s an interesting thought experiment to consider how this year’s PECOTA rollout would have unfolded if the algorithm had projected the Reds and Pirates to be slightly worse—and the Cubs for third place (even with the same win total). Would the Cubs’ players and coaches have turned PECOTA’s wins forecast into bulletin board fodder? Would senior team officials be trashing the algorithm? I suspect PECOTA would have offended far fewer people by suggesting that the Cubs are mediocre than suggesting that they are a last-place outfit, even if they are both.

There are valid reasons to be skeptical of a team’s projection. PECOTA lacks knowledge on the specifics of each player’s makeup, from their ability to make adjustments to their health. Fans may know more about those factors than a projection system. To PECOTA, any player is only a sequence of numbers, nothing more, nothing less. All the algorithm does is guess the next digits in that sequence.

If you’re a Cubs fan, you may be inclined to credit Kyle Hendricks or Jon Lester (or Kris Bryant, or Yu Darvish) with abilities exceeding PECOTA’s purview. That’s fair; you probably know more than PECOTA does about any particular player on the roster. But what PECOTA knows, and what you likely don’t, is how all of the hundreds and thousands of previous players throughout baseball history who look like Hendricks and Lester did the next year. And what PECOTA does better than the vast majority of humans is synthesize that knowledge into a prediction.

PECOTA has its limits. One of them is forecasting the order of finishes in a tight division. You shouldn’t depend on an algorithm to tell you that one team will win one or two games fewer than a division rival. That’s not what projections are built to do, nor are they capable of doing so accurately. The best they can provide is a rough measure of a team’s strength. For better and for worse, and no matter how much it offends the Cubs themselves, PECOTA sees them as roughly mediocre. The rest—including whether they can chase down the Brewers to claim the division, or make the playoffs at all—is up to the random chance of baseball.