Cold Takes: What PECOTA Got Wrong

Image credit: USA Today Sports

You’re familiar with the Santayana quote, I’m sure: “Those who cannot learn from history are doomed to repeat it.” You know why it happens, though, right? Because learning from history sucks. You generally don’t sit down and critique your victories, unless you’re writing a reflective journal for the first year of a grad school teaching program. (And in that case, you’re probably not recounting many victories; teaching is hard.) No, when you’re right, the most you’re likely to do is pat yourself on the back for your wisdom and foresight, then move forward with the assurance that you’ll never make a mistake ever again.

So, despite the price it demands in time and pride, it’s a good time to look back on some of our misses. This year’s PECOTA projections are now out for subscribers, and we’ve had a week or so to digest its secrets: the teams it loves (the Rays, like always) and hates (the Orioles and Royals, like always, but also new fun teams, like the Angels!). The familiar arguments are already peeking out from the February soil. But as we prepare for a new season, it’s instructive to look at who PECOTA missed out on, and see if there are any patterns for what happened.

To explain the column totals shown below: The red and yellow columns compare each player’s projected 2017 performance with their actual 2017 performance; green compares WARP. The blue column, on which these rankings were sorted, adjusts the initial projection against a shift in playing time. (Zack Godley, for example, was projected to be a 0.1-WARP player, but after a 2016 spent in relief, PECOTA only allotted him 25 innings.) Finally, the purple column assigns this year’s projected WARP, current as of February 9.

2017 Hitters, Underestimated

PECOTA is a projection system, and projection systems are conservative. This is their primary value. Human beings are creative creatures; we’re very talented at imagining the worst- and best-case scenarios for everyone, ever. Ask a hundred people “think about what your next birthday will be like” and none of them will evoke imagery of it being a Wednesday—so going to work, coming home, playing Chutes and Ladders with the kids, and then maybe, if you remembered to buy some, a small glass of middle-shelf Scotch.

You can tell which players PECOTA was unapologetic about because their 2018 projection swung violently in the face of new data; it just didn’t know before. “So that’s what Aaron Judge looks like,” it basically says, pretty much the same way we all did the second week of April. Same with Chris Taylor. A huge chunk of J.T. Realmuto’s disparity in value came from being seen as one of the worst pitch framers in baseball, and suddenly being amazing at it. I’m okay with giving PECOTA a pass on that one.

But there’s also a class of older players for whom PECOTA’s conservative nature stumbles into full-on stubbornness: players like Charlie Blackmon and Zack Cozart and Daniel Murphy and Anthony Rendon, for most of whom our Grand Old Computer is willing to completely deny that 2017 ever happened. The confluence of the not-technically-juiced ball and the swing elevation revolution has led to a lot more random breakouts, and as Matthew Trueblood pointed out, the oracle may be struggling to keep up. Or it just refuses; there have always been one- or two-year blips in the record, and there have always been compelling narratives for them. Maybe the best route is to think of PECOTA as your cynical high school buddy.

2017 Hitters, Overestimated

We’ve spent a few paragraphs talking about how stodgy our friend is, but don’t be fooled; PECOTA is never afraid to toss out a bucket of hype on a rookie now and then. The most famous one might be Matt Wieters, who was crowned in the 2009 BP Annual with a projected 7.9 WARP his rookie season. (He’s yet to double that as a career total, incidentally, nine years in.) So there’s no real blame to assign for the relative failures of Andrew Benintendi and Dansby Swanson, each of whom saw universal hype in their rookie seasons.

But there’s an interesting archetype developing among these names: the dead cat lack of bounce. Projection systems are trained to remember the past, and they’re instructed to regress, which means that they tend to struggle with the Dale Murphys of baseball: the guys who, at some random point in the aging curve, simply lose it. Sometimes a player has a bad year, or plays through an injury; but also, sometimes a player plays through an injury and it simply never heals. Alex Gordon is a good example of this, the definition of the phrase “nowhere to go but up” that insinuates that going any direction is even necessary. Albert Pujols also falls into this category; it’s strange to say that one-third of a win is optimistic, but it’s pretty optimistic at this point.

Which brings us to Miguel Cabrera. The trouble with an excel spreadsheet is that it can only provide the average of a player’s potential outcomes, and doesn’t differentiate between one player’s variance and another’s. Cabrera’s 90th and 10th percentile outcomes are four wins apart, a pretty high number, but it’s clear that PECOTA simply can’t imagine a second sub-replacement year. But we’ve seen it, and we know that even if it’s not likely to happen to a hitter as great as Cabrera, it happens every once in a while to someone. Cabrera may simply never be Cabrera again, just as it happened with, say, Joe Mauer. In baseball, no matter the talent, there is no floor.

2017 Pitchers, Underestimated

This list falls into two categories as well. There are a couple of breakouts that few could have foreseen: I wrote about Godley last week, but the emergence of Brad Peacock and Luis Severino are equally mystifying. PECOTA is on board for the latter, inserting him among the elite 2018 starters in nearly every major category; Peacock it banishes to middle relief, and Godley it doesn’t quite buy based on the random skill set. That raises another point you should consider when evaluating a player’s WARP: the player’s production is a combination of the skill that the system evaluates, multiplied by the number of innings or plate appearances estimated by very real human beings. Depth chart estimations are difficult in the cleanest of springs, and this is not that. So when Mike Moustakas signs and bumps a guy into a platoon DH role, or the Orioles fall in love with another Rule 5 pick, some of these numbers will change just for want of opportunity. Keep that in mind.

But that’s not what you noticed when you read these names, of course. Surrounding them are nearly every elite starter in the game. One’s first suspicion might be that the computers are capping pitcher’s innings, averaging their worst-case, injury-marred seasons in with the more common workhorse editions. But those projected DRA totals are high across the board as well; PECOTA in 2017 projected a mere 13 pitchers to break the 3.50 DRA threshold, three of them starters. (In reality, 85 pitchers ended up managing the task.) This year’s edition is begrudgingly contrite: the total has risen to 35.

That adjustment is a sign of progress. The BP stats team overhauled pitching evaluation over the offseason, and the full details will be conveyed in the near future. One could see the argument that pitching is a fragile business, an anti-actuarial occupation where no one, logically, should survive. But the hard cap that PECOTA appears to place on its pitching would seem to punish the elite pitchers who have thrived year after year. If you disagree, adjust your expectations accordingly.

2017 Pitchers, Overestimated

Perhaps the least interesting list, so it’s a little unfortunate that I saved it for last. Unlike the chart above, this is a collection of mediocre pitchers who happened to implode in 2017; there are always a few. The thing about pitchers is that projecting them is kind of arbitrary; most of them have used all their rope, so even if they earn another opportunity, a bad start will lead to a drastic reduction of innings. Besides, even more than with the hitters, production is simply a product of health; PECOTA can read the tea leaves to guess what haunts the shoulders and elbows of these men, but in reality we’ll all find out together.

There are pitchers who delve deep into the negatives on projected WARP, but most are men already driven to retirement, like Bronson Arroyo and Kyle Kendrick, or low-minors arms still to fix, kids who could only see significant major-league time in a King Ralph situation. Perhaps the most interesting name on this list, by relative comparison, is the uninteresting Kyle Gibson, whose ground-ball tendencies and decent control (if not command) give him a couple outs on the path back to productive innings for the Twins.

At this point I feel like we should all be accustomed to the values and the limitations of projection systems. They’re not fooled by the color of the world: they don’t know who’s changing their swing paths, but they also don’t know or care who’s in the best shape of their lives. They’re enormous shrug machines, designed to tell you what you already know. That seems like a bad thing, but the truth is that it’s actually a service: Most of us need to be told what we already know, because we’re so good at forgetting or ignoring it. Think of your favorite projection system, then, as a pastor delivering a well-worn sermon. The advice it offers may not hold up in your specific set of circumstances, but as a general rule it’s good to follow.

To celebrate the launch of the 2018 PECOTA projections, we’re offering 18% off new subscriptions and extensions of existing subscriptions. To take advantage of the limited-time offer, pick a subscription and enter the discount code “PECOTA18” at checkout.