Flu-Like Symptoms: Taking Stock of 2018 Preseason Predictions

“It’s tough to make predictions, especially about the future.” — Yogi Berra

Everybody makes projections at the beginning of the season. We do, FanGraphs does, ESPN does, Sports Illustrated does. Many others. Maybe you, too. It’s kind of a silly exercise, in that it has no bearing on what happens in the coming season. A projection of who’s going to win the six divisions doesn’t convey information the way an analysis of a pitcher’s repertoire, or a hitter’s platoon differential, or a manager’s bullpen strategy does. But they’re popular, people like them, so we do them.

They have a short shelf life, though. Generally speaking, people don’t look at projections much once they’re made. For player projections, that’s a mistake. Projections from a good model, like our PECOTA, are a better predictor of both hitters’ and pitchers’ rest-of-year performance than their actual statistics until pretty late in the season. (Click on the links, or re-read the sentence, if you’re not familiar with this concept; it’s counterintuitive as all get-out.)

But projections of team won-lost records … who really cares about them, once they’re made? We know that the Braves did surprisingly well, and the Orioles, well, nobody thought they’d be that bad. We don’t need preseason predictions to quantify that.

But rather than ignore them, let’s look back on preseason projections. Who did well? Who didn’t? Not that it means much of anything, unless you’re gambling on baseball, but just the same, it’s something.

Fortunately, the heaviest lifting has already been done for us. For each of the past four seasons, Darius Austin of our fantasy team has posted results at Banished to the Pen, a blog started by fans of the Effectively Wild podcast. Austin evaluated preseason projections of teams’ won-lost records. He used seven sources:

• Baseball Prospectus’ PECOTA
• FanGraphs’ projections based on a mix of the Steamer and ZiPS systems
• Clay Davenport’s projection system
• Banished to the Pen writers
• Guests on Effectively Wild
• A composite of the above five
• Adjustments to PECOTA based on a poll of Effectively Wild listeners

Now, a big caveat: BP, FanGraphs, and Davenport all hew to the discipline of having wins and losses match up. That is, our projections for the 30 teams will sum to 2,430 wins and 2,430 losses (subject to rounding). The other systems don’t, and are often subject to doses of optimism. Austin addressed this issue in 2017.

To analyze the systems, Austin calculated both mean absolute error (MAE)—the difference between projected and actual wins—and root mean squared error (RMSE)—the square root of the average of the squares of the differences. The key difference between the error measures is that RMSE, by squaring the difference, penalizes systems more for large errors (thanks, A’s) than does MAE.

Austin does a nice job of explaining all this and illustrating the results, in sortable tables no less, in this link. If you don’t feel like heading over there, though, here’s the conclusion: The top three prediction systems by MAE were PECOTA, the composite, and Effectively Wild, in that order. The top three by RMSE were exactly the same.

Now, we’re not going to take too much a victory lap, even though PECOTA led the field in 2017 as well, because it didn’t have a good year in 2016. We’re constantly tinkering, though, and for the past couple years, it seems to have served us well. And we’ll take it, given the grief that PECOTA reliably gets every spring.

Everyone on staff at BP was invited to make preseason projections as well. We had 74 writers, statisticians, technologists, and editors answer the bell. We didn’t have to project win totals, only list projected order of finish. That kept things pretty straightforward.

I used absolute error and squared error to evaluate our projections. For example, here’s how I did. I’ll list teams in the order I had them before the season:

• AL West: Houston (off by 0), Los Angeles (2), Seattle (0), Oakland (2), Texas (0)
• AL Central: Cleveland (0), Minnesota (0), Chicago (1), Kansas City (1), Detroit (2)
• AL East: New York (1), Boston (1), Toronto (1), Baltimore (1), Tampa Bay (2)
• NL West: Los Angeles (0), Arizona (1), San Francisco (1), Colorado (2), San Diego (0)
• NL Central: Chicago (1), St. Louis (1), Milwaukee (2). Pittsburgh (0), Cincinnati (0)
• NL East: Washington (1), Philadelphia (1), New York (1), Atlanta (3), Miami (0)

That’s a total of 28 absolute error and 46 squared error, both of which were worse than average. Oh well.

Here’s how everyone on staff did, ranked by smallest to largest error:

Absolute	Staff
18	Gregory J. Matthews, Keith Rader, Wilson Karaman
20	Andrew Salzman, Scooter Hotz, Stephen Reichert
22	Eddy Almaguer, Frank Firke, Jason Wojciechowski, Joshua Howsam, Mark Barry, Martin Nolan, Nicholas Zettel, Scott Delp
24	Anthony Rescan, Brian Duricy , David Brown, David Lesky, Jeffrey Paternostro, Jon Hegglund, Nathaniel Greabe, Victor Filoromo
26	Aaron Gleeman, Andrew Gargano, Ashley Varela, Bryan Grosnick, Craig Goldstein, Darin Watson, Derek Florko, Jacob Devereaux, Kevin Carter, Lance Brozdowski, Mark Anderson, Martin Alonso, Matthew Trueblood, Nick Stellini, Stacey Gotsulias, Tommy Meyers, Zachary Moser
28	Ben Carsley, Brett Cowett, Clinton Scoles, Colin Anderle, Darius Austin, Demetrius Bell, George Bissell, Hunter Samuels, Jay Markle, Jeff Euston, Matt Collins, Nathan Graham, Nick Schaefer, Patrick Dubuque, Paul Noonan, Rob Mains, Rob McQuown, Tim Collins, Zach Crizer, Zach Steinhorn
30	Ben Murphy, Bret Sayre, Collin Whitchurch, Ken Schultz, Kevin Jebens, Matt Dennewitz, Matt Sussman, Tyler Maher
32	Kazuto Yamazaki, Michael Engel, Mike Gianella, Seth Victor
34	Alexis Collins, Jarrett Seidler, Sean O’Rourke

Squared	Staff
24	Wilson Karaman
26	Gregory J. Matthews, Keith Rader
28	Scooter Hotz
32	Eddy Almaguer, Frank Firke, Jason Wojciechowski, Mark Barry, Scott Delp
34	Andrew Salzman, David Brown, Joshua Howsam
36	Ashley Varela, David Lesky, Jon Hegglund, Martin Nolan, Stephen Reichert
38	Zachary Moser
40	Anthony Rescan, Jeffrey Paternostro, Mark Anderson, Matthew Trueblood, Nathaniel Greabe, Nick Stellini, Stacey Gotsulias
42	Andrew Gargano, Brian Duricy , Derek Florko, Nicholas Zettel, Tommy Meyers
44	Aaron Gleeman, Brett Cowett, Craig Goldstein, Jacob Devereaux, Kevin Carter, Lance Brozdowski, Martin Alonso, Nathan Graham, Nick Schaefer, Paul Noonan, Zach Crizer
46	George Bissell, Kevin Jebens, Patrick Dubuque, Rob Mains
48	Clinton Scoles, Darin Watson, Demetrius Bell, Hunter Samuels, Matt Dennewitz, Rob McQuown, Victor Filoromo
50	Ben Carsley, Bryan Grosnick, Colin Anderle, Collin Whitchurch, Darius Austin, Jay Markle, Jeff Euston, Ken Schultz, Matt Sussman, Tim Collins, Zach Steinhorn
52	Bret Sayre, Tyler Maher
54	Kazuto Yamazaki, Matt Collins, Mike Gianella, Seth Victor
56	Ben Murphy
60	Jarrett Seidler, Michael Engel
62	Alexis Collins
64	Sean O’Rourke

Matt Sussman was the only one of us to predict a Red Sox World Series title.

For whatever it’s worth, the staff did a lot better in 2018 than in 2017. The average absolute error was 26 compared to 33 last year. Maybe we went through a rigorous weeding-out process, adding new personnel with superior predictive skills. Or maybe 2018 just played pretty close to form. You can draw your own conclusions.

We also predicted the award winners. There were two surprise winners: Nobody had Blake Snell winning the American League Cy Young (Wilson Karaman had him in third) and nobody had Christian Yelich winning the National League MVP (Stephen Reichert had him in third). Only two people predicted a Mookie Betts MVP (Joshua Howsam and Gregg Matthews), though 20 people had him in the top three. Only Collin Whitchurch, Jeffrey Paternostro, and Matthew Trueblood expected a Jacob deGrom Cy Young, and 12 had him in the top three.

On the other hand, everybody (31 voters) expected Shohei Ohtani to be the AL Rookie of the Year. Everybody and their mother (49) saw Ronald Acuna winning the NL ROY.

Thanks to Rob McQuown for putting this year’s predictions into an easy-to-sort spreadsheet.