Pebble Hunting: Are GMs Smart or Not Smart?

The answer is they are smart. Come on guys, be serious. Of course they're smart. 

But…

Once a year, ESPN’s Jerry Crasnick polls baseball executives on Hot Stove topics. This year Crasnick surveyed “22 general managers, assistant GMs, advisers, scouting directors and talent evaluators in the field for their opinions on seven questions that are likely to drive media coverage,” and the results are characteristically fun: Josh Hamilton to the Brewers! One year and $2 million for Melky! Many other things!

It’s one of my favorite pieces each offseason, and I hope it goes on forever, or at least until I die, because it makes me sad to think that the world will keep going when I’m gone. The hardest part of writing about baseball is the realization that, by not being in the room, I’m getting maybe two percent of the relevant information, which is probably canceled out by the flood of misinformation that does make it to me by virtue of being false and thus having nobody trying to protect it from the public. The guys Crasnick talks to are in the room, and they’re friends with other people in other rooms, and because they’re anonymous there are no obvious incentives for them to lie, so it’s fascinating to read what they believe to be true.

But “believe to be true” is still not necessarily true, and I wondered how much extra information these guys really have. Crasnick has been doing this long enough that a pretty good archive of these pieces is available. Thanks to all those phone calls Crasnick made, we can now review the executives’ predictions from 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, and 2011.

I put every prediction into a spreadsheet, a simplified version of which appears below, a more detailed version of which you can see here. I limited it to predictions that are specific and not vague. For instance, I deemed “Which contract was the worst signed so far” not specific enough, but “Which of (five players listed) will be best next year” specific enough. With one exception, I ignored executives' salary predictions, because of the wide range of responses and because the multivariable aspect of contracts (length + AAV) makes it hard to evaluate each prediction's correctness.

The last column splits the predictions into two categories: 1 for predicting player movement; 2 for predicting player performance. We'll get to that later.

So that gives us one prediction (the Bonds prediction from 2007) that you could argue was right or you could argue was wrong. Of the other 45, there are 21 correct predictions and 24 incorrect predictions. Worse than coin flips!

(Quick aside: this exercise may remind you of Ben Lindbergh's piece last month on predictions that managers, executives, and players made for themselves before the season. It is similar, with two significant differences that I believe advance the discussion. One is that the predictions Ben collected were made publicly and on the record, so the predictors had incentive to lie, spin, or boast. The other is that Ben's piece had zero tables and my piece has two tables.)

Except that most of these aren’t either/or questions; there’s not really a one-in-two chance of picking which team will sign Aroldis Chapman, when there are 30 potential answers. There’s not really a one-in-30 chance, either, as not every team was linked to Chapman, had scouted Chapman, etc. So it’s a bit trickier to compare the executives to random chance.

But if we split these up into categories by how many options there are, then we get a bit more insight:

The remaining 15 have an unknown number of true options. But of the questions that have a finite number of possible answers, the executives’ success rate is just a shade better than random chance: 52 percent rather than 50 percent when there were two possible answers; 30 percent instead of the 26 percent random expectation on the three-to-six-answer questions.

Surprisingly, to me, the executives don’t seem to do any better or worse when they’re predicting player performance than when they’re predicting transactions. They were 47 percent successful when asked which player or team would be best; they were 46 percent successful when asked where a player would sign, whether he would be traded, of it he would return the next season.

Obviously I’m going to include a few of the most excellent excerpts. Three:

• "I saw (Kaz) Matsui four years ago, and I thought he was a better player than Ichiro," said an AL GM. (2003)
• "If the Yankees were able to lock Lackey into a rotation with Sabathia and Burnett, it could be the start of another dynasty," a National League assistant GM said. (2009)
• Which player is more difficult to trade — Cubs outfielder Milton Bradley, because of his reputation and other baggage, or Blue Jays outfielder Vernon Wells, because of his contract? Responses: Wells 20, Bradley 0. (2009. Bradley was traded that offseason in a bad-contract swap. Wells was traded for value a year later.)

I think there are two possible ways of explaining this low success rate. One is based on the work of Philip Tetlock, who spent 20 years studying thousands and thousands of pundit predictions. He concluded that expert pundits are barely more predictive than random chance. 

First, as the skeptics warned, when hordes of pundits are jostling for the limelight, many are tempted to claim that they know more than they do. Boom and doom pundits are the most reliable over-claimers.

There's a difference here, in that the "experts" we're evaluating aren't in the limelight and have no obvious incentive to over-claim. But it is possible that, because they self-identify as experts, they naturally view and present themselves as more certain than they should. It might also be the case that it is more satisfying to have an interesting opinion (Kaz Matsui > Ichiro; Alex Rodriguez most likely to sign with the Marlins; Melky Cabrera to sign for one year and $2 million) than a conservative but accurate opinion. 

The other is that baseball falls under the efficient-market hypothesis: 

In finance, the efficient-market hypothesis (EMH) asserts that financial markets are "informationally efficient". In consequence of this, one cannot consistently achieve returns in excess of average market returns on a risk-adjusted basis, given the information available at the time the investment is made.

Crasnick is asking these questions specifically because they are perceived, based on publicly available information, to be difficult to answer with much confidence. He is asking GMs because they are perceived to have more than the publicly available information. But it might just be the case that, when it comes to evaluating major-league players and predicting offseason moves, most of the relevant data is already publicly available and priced into our expectations; that the relevant data that isn't publicly available is not shared between teams; and that, when it is shared between teams, it tends to trickle out to the public, via excellent reporters like Crasnick. In other words: for the purposes of these two types of questions, there's not a significant advantage to being in the room. 

Like I said at the beginning, of course GMs are smart. A winning organization depends on so many skills in the front office: contract negotiating, strategizing, scouting for weaknesses, managing risk, motivating players, motivating non-players, generating revenue, working with other teams' front offices, keeping players healthy, developing young players, collecting information, measuring and assessing performance accurately, approving timecards. GMs are, presumably, smart at those things, and many others.

But predicting baseball might just be impossible, and a team that puts too much faith in its own predictions might be falling into a trap. So don't feel bad if you're bad at it. It's not your fault. It's baseball's fault.