The Quinton: Early Misses, Future Misses and Analyzing our Process

Baseball Prospectus colleague George Bissell tweeted the following on 4/15/2017 regarding the high price of acquisition for James Paxton in drafts and auctions this year:

This got me thinking: Was not paying up for James Paxton a mistake? In terms of results, it certainly seems so—the money at the auction I spent on Danny Salazar and Joe Musgrove would have been better spent on Paxton and another pitcher. Similarly, money spent in my NL-only auction on Lucas Duda and Josh Harrison would have been better used on Eric Thames and literally any middle infielder.

Mistakes were made. But here’s the thing: A lot of mistakes were made by me in this particular auction—if we are defining a mistake as "spending money in any way that does not lead to the best result (or a better result)." Moreover, if we are defining "mistake" in this way, then we make mistakes with nearly every decision we make in any given draft or auction, because there is almost always someone better we could have selected. Beating up ourselves over not having a perfect draft or auction is thus, at best, a waste of time and, at worst, a potential cause for poor future decision making as we attempt to right our so-called wrongs.

I made this line of argument because the point is worth noting, but it is disingenuous to the original topic because Paxton and Thames are not just misses, they are a specific type of miss; that they were missed because of their perceived risk profiles. In other words, those of us who missed with Paxton and Thames did not believe the risk to be as worthy of the reward as the teams that added these players. When put this way, we realize that we might not have missed on these players at all. There always are going to be players that hit their near-highest (and near-lowest) projected outcomes, and thinking that we will be able to predict each one of them is a fool’s errand.

Why is it a fool’s errand? Isn’t player evaluation at the core of fantasy baseball?

While player evaluation is at the core of fantasy baseball, informing and/or changing our process because of individual outcomes is probably more likely to be harmful than it is to be helpful. Why? Because a player’s output in a given season, let alone a given month, is a single data point. Why is that an issue? Because a single data point (or any small sample) essentially begs us to believe what we want to believe, and thus allows us to be at the whim of many cognitive biases. In this instance, we either are likely to seek confirmation that our process was sound, or look to reverse our process in hopes of correcting our mistake—both of which are more about making ourselves feel better than actually improving our decision making in the future.

So, now that we have taken over 450 words to reach a basic statistical concept, should we ignore our misses and trust our process? No. What we should be doing, instead, is using these misses to inform and create hypotheses about possible flaws in our decision-making process. For example, missing on Thames and Paxton should lead us to ask if we are building in too much risk when evaluating and ranking players with higher degrees of uncertainty or larger ranges of possible outcomes. To test such hypotheses, though, we need to look at how our valuations and ranking performed on a larger sample (for example, ten of such players from each of the past three seasons), and see how they played out.

Obviously, evaluating a larger sample when analyzing our process gives us a better chance at discovering bias or flaw in our process than does looking at our process on a single player. But there is another advantage to be had in using this process for analyzing process (as opposed to reviewing on a player-by-player basis)—which is the mental and emotional relief.

What? Hang with me here, please. Whether it is rational or not, being wrong takes a mental toll on us and can negatively impact decision making in the future. When combined with how most of us now consume news and information (Twitter, the internet, our phones), the immediacy and frequency with which we are reminded of our misses on breakout stars (every other article and every fifth non-wrestling and non-food opinion tweet now seems to be about Paxton or Thames) is likely at an all-time high. The result of this is that we are more likely to make decisions that make us feel better about these misses, as opposed to making decisions that will give us the best odds for success going forward.

This is all to say that performing a more-rigorous analysis of our misses as discussed above will not only improve our decision making come the next draft or auction by informing us if we need to adjust our valuation process, but it also will improve our immediate decision making by making us less likely to overreact to misses. Doing so will make us less likely to prematurely drop the disappointing player we selected instead of the breakout star because we will have either determined the miss to be either (i) bad luck or (ii) poor process, but one that we feel we have improved for the future—as opposed to feeling as if we need to make an immediate “improvement” in order to feel better about the miss. So when the next great Paxton start or the next Thames home run makes us feel bad about missing on them, let us endeavor to use that feeling to check our process and see if it needs to be improved.