Moonshot: Another Look at Prospect Values

A recurring question this offseason has been about the value of top prospects. We have seen San Diego accomplish a significant rebuild without trading away much of the top of their farm system, soon after Billy Beane was compelled to send BP’s no. 2 rated prospect, Addison Russell, to Chicago for a one-year rental of Jeff Samardzija. Meanwhile, the Cole Hamels saga is ongoing in Philadelphia, with Ruben Amaro, Jr. coyly refusing to ship Hamels for less than a handful of top young talents (or so the reports claim).

These scenarios suggest that the value of prospects is a complex and contentious topic, even within baseball’s front offices. Admittedly, I am not the first and will not be the last to approach this question. But the quandary of prospect valuation is worth reexamination both because of its complexity and importance.

Others have approached the topic from the standpoint of minor-league statistics, and found some ability to project future success with that information. That’s PECOTA’s beat, though, and any way, previous research has shown that skilled talent evaluators are able to outperform minor-league-based projections. As a way of harnessing the scout’s eyes, I borrowed Chris St. John’s prospect rankings database, which covers various top 100 lists going all the way back to 1990. St. John’s collection aggregates from a wide variety of sources, including BP and Baseball America.

The most straightforward way to approach the question of prospect valuation is to build the following simple graph, which charts the average rank of a position player prospect against their lifetime WARP. This method takes the scouts at their word, and simply asks how well their rankings reflect the eventual value of a player.

The shape of the curve is similar to what’s been previously reported, but the magnitude or depth of it is very different. I find that a top-5 hitting prospect is worth on average, about 15 WARP over the course of his playing career. That number comes with a massive variance, of course, ranging all the way from negative WARP to more than 100. But, importantly, the average outcome is substantially higher than lower-ranked players.

I will note here that other studies have found lower numbers for the value of top-ranked prospects. There are a couple of ways my study is different. One, I am using a slightly different timeframe (1990-2010[1], although this would tend to decrease the relative value). Two, for now I am excluding pitchers, who may pollute the overall rankings with their injury issues and greater uncertainty (TINSTAAPP). Three, I take the average of all of a player’s appearances, across years and different lists. I find that the average is a better way of examining prospect worth, because it exploits variation between lists as well as variation across years to get a clearer picture of the consensus on each prospect. A guy who is rated very highly for one year is less likely to succeed than one who consistently occupies the top ten for two or three years in a row.

Adding additional lists proved surprisingly unhelpful. In the above graph, I show the same curves but generated from the lists of only one publication each: Baseball Prospectus, in blue, and Baseball America, in red. As you can see, they mirror the consensus, in black, with minimal differences. If anything, the addition of more lists tends to dampen the success rate of lower-ranked prospects.

Other Factors
To put my own spin on the subject, I also wanted to look at a couple of other factors that might shape prospect valuation. One was prompted by the observation that a number of young hitters managed to repeat across this year and last (Byron Buxton, Miguel Sano, Carlos Correa, etc.). I wondered if there was any link between the number of times a prospect appears on top 100s and their eventual performance. From the start, this line of inquiry seemed like it must result in a Goldilocks-type answer, where extremes on either end are detrimental, with a sweet spot in the middle.

It turns out that two years is the sweet spot. Prospects who go more than two years on the lists tend to gradually fall off toward lower contributions, and prospects who appear on only a single year’s list are usually not terribly good prospects to begin with. So, while Buxton is safe, Sano, who has appeared on five consecutive years of lists, is probably beginning to tip toward a worrisome fate.

Another factor of interest is age. The younger a prospect is, the better off he is in terms of lifetime WARP production, all else being equal. In theory, age is already baked into some of these prospect lists (depending on the individual philosophies of the evaluators who made each list). However, my results show that a highly rated prospect at 19 is a better bet for lifetime WARP production than one at 26. I don’t think that should surprise anyone.

Finally, we can look at the position a player fields. Generally, we should expect that players who field more challenging positions will do better in the long term. Since athletic ability is tied to position, and athletic ability tends to diminish, not increase, over time, players who start from loftier positions on the defensive spectrum have more spots on the diamond to fall down to as they age and become worse fielders.

As Sam Miller has noted, a surprisingly large portion of the players currently in MLB played at least some shortstop in the minors—even a few nearly immobile guys (e.g. Mike Morse) who we do not think of as defensive wizards logged some time at SS. Their initial competence may have given them the room to decline into becoming merely poor defenders in MLB, as opposed to unacceptably terrible.

The data supports this notion, with some caveats. Since many players try out multiple positions in the minors, there is no perfectly clean way to divide them. One simple partition to make is based on whether a player ever played shortstop. Those who did averaged 10.6 WARP (this is heavily inflated by the tail of players who had superlative careers). Those who didn’t averaged 9.3 WARP, which is a bit less. Similarly, the median WARP of non-shortstops is 2.6, vs. shortstops at 3.7.

Admittedly, this is kind of a clumsy way to parse the data, because it tells us nothing about their performance in the time that they played shortstop. They could have been terrible, and switched immediately to first base, for instance, which doesn’t say much positive about their fielding ability. Nevertheless, that someone even attempted to put them there suggests that they could run at least a little bit, and bodes well for their future athleticism as they slide downwards towards DH with age. There was a similar effect, albeit slightly muted, for catchers, as well.

If we put all of these factors together into a giant regression[2], and then attempt to predict the WARP of each player, we get a graph that looks like this:

There is a bit of correlation, but it isn’t terribly reliable (R²=.18). This isn’t a sterling R² number. But we shouldn’t have expected it to be. Forecasting the future production of 18- to 22-year-olds decades in advance is no easy task. Even leveraging teams of trained professionals or heartless but brilliant machines, we can scarcely make headway on this difficult problem.

Indeed, the lack of predictability suggests that teams will be disagreeing about the value of prospects as long as there are prospects. In the presence of so much uncertainty, a huckster can argue that his asset is a Hall of Famer while his adversary can wave Delmon Young’s impressive prospect credentials around instead. The truth is that while highly rated prospects are likely, on average, to make substantial contributions to their teams, any individual prospect could be worth as little to the team’s success as a quad-A replacement.