Baseball Therapy: Can Draft Lightning Be Bottled?

It was a boring weekend for baseball rumors on the major league side, because all of the front-office sources that your favorite columnist goes to were busy with the draft. Maybe you kept yourself entertained by getting to know the names that will be gracing your favorite team’s roster five years from now, and hopefully, that knowledge will come in handy. Unless those guys flame out in Double-A. That happens too.

Of course, in five years, some of those front-office sources your chosen columnist relies on will no longer be working for the same teams, maybe because too many promising players did flame out in Double-A, and their successors will have to live with the results of those drafts. Then again, the flip side is that there are plenty of people in front offices right now who are feasting on the fruits of previous administrations who didn’t survive long enough to see them ripen. Baseball is a cruel game.

At this point, just about every team swears that they are a “draft and develop” team (or some variant of that). It’s for good reason. Players who are drafted (or signed internationally) come with six years of cost-control, and that value generally comes at about half the price of wins on the free agent market. In fact, I’ve previously estimated that the average player development system, league-wide, is worth about 10 wins of value—and that’s just the average. Having a good drafting and development system can be the difference between a playoff appearance and a participation trophy.

However, when I looked back at the drafts of 2003-2008 a couple weeks ago, I found that the league as a whole showed only moderate ability to pick out which players would end up being the good ones (or even the minimally useful ones) in the first round of the draft. By the second round, they were doing no better than chance. The “draft” part of “draft and develop” is a bit random, or perhaps the “develop” part is much more important than we had initially thought. It seems unfair that what can be something of a grab bag has such a big influence on a team’s chances. In 2006, a bunch of smart people had pitchers Luke Hochevar, Greg Reynolds, Brad Lincoln, Brandon Morrow, Andrew Miller, and Clayton Kershaw all bunched together in their rankings before the draft. They were picked in that order. We now know how well that turned out. Of course, the team that drafts the best—or maybe we should say the team that gets the luckiest in the draft—is going to have a big advantage. So, how good (or lucky) in the draft were teams during that six-year period?

Warning! Gory Mathematical Details Ahead!
Again, we’re going to look at the drafts from 2003-2008, because those players have had a chance to mature and make their mark in the big leagues. We have signing bonus data from BP, and career WAR totals from Baseball Reference.

First, I looked at how efficient teams were in spending their money. For example, we can look at the total percentage of money that teams spent in signing bonuses on players who did make it to the majors versus those who didn’t. Looking at the raw percentages of players who made it versus those who didn’t is misleading. After all, if a team had the second pick in the draft (and picked a player who likely got a bigger signing bonus) they should have higher hopes for him than the player picked 22nd. We shouldn’t penalize teams for having low picks and being stuck with whichever player happened to fall to them. Nor should we give them too much credit for the $1000 bonus guy who somehow made it to the bigs. The team didn’t think too much of him back then, and they got a little lucky.

Here are the five best and five worst organizations for the percentage of 2003-2008 draft bonuses spent on players who made the majors:

It’s one thing to get to the majors. It’s another to do something useful while you’re there. Let’s repeat the same basic analysis, now looking at which teams spent the highest proportion of their bonuses on players who contributed at least five career WAR.

It looks like the Diamondbacks were efficient in spending their draft money last decade, although there’s not a lot of crossover on these lists. Some teams were good at producing warm bodies to fill out a roster. Some were good at spending their money on players who made a sizeable contribution. Of course, these figured can be skewed by, say, a no. 1 pick (Justin Upton?) who gets a big bonus and makes it, while the team doesn’t spend a lot for the rest of the time. But this gives us a decent idea of who was thrifty with their money.

Now let’s look at the process through a different lens. If a team brings one player to the majors, there are still 24 other roster spots to fill, so there’s a big benefit to bulk players who can make a contribution. Again, we should compare a team’s performance only to the raw materials that it’s been given, but the team that’s getting the best deals is the one that’s beating the market for developing quality players based on their spending.

I constructed a logistic regression that used signing bonus (this time, expressed as a percentage of the overall amount of money spent in that year’s draft—a player who received a $1 million bonus in a draft where $100 million was spent received one percent of the bonus pool) as a predictor of whether or not the player made it to the majors. We might expect a player with a bigger signing bonus to have a better chance to make it. So, we would expect the teams who are handing out bigger signing bonuses (often as a result of drafting higher up) to produce more big leaguers out of their drafts. If they produce even more than expected, maybe they’re doing something right. Or maybe they’re just lucky.

Here are the top five and bottom five organizations (again, from the 2003-2008 drafts) ranked by the ratio of actual major leaguers that their drafts produced versus their predicted number.

And again, we run the same analyses, this time looking at players who’ve been worth at least five career WAR.

I guess it’s safe to say that the Indians didn’t do so well in the draft between 2003-2008. What’s interesting is that again, we don’t see repeat entries in the top five for teams who produce warm bodies versus those who produce five-WAR players. The Cardinals and Giants are famous for stocking their recent championship teams with homegrown players, but it seems that the team that’s been best at producing quality players has been the Braves, followed by the Red Sox.

Finally, now that we know who has been good at getting the most out of the draft, let’s see how reliable those numbers are. I looked at all draft picks, again, from 2003-2008 for each team, lined up in chronological order. I looked to see how many draft picks it would take until the (raw, unadjusted) rate at which teams produced players who would eventually make it to the majors stabilized. (For the super-initiated, I used the Kuder-Richardson formula for that one.) I also used the adjusted rates above to determine, based on signing bonus, a team’s chances of developing a major leaguer, giving them credit (or blame) for each player based on that predicted chance.

For example, if a team picked at the high end of the first round (and dished out a big bonus), we expect greater things from that pick. If we might expect him to have an 80 percent chance of making it to The Show, the team gets only 20 percent credit for finishing the job. I found the split-half reliability for that one, too (using Cronbach’s alpha). Now, we have a sample size of only about 60 picks per team to work with, meaning that for a split-half method like Cronbach, we’re going to be able to test sample sizes of only 30 picks. However, entire front offices are fired because of a three-year stretch of poor draft results.

The short version is that the values were so low that they didn’t register. The same happened for the rate, both unadjusted and adjusted, at which teams produced players who achieved five career WAR. I should point out that while a team might have a new GM and scouting director from year to year, and that the team that drafts a player isn’t always doing the developing, the numbers were so low that it probably doesn’t make a difference.

Don’t Let Your Babies Grow Up to Be GMs
Now that the draft is over, you’ve probably already read plenty of reaction pieces. It makes sense, because cost-controlled talent might be the most important resource in the game, and this is the big chance that teams have to grab some. Everyone wants it, which of course makes finding it all that much more difficult. Then again, there’s real evidence to suggest that teams don’t have a good grasp on knowing it when they see it.

We can look back and see that some teams did very well in the draft in the past 5-10 years and that some of that talent is now stocking their roster. This is a wonderful thing for them, but much like a couple weeks ago, when I asked whether we should give out “genius” tags to hitting or pitching coaches (and the answer was a resounding “no”), we should also be wary of tagging teams as geniuses in the draft. Aside from the fact that the groups of people that were running a team’s draft in 2003 and in 2008 could have been completely different, we should still be wary of judging a GM and his scouting staff on the basis of two or three drafts. Yes, things might have gone well (or badly), and maybe they’ll go similarly well (or badly) the next time around, but past performance here is not indicative of future results. Even if 2014 turns out to be the best draft ever for your favorite team, you can’t call your GM a genius based only on that.

I want to shy away from the idea that the draft is a complete and total crapshoot in which there’s no sense in even trying. There probably are ways to figure out who’s going to be a good player down the road, or at least to improve the odds of guessing. Teams are coming up against the reality that they are being asked to predict what a player will be doing in 10 years. That’s hard for anyone to do. (Take a look back through your high school yearbook if you don’t believe me.) There may also be the problem that most of the ways that there are to figure out who’s a good player are common knowledge. It’s hard to keep a secret in baseball. If a kid has a realistic possibility for a big league career, chances are that everyone already knows about him. Maybe everyone’s playing out of the same playbook, in which case it would be hard to gain an edge. There’s also the possibility that in the past six years, teams have developed new capabilities to sort out the wheat from the chaff in the draft and that those results won’t become obvious for a little while.

It also points out an—Interesting? Unfortunate? Wonderful?—property of baseball. There’s a lot of dumb luck that goes into whether a team succeeds or fails. The unfortunate part is that people get fired because of it. Note that well, readers who want to get a job “on the inside.” Your job will be at the mercy of the possibility of a couple of bad drafts. Then again, if your talent is for figuring out a way to improve on a team’s hit rate in the draft and development process, then you will be most welcome in the industry. Just understand that it’s not all that easy. Baseball is a cruel game.