What do you know about what the baseball draft produces? Most fans probably have some sense of the history of their favorite team’s high profile draft picks. For instance, I’m a Phillies fan, and most Phillies fans know that many of their current stars, such as Cole Hamels, Chase Utley, and Jimmy Rollins, are the result of the team’s drafts. They also probably know enough to lament draft picks such as Jeff Jackson (picked ahead of Frank Thomas in 1989) and Ryne Sandberg (drafted in 1978 but then traded for Ivan DeJesus after only 13 games with the Phils). Fans might also know some random tidbit about other teams, such as the Mariners‘ successes with Ken Griffey Jr. and Alex Rodriguez or the Dodgers‘ drafting Mike Piazza 1,390th in the 1988 draft and getting a Hall of Famer. But beyond that, do you know much about what the draft as a whole produces?
Because I fall into the same boat described above, I have been planning on researching the draft and the minor leagues for some time. Given that this week’s topic is “Below the Big Leagues”, now is as good a time as any to start.
Without much readily available data, I researched the first two rounds of the last 31 amateur player drafts, compiled attributes that I felt were important, and then asked a series of questions that I found interesting.
Like the start of any good research project, this initial analysis will create more questions than answers, but will serve to jumpstart future work. What’s below is a discussion of some basic trends I’ve learned about the players drafted in the first two rounds since 1978.
WHO MAKES THE MAJORS?
The first question I wanted to ask was how frequently highly drafted players actually make it to majors. The majority of amateur players drafted will never set foot in a major league dugout. However, aren’t first and second round picks different?
In total, 51% of first and second rounds picks make the majors. Consider the following graph below. The blue line represents the percentage of first round draft picks each year that eventually made the majors (played at least one game), and the pink line represents the same for second round picks.
Two things are pretty clear to me from this data. One is that first round selections do make the majors more than second round selections. It would be fair to say that to some extent teams do know what they are doing with these draft picks!
The other thing that is clear is that fewer and fewer first round draft picks ever end up making the major leagues. Does that mean that teams are getting worse at identifying talent? I don’t think so. Instead, the emergence of international players has grown rapidly during this time frame, resulting in more major league spots being taken up by non-drafted players. Interestingly, the trend is only downward for first round picks and not for second round picks. It would be worth checking how this pattern holds for later rounds.
WHO IS SUCCESSFUL IN THE MAJORS?
Next I went through the number of picks who achieved some level of success in the majors, rather than simply debuting at some point and fading away. I set up a pretty minimal definition of success-accumulating a WARP3 of 10.0 in their careers. Think Travis Lee (10.1) and Bruce Chen (9.1)-players who got some regular playing time over an extended period. I went through each set of five draft picks (picks 1-5, 6-10, etc.) and tabulated how many made the majors and how many had a WARP3 of 10.0 or above. Here is that chart:
Picks: Made MLB WARP3>10 1-5 122 62 6-10 105 43 11-15 88 31 16-20 97 34 21-25 97 25 26-30 75 15 31-35 86 20 36-40 62 17 41-45 64 7 46-50 71 18
It does seem like teams are pretty good at identifying the top players in the draft. In fact, 28 of the last 31 first overall picks have made the majors, and 19 have WARP3s of at least 10.0. After that, while players with better chances seem to be drafted earlier, there is not a huge difference in between getting the 11th and 25th pick.
DID MONEYBALL HAVE AN EFFECT ON THE DRAFT?
Few books have changed baseball the way that Michael Lewis’ “Moneyball” changed the game in 2003. In this book, Lewis documents the success of the Oakland A’s, as they used economic theory to win with a small payroll. It has been well documented that teams put more value on OBP in a post-Moneyball world. However, another thing that Lewis discussed in detail was that in the draft, the A’s targeted college players more than high school players, since college players had a higher chance of making the majors. What I found most interesting of all of my early results is the following: since 2003, there has been a sharp drop in the rate at which high school players are drafted in the first two rounds.
From 1978-2002, teams drafted players out of high school in 51.3% of their first two rounds of picks. Suddenly, in 2003, only 40.3% of players drafted in the first two rounds came out of high school, and from 2003-2008, only 41.5% of players drafted in the first two rounds came directly from high school.
There is no way to prove that this change was because of Moneyball. However, statisticians approach this problem by doing what is called a t-test. A t-test asks the following question: what is the probability that we would see such a significant change if this was nothing other than good old randomness? The answer to this question here? 0.003%! The chart below plots the percentage of players drafted in the first two rounds that were high school students in each set of six years from 1979-1984 up to 2003-2008.
If something did change in MLB front offices as a result of Moneyball, there was good reason-from 1978 to 2008, 59% of players drafted in the first two rounds out of college made the majors, and only 42% of high school players.
The counter-argument to the Moneyball idea of drafting college players is that high school players have higher upside and are more likely to succeed while in the majors. But research shows that’s not true for the first two rounds. Of the college players drafted in the first two rounds, 17% of them reached a career WARP3 of 10.0, and only 12% of high school players did. It seems pretty clear that the A’s must have known what they were doing.
HOW OFTEN DO TEAMS TRADE AWAY MINOR LEAGUE SUPERSTARS-TO-BE?
In 2008, Phillies pitcher Joe Blanton became the second pitcher since 1977 who was traded midseason and won a World Series game. (Can anybody name the first?) For all of the rampant trade rumors and discussions about making a move to get over the hump, it seems like the ultimate goal of those moves is rarely achieved. Of course, the Blanton example is a limited one, but it paints a pretty clear picture of how rarely midseason moves work out for contenders.
One might take that to mean that contending teams are recklessly trading star prospects midseason for supposed difference makers who never materialize. Trades like the famous Mets‘ surrender of Scott Kazmir for Victor Zambrano stick out in our memories, but I decided to take a methodological look at how often teams actually made this mistake. It turns out that it is nearly as rare as those difference makers making a difference.
Of the 2052 players in the study, 1041 of them made the majors. Of those, only 109 players were traded and then debuted with a different team than the one that had drafted them. Of that group of 109, only 19 accumulated a WARP3 of 10.0 in their careers. As it turns out, for all the fans who scream at GMs for trading away the farm system, rarely do the GMs trade away impact prospects.
FURTHER THOUGHTS
This is obviously just a starting point. It takes a long time to seriously look at a topic such as this (and easily longer to actually gather data!), but this is a good start. The interplay between the emergence of international stars and the success of early draft picks is certainly worth looking into further, and I will continue to study this. Looking at subsequent picks after the first two rounds will be interesting as well. Furthermore, it will be interesting to see if teams are more successful at drafting high schoolers now that they are more selective in doing so. That will take some time to study, but the information will eventually be available as the recent crop of high school draftees grow up. Naturally, the value of a trade of a star for a prospect is one of the hottest issues in the majors today. It is not useful to look at anecdotes to condemn or exalt a trade, but looking at this type of information more clearly will help determine where the trade market goes.
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.
Subscribe now
Except that a couple of his conclusions seemed a bit off. I immediately asked Rany Jazayerli, who's seminal work on the draft just a few years ago, seemed to contradict this in places. I asked Rany, who said "I wouldn't say he contradicts my series - the main finding of my work was that the high school/college pendulum swung the other way from 1992 to 1999, but since he's looking at the data all the way back to 1978, his conclusions that college players make for better draft picks are valid."
I wonder whether Matt has seen Rany's pieces and whether he really re-invented the wheel here. Part of that is the fault of BP and the sabermetric community at large, in really not having one central comparative database of the work out there. Regardless, this is another solid piece and Matt's not letting off the gas after lapping the field last week.
Nonetheless, solid research, really well thought out, and well executed.
From among his elements, I especially liked the second topic (the "how good?" bit), and I had a problem with the brief bit on so-called Moneyball-inspired draft strategy, since there isn't any reference to the economic factors that come into play. But then he comes up with a compelling, substantive bit of info in his Blanton-related point about the quality that teams deal away.
In short, while I was disappointed that he went the grab-bag route, it's clear he's got plenty of goodies to dig out and develop. I would love to see more substantive treatments of any of these featurettes.
However, the piece seems more like a series of notes - like a draft - than a polished piece of B-Pro quality. I realize that that's because he's trying to tackle a huge topic. But he might have focused the piece a bit better.
Oh, by the way, best article I've read so far. As always Matt is one of the best, and is emerging as the favorite, IMO.
So trading for an ace may not get you a World Series this year, but if you expect to keep him around it may not be the worst of ideas. In other words, trading for Jake Peavey could very well be a really good idea, especially if all the Padres are looking for is a package similar to what the White Sox were going to give up.
That said, I haven't read all of the other pieces as of yet, and this may still prove to be one of the best this week.
I had a similar thought to others when reading this that it seemed similar to Rany's work. You're looking at a different sample and reached different conclusions, so I thought it was a good article, but any follow up would definitely benefit from reading Rany's series if you have access to it.
I broke the data into 6-year groups only for the graph. Otherwise, it would have looked too messy due to year-to-year variance.
I recognize that WARP of 10 seems arbitrary, but really, I needed a number that a large enough sample of players achieved, something I could get quickly in the time frame I had to generate the data set, and I made an assumption that is probably reasonable-- if the percentage difference in rate of making the majors at all for college vs high school is the same as the percentage difference in rate of accumulating 10 WARP for college vs high school (which it was (59%-41%)/41%=44% and (17%-12%)/12%=42%, then the ratios were probably pretty similar for 20 WARP, 30 WARP, etc. With more time, I would have liked to done this more seriously as it seems Rany Jazayerli did quite well in his series several years ago.
The realist says, "The shuttle stops running in an hour so I don't want to get too drunk, but we can stay for one beer."
The hedonist says, "Aw heck it's a Friday and I want to party harder, I'll get a beer and a shot."
The economist says, "I'll pay for the first round. You each can get a beer and a shot on me."
After the first round, the realist is so happy he returns the favor by buying the economist another drink and a shot.
The hedonist is so happy he's buying drinks for everyone for the rest of the night.
Finally, the economist is happy because he drinks the rest of the night for free and knows the realist budgeted enough money to pay for a cab ride back to the hotel.
I think its tough to say "college players are a better bet" in a vacuum, because it all depends. In some drafts, the 10th best college guy might be rated below the 10th best prep guy. What a player does after he is drafted is exhibit B, but you draft a guy based on what he's done, and then project what you think he'll do, that's exhibit A. Lots of guys have had decorated college careers, only to fall flat in the majors against the consensus opinion, just like lots of prep guys destroyed high school pitching, looked like the next Mickey Mantle, and couldn't make it out of AA. The draft in baseball, more than in any other sport, seems to be a lottery of sorts. The teams with the best scouts and the best owners (ie, willing to spend) generally seem to get the most out of the draft, but even the best teams make high profile mistakes due to the nature of the beast.
It should also be said that the "Moneyball philosophy" is to target undervalued assets and traits. If teams had valued college players as X prior to the A's 2002 draft, and then valued college players as Y after, it makes sense for the A's to adjust their thinking. Its just like a value investor like Warren Buffett who says the best time to buy stocks is when others are fearful and panicking. Likewise in baseball, maybe the value of prep prospects to the A's changed when others started to value college guys more.
The reason I did do the chart in clumps of six years is specifically because I figured (and you confirm) that some years will be particularly strong in one group, like college arms, but then not the next.
I considered talking about the A's changing their approach in response to the change in valuation of high school draftees (as in the 8th best HS player used to already be drafted by the 20th pick, but now is not and is a better bet than the 13th best college player), but as I mentioned in another comment, I didn't want to overplay the valuation theory stuff after I'd done so much economics in my first few articles.
One potential bias I wonder about is organizations' desires to push draft picks to the majors just because of where they were selected. For example, given the same minor league performance, perhaps top ten draft picks are allowed to see the big leagues more often than second round draft picks. Heck, this might not even be a bad bias, as there could be an innate skill in the top ten draft picks that makes them more likely to succeed with the same minor league performance (such as being younger, or a better ability to kick it into another gear against top of the line competition.)
If anybody does want the excel file, I'm happy to share it. I don't want to get spammed, but for those who don't have it, you can go to www.econ.upenn.edu, go to graduate students, and click on my name to get my email address.
Thumbs up, Matt. Your article wasn't as good as last week's, but your next-to-last paragraph rocked. Great job.
1) The 51.3% form 1978 - 2002 doesn't seem to jive with your chart. From the chart it seems that only one of the bars is above 51% (the most recent), but the first two are well below 49%. The only way that 1978-2002 could be 51.3% is if there were lots more trials in 1997 - 2002. Is this because of the expansion of supplemental picks recently?
2) Matt is right in his use of t-test to assume the two are from a similar group, though I have typically found these t-tests to usually fail when there are lots of trials. For example even hairline differences when there are 10,000 trials always seem to suggest they are different. The one concern that I have is that by using the t-test you have assumed that each draft pick is independent of every other draft pick. That may be a pretty big assumption in drafting. It may work in random phone calls for polling, but I would bet that whoever picked 4th in a draft, who they pick may be highly dependent on who was picked 1st, 2nd, or 3rd. Likewise there may be some dependence of what a team picks in the 2nd round (and or supplemental rounds) based on the player they picked in the 1st round.
I see what you're saying about independence, but when I get a t-stat around 4, and my 2003-2008 sample was only 473 picks, that couldn't really change the effect much. No one team gets that many picks to make a huge difference, and there's some negative dependence where it comes to the fact that if the best high school player was picked with the first pick, the second pick is LESS likely to be a high school player. I'm guessing the effect is even stronger.
Sample 1: mean = .513, std.dev of estimate error = .01258; SQRT(.513*(1-.513)/1578)
Sample 2: mean = .415, std. dev of estimate error = .02266; SQRT (.415*(1-.415)/473)
So to compare two sample means, our NULL hypothesis is that the difference is zero. To do this our mean is .513 - .415 = .098, and our std dev of our estimate error is (.01258^2 + .02266^2) = .025915. Therefore the t-statistic is 3.78 which would fail for any reasonalbe confidence, so yes, there is strong evidence (in agreement with his conclusions) that these are not from the same rate and suggests strong statistical evidence that clubs are valuing college players more highly than high school players.
The real interesting question though is if the pendulum has gone too far, i.e., is there roughly a "right" percentage which suggests that the pendulum has found the optimal equilibrium, i.e., is the reason the A's are now drafting more high school players is because there is now evidence that everyone is overvaluing college players?
(Note to Matt and other contestants: I worry that everyone's going to start worrying about using any stats work since the comments seem to jump on small details everytime (I'm guilty of this as well), but I'd like to encourage you to just do it anyway.)
If you know statistics better than I do, which is certainly possible because my fields are engineering and business, not statistics, refute my points. This is a competition, I've nothing at stake, and I am cheering for you: if my understanding is imperfect, please educate me.
Besides, whether the underlying distribution of interest is binomial, bernoulli, or anything else, he's got a large enough sample size that surely the central limit theorem applies, right? A simple difference-of-means t-test like Matt used is more than fine.
Stoopid trivia.
"The counter-argument...is that high school players have higher upside and are more likely to succeed while in the majors....Of the college players drafted in the first two rounds, 17% of them reached a career WARP3 of 10.0, and only 12% of high school players did."
High school players are drafted for upside, college players for (presumably) higher degree of safety. So, how do those HS draftees compare to the college ones? That is, while a smaller percentage has reached the majors, have they done so while achieving their supposedly higher upside? Or has chasing upside in 18 year-olds proven to be a fool's errand?
The question then, is will college vs. high school strategy be 'more successful' in that regard? If I remember correctly from Rany's research, he concluded that the first coupla picks were far more likely to have 'star success' (my term, not his), regardless of whether they were high school or college draftees. Every other consideration after the draft slot was not statistically significant. (That's my recollection, and I hope somebody will correct me if I'm off base).
If my recollection is correct, my suggested research would be whether you can identify 'star' potential through a high school/college correlation for the rest of the draftees, assuming you're not picking in the top coupla slots each year like the Rays. Example: if you've got the eleventeenth pick in the first round of the 2009 draft tomorrow, are you more likely to get a 'star' by drafting from high school, or from college? What about pitcher vs. position player? Anything else that might catch an emerging star to propel your club during the 2012-2015 seasons? If you can identify those characteristics, well... there's folks who would pay for that information.
My goal in doing it the way I did was that I only had so much time to collect information, and I figured that if I used the qualification of making the majors and the qualification of becoming a reasonable regular (10.0 WARP3), and college players were equally more likely to meet either qualification, then chances are the distribution of talent was such that I had made a good guess. According to Rany's study, it seems that if I had grouped together the same time frame together using a large WARP3 cutoff, I would have come to the same conclusion.
"Draft Rule #5: In the first three rounds, not only are college players about 50% more likely to reach the major leagues than high-school players drafted in the same slot, they produce approximately 55% more value over the course of their careers. This advantage is persistent at every point after the #1 pick."
http://baseballprospectus.com/article.php?articleid=4042
"High School College
Never reached majors: 459 281
WARP Between 0 and 1: 58 66
WARP Between 1 and 5: 63 101
WARP Between 5 and 10: 30 47
WARP Between 10 and 20: 41 56
WARP Between 20 and 30: 27 43
WARP Between 30 and 40: 16 23
WARP Between 40 and 50: 6 15
WARP Between 50 and 60: 2 8
WARP Between 60 and 80: 6 11
WARP Greater than 80: 4 13
Total: 749 715"
http://baseballprospectus.com/article.php?articleid=4064
When I was reading the various charts and graphs and analyses, other thoughts were coming to my mind since I had already seen that kind of analysis before. The graph of first and second rounders seemed to indicate to me that there is less difference in 1st round and 2nd round talent than there used to be. While it is very possible that international signings have prevented fewer draft picks to reach the majors, I was wondering if advancements in medicine/surgery/training and longer career life in general had had an impact. I also didn't quite buy the argument that high school players have more potential than college players, I think they would just appear to since there is less data on their performance against solid competition and if they are pitchers, then their arms might be "fresher".
Overall, a good, solid article. I also realize it was Matt's first foray into the topic and I shouldn't expect to be wowed every week. So, another good, solid thumbs up.
High school players on the other hand are more potential and projection than certainty. So while it's harder for scouts to try to figure out who can actually make if from high school to the majors, if you really believe in your organization's player development, maybe you are willing to take some chances on "toolsy" players who are all potential because you trust your minor league coaches and trainers to mold him into an actual major leaguer.
I got to thinking about this when when I was reminded of the NFL draft, and how over at FO Outsiders there was a lot of talk that while the Pittsburgh Steelers often nail the draft, that doesn't mean that their scouts are necessarily better, but instead they let their rookies and younger players sit on the bench and develop, so that when they are called upon to start in their third or forth year they often surprise and succeed.
I'd imagine that a baseball organization could function the same way, get the right instructors and coaches down in the minors who know how to take certain tools and help prospects turn those tools into actual skills and results at the major league level.
Just looking at college pitcher vs high school pitcher, in theory I really like the idea of drafting the high school pitcher and then letting him spend then next three to four years down in the minors under organizational control in the organizational training program learning from organizational coaches and instructors. The college pitcher on the other hand will probably be raced through the minors where he only needs to prove that he can handle that level's hitters, while not receiving all that much training and development.
There seems to be very few players that they've drafted that reached the majors compared to others (i.e., it's not like I scope other teams rosters and say -- oh, that's a player the Royals drafted, the way I would with the Rangers).
Is it simply that the Royals and Pirates are picking the wrong players, or is it that the pick the right players but their early development of minor leagues are so bad that they "ruin" players with potential. Now THAT would be a great article.
Good none the less.
Starting off with a quibble: " But beyond that, do you know much about what the draft as a whole produces?" That's a bit of an awkward lead into your article - and isn't a vivid picture of what this is going to be about either.
Not a quibble, but a question inspired by this article: Bill James pointed out that college players have more success in the Majors than high school draftees almost 20 years before Michael Lewis. Why did Lewis apparently affect this change and not James? It is interesting from the bar graph, there was a bit of a dip in high school picks around the time immediately after James' first exposed this fact. Why didn't that trend continue?
Back to quibbling: Matt keeps repeating his mini conclusions that teams know what they are doing in the draft. I wince each time he does that. It is not worth noting even once. What we are measuring is the degree to which they know what they are doing.
Trivia try: Without looking it up or reading the comments I guessed Don Sutton - Brewers '82, but then I thought, no, what about David Cone - of the Blue Jays '92 or '93 teams? (Reading the comments, I see someone mentioned Cone's work with the Yankees, but all of that success mentioned came in years after the trade. I inferred that the question pertained to immediate help. Yeah, Tudor is a good guess.)
Wow, this essay took a steep turn into an entirely tangential area (trading of top picks) leaving the original course missing begging for further exploration. Matt ties it back with a fourth "gee wiz, those GMs know what they are doing" comment.
I sympathise with Matt's final conclusion. Yes, it takes time to do a worthwhile study. He isn't getting paid a wage to live on in order to have time to do that. But, that and the list of unanswered questions was a most dull conclusion.
The winners of '88 WS were: Pena, Hershisher twice, Honeycutt, Belcher.
The winners of '92 WS were: Glavine, Smoltz, Ward twice, and Key twice.
Michael Lewis' conclusion affected the league more than Bill James' for a number of reasons. For one thing, when we talk about this concept-- of new information becoming available and the market adjusting-- we have the Efficient Market Hypothesis in mind. The EMH essentially states that if information comes out that says that there is an undervalued commodity, those who get this information will immediately jump upon the market and drive the inefficient competitors out of business. Baseball is not a free market system. There are 30 teams and there is no entry. In essence, this leads to market inefficiencies that can persist over long periods of time, unlike with other markets. The 30 teams do not often compete with each other for fans, and draft picks are allotted more or less equally among the teams. For that reason, Bill James stating something that maybe one or two teams noticed would not necessarily move the market. Michael Lewis stated it on a much larger stage, and it pushed GMs to adjust to keep their jobs.
Besides, at this point, each contestant has a body of work and overall I think you are in great shape. You're the first person to break the 50% thumbs up barrier and that won't be forgotten soon. Meanwhile, those who looked shaky at the beginning have been improving... maybe not to the point where they can become the BP Idol, but to be able to add "Published writer on Baseball Prospectus" to a resume _and_ to learn better techniques has got to be a real benefit. As I said, I hope quite a few people get hired by BP when this is all said and done, and those that don't just might latch onto sites elsewhere.
"For all of the rampant trade rumors and discussions about making a move to get over the hump, it seems like the ultimate goal of those moves is rarely achieved"
I don't think the "ultimate goal" is appropriately defined as winning a WS game. How about started a playoff game?
However, Matt picked a topic that has been covered by others in a lot of depth. I think that Baseball America (as well as Rany) did a lot of work on this topic (albeit by using a bunch of less valid statistical measures of success).
I just do not see why one should applaud a less well done version of a Draft Analysis. Artists do not receive pundits for copies of others work (even as a training piece), and sorry Matt, neither should you.