For me, this is a lot of fun, but as a refresher, here’s how these rankings are calculated. First, we find each team’s expected revenue, based on their third-order winning percentage, and how big their market is. Then, you divide that by what each team’s marginal revenue should have been, had they won exactly as many games as their payroll would have predicted. (Draft pick value is also factored in, so the worst teams get slightly more credit than the vanilla mediocre teams.) The end result is PER-Payroll Efficiency Rating-which tells us how well each team spent their payroll dollars.
To run through a quick example, the Rockies spent $75 million on payroll last year, a bit below average. That should have led to around 79 wins, which, given their local market, would have created around $41 million in marginal revenue. But the Rockies actually had 90 third-order wins, which likely created somewhere around $58 million. Divide the two, and you get a 1.43 PER. Since 1.00 is average, we can say the Rockies’ front office performed 43 percent better than average in 2009.
On to the lists. Starting with the bottom ten, among all general managers with at least three years on the job this past decade:
GM | Years | PER |
10. Randy Smith | 2000-02 | .976 |
9. Dave Littlefield | 2002-07 | .970 |
8. Wayne Krivsky | 2006-08 | .966 |
7. Doug Melvin | 2000-01, 2003-09 | .954 |
6. Mike Flanagan/Jim Beattie | 2003-07 | .954 |
Smith gets credit for 2002-he was fired a week in-which keeps John Hart out of the bottom ten by the skin of his teeth. Hart had the misfortune of leading the Rangers through the last two years of the A-Rod era, which, along with Melvin’s last year in Texas, were three of the worst years in the entire database. Melvin has been better since moving to Milwaukee, but Branch Rickey couldn’t have made up for those last couple of years with the Rangers.
Unlike Hart and Melvin, the other three didn’t have any years that were overly terrible-they just didn’t have any really good ones either. Flanagan/Beattie, Krivsky, and Littlefield averaged 71, 73, and 78 third-order wins, right in that dead zone where it’s impossible to make the playoffs, and extremely hard to get the first overall pick.
GM | Years | PER |
5. Chuck LaMar | 2000-05 | 0.935 |
4. Brian Cashman | 2000-09 | 0.869 |
3. Syd Thrift | 2000-02 | 0.807 |
2. Bill Bavasi | 2004-08 | 0.786 |
1. Steve Phillips | 2000-03 | 0.670 |
The only name that doesn’t fit here is Cashman. The Yankees made a lot of mistakes this past decade, and it’s not totally clear which of those were his and which were George Steinbrenner’s. But it’s pretty easy to look at that list and pick the one that doesn’t go with all the others. We’ll probably have to wait another five years to really judge him on his merits, but there’s already been significant progress — the Yankees are spending much less now than they did earlier in the decade (after you adjust for baseball inflation), and they just fielded their best team since 1998.
As for the rest, I’m not sure the system could have done any better. Phillips, Bavasi, Thrift, and LaMar were all pretty abysmal, and would have made just about anybody’s bottom ten list (objective, subjective, or otherwise). If there’s a surprise, it’s how badly Phillips lapped the field, despite having a World Series team in 2000, but his 2001-03 stretch is even worse than the A-Rod-era Rangers.
Now, for the best:
GM | Years | PER |
10. Terry Ryan | 2000-07 | 1.154 |
9. Brian Sabean | 2000-09 | 1.162 |
8. Ned Colletti | 2006-09 | 1.195 |
7. J.P. Ricciardi | 2002-09 | 1.216 |
6. Mark Shapiro | 2002-09 | 1.228 |
Let’s get the obvious ones out of the way first. Shapiro has had some really tough breaks, particularly in 2006 and 2008, but the third-order standings see through that and reward him for building some very good teams on limited budgets. Ryan led four division winners on miniscule payrolls, and was generally considered one of the best at drafting and developing young talent right up until his retirement in 2007.
The other three took less obvious paths. J.P. Ricciardi has been torn apart on this site and others, but had the Blue Jays been in any other division, his record could look very different-the 2006-2008 Blue Jays were very good teams, but were simply in the wrong place at the wrong time. Colletti has also taken his lumps, but he had an outstanding year in 2009-the Dodgers had 99 third-order wins-which pushed him up to number eight despite three middling seasons from ’06-’08. As for Sabean, if that doesn’t get Barry Bonds into the Hall of Fame…
GM | Years | PER |
5. Walt Jocketty | 2000-07, 2009 | 1.232 |
4. Gerry Hunsicker | 2000-04 | 1.292 |
3. Pat Gillick | 2000-03, 2006-08 | 1.386 |
2. Andrew Friedman | 2006-09 | 1.428 |
1. Billy Beane | 2000-09 | 1.518 |
If we had taken a poll on BP, there’s a pretty good shot Beane and Friedman would have been the top two. Beane dominated the first half of the decade, Friedman the second half. Together, they accounted for six of the top ten individual seasons of the decade (see below), with the Moneyball-era A’s taking the first, third, and fifth spots.
The other three aren’t exactly sabermetric favorites, but they were all very successful nonetheless. Jocketty’s Cardinals tallied 90 or more third-order wins five times, and won the World Series in one of the years that they didn’t. Gillick led one of the best teams of all-time-the 2001 Mariners-as well as the world champion 2008 Phillies. Hunsicker just missed the Astros‘ World Series run in 2005, but put together several teams that were actually better, before leaving in 2004.
How about the best and worst single season performances:
GM | Year | PER |
1. Billy Beane | 2001 | 2.10 |
2. Mark Shapiro | 2005 | 2.06 |
3. Billy Beane |
2002 |
2.06 |
4. Bill Stoneman | 2002 | 2.01 |
5. Billy Beane | 2003 | 1.97 |
6. Andrew Friedman |
2008 | 1.90 |
7. Andrew Friedman | 2009 | 1.88 |
8. Billy Beane | 2000 | 1.88 |
9. Brian Sabean | 2000 | 1.85 |
10. Ned Colletti | 2009 | 1.84 |
— | — | — |
296. Syd Thrift | 2000 | 0.61 |
297. Dan Duquette | 2001 | 0.59 |
298. John Hart | 2002 | 0.59 |
299. Dave Dombrowski | 2008 | 0.58 |
300. Steve Phillips | 2003 | 0.53 |
It’s a bit surprising that Gillick’s ’01 Mariners just missed (they were twelfth), but the marginal gain for each win over 100 is minuscule, and several of these teams were within a few games of the Mariners’ 109 third-order wins. The 2001 A’s, for one, had 105 third-order wins, despite spending less than half what the Mariners did.
Finally, the best- and worst-run teams of the decade:
Team |
GMs |
PER |
1. Oakland Athletics | Billy Beane (2000-09) | 1.52 |
2. St. Louis Cardinals | Walt Jocketty (2000-07), John Mozeliak (2008-09) | 1.25 |
3. Cleveland Indians | John Hart (2000-01), Mark Shapiro (2002-09) | 1.18 |
4. San Francisco Giants | Brian Sabean (2000-09) | 1.16 |
5. Toronto Blue Jays | Gord Ash (2000-01), J.P. Ricciardi (2002-09) | 1.16 |
— | — | — |
26. Pittsburgh Pirates | Cam Bonifay (2000-01), Dave Littlefield (2002-07), Neal Huntington (2008-09) | .97 |
27. Los Angeles Dodgers | Kevin Malone (2000-01), Dan Evans (2002-03), Paul DePodesta (2004-05), Ned Colletti (2006-09) | .95 |
28. Baltimore Orioles | Syd Thrift (2000-02), Mike Flanagan/Jim Beattie (2003-07), Andy MacPhail (2008-09) | .92 |
29. New York Yankees | Brian Cashman (2000-09) | .87 |
30. New York Mets | Steve Phillips (2000-03), Jim Duquette (2004), Omar Minaya (2005-09) | .81 |
With that, on to the debate on the results.
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.
Subscribe now
Also, if this was written about before as hinted at in the line "but as a refresher, here's how these rankings are calculated", can we get a link to it?
I'll put the lists in a google doc and post them here in a bit.
That doesn't mean just include every data scrap in an endless series of tables. We don't need to see the full 300 single season list for example. But all 30 teams and a deeper list of GMs to include big names like Epstein would seem obvious to include.
http://spreadsheets.google.com/pub?key=t8gbaBxUTK-iV_FuKzJGFyQ&output=html
The GMs with "a lot" have higher expectations -- the system ranked the Yankees, Sox, and Dodgers extremely high this year, b/c they were all very good teams. But any slip up can really hurt -- i.e. the Yankees in '08, the Red Sox in '06, etc.
I think we would have to see Cashman running a more fiancially-limited team to see where his true abilities lie. Were he to take control of the Cardinals or the Pirates and continue to overpay, then that would be something else. Until then, for all we know Cashman is simply taking advantage of a resource he knows he has access to. It still doesn't mean it's smart to overpay, because it isn't and Cashman should get dinged somewhat for that, but it's definitely a different situation in NY than it is elsewhere.
This year was very good though. And like I said, the Yankees are spending less relative to the rest of the league than they used to, so I would expect Cashman to do well going forward as long as the team is still winning.
This is a vey interesting article as it is and I'm not sure how you find the formula to handle higher payrolls with more accuracy. For the time being I'll just take the Yankees, Mets, and Red Sox rankings with a grain of salt.
Can you comment more about why this system is so kind to Sabean, who many Giants fans appear to believe is the worst GM on the planet?
Last five years pre-injury: 1.85, 1.28, 1.63, 1.38, 1.06
Since: .75, .84, .87, .94, 1.02
I think he's just a Jekyll and Hyde-type GM: Someone who will have some good moves that do make the team better, but then at the same time undercut his good work by not filling out the roster with the right parts.
To his credit, he did trade for Jeff Kent and Jason Schmidt, and drafted Tim Lincecum, all of which helped make up for a lot of mistakes. But without Barry, he's a lot lower on this list.
I'm not sure it's right to base how well the GM did using entirely third order wins with no weight on actual wins. For example, if Mark Shapiro insists on keeping Eric Wedge as his manager despite the fact that Wedge has knack for delivering fewer actual wins than expected, doesn't that eventually reflect on Shapiro too?
It does seem like the model was not work well with the Yankees, who are an extreme case. Given that they made the playoffs 9 out of 10 years in the decade, I'd be hesitant to call them the second least efficient team of the decade.
--trading for Jason Jennings
--signing Woody Williams
--having an opening day lineup featuring stalwarts Willy Taveras, Adam Everett, and Jason Lane,
--giving a full-time gig to Mike Lamb,
--paying Carlos Lee a contract that was probably 130% of what he was worth to the rest of the market,
--letting Carlos Beltran get away, and
--presiding over the complete collapse of the farm system that was third in 2002 to dead last by the time he was ousted.
But because the Astros made the World Series in 2005 riding on the arms of no-brainer-signees Roger Clemens and Andy Pettitte, along with a host of Gerry Hunsicker assembled pieces (Roy Oswalt, Lance Berkman, Craig Biggio, Jeff Bagwell, Brad Lidge), and remained competitive for the next couple of years with the same predecessor-built club, Purpura gets a pass.
For shame!
For 1, it is not always the job of a General Manager to win the most games he can for the price paid. Some GM's are told to win at any cost. Sometimes that means going out and spending money and assembling a relatively expensive roster. Unless that team wins a huge number of games, he's not going to do well in the evaluation you listed. If that team does poorly, well, that means that he chose poorly as to where the money went.
On the other hand, some GM's are told to strip cost and attempt to rebuild the organization. Eschewing current wins for future promise. These teams will often have low payrolls, but also poor results. If the team gets lucky with their prospects (or they are just very good at it), their team might become successfull before those players get expensive. Also, your formula says nothing about trades, or farm systems, or the like. GM's are responsible for all of that.
But I spend a lot of time on sites dedicated to statistical analysis of sports, and frankly I'm sick of people pointing out the INHERENT LIMITATIONS OF STATISTICS in a completely general way. Either accept those limitations or don't, but pointing them out does not do anyone sophisticated enough to read the articles on this site any good.
Furthermore, it's quite interesting, at least to me as I chew on Cashman's spot here, that over the past decade the Yankees ranked third in the majors in third-order delta (D3), the amount by which they exceeded their third-order record. They led the majors in both third-order wins (932.5) and actual wins (965), but the margin of their lead over the #2 team, the Red Sox, increases from 10 (W3) to 45 (W0). Data is here in a Google Doc.
Granted that the margin between 93 and 96 wins in a given isn't huge, but looking back on Nate Silver's BBTN graph of the marginal win curve (see here, that appears to be the backside of the peak portion; beyond 96 things level off considerably. So I guess what I want to know is how the team rankings at the end look if you use real wins instead of theoretical ones.
I understand that using 3rd order wins has some advantages too. I don't mind continuing to use them, but I'd like to see some other simple variable included based on actually making the playoffs. That's where the revenue really comes from so there has to be some kind of distinction made between good 3rd order win teams that don't make the playoffs (their revenue multiplier is docked some) and teams that make the playoffs regardless of their 3rd order wins (their multiplier is boosted some).
Also, I think this piece should be entitled best and worse GM in terms of efficiency- or something of the like. To just use this as the criteria for best GM seems silly.
And you can talk about training staffs and contingency plans all you want - if the 27 year-old defending batting champ and three-time All Star goes down at the beginning of Spring Training with an injury suffered off-site and unrelated to baseball, there aren't many contingency plans that deal with that effectively.
The Rays had lousy attendance and revenues in 2007 when they massively overachieved relative to budget. The Red Sox in 2006 and the Yankees in 2008 had excellent attendance (both teams having collapsed in September after being in the race all year) and therefore, high revenues, in spite of finishing out of the playoffs.
It seems to me that the marketplace cuts these GM's a bit more of a break in terms of the primary goal of any team- turning a profit- than does this metric.
This comes up with some pretty counter-intuitive results which people don't dig that much, but once you remember to interpret the results as a metric instead of GM grade, these counter-intuitive results are fun to see. I'm a big fan of seeing an objective take on the efficiency of certain teams, and it really does help you take a second look at the Sabeans and Epsteins. And re: Cashman-- we all hear so much about the Yankee's financial hegemony, but articles like this do a great job of putting it into perspective. Understanding that Big George had a lot to do with this can then help clarify our mental impression of Cashman's work.
No this system doesn't present a perfect measure for GMs (actually, i think the biggest problem has rarely been touched on with all these complaints -- the fact that one of the GMs biggest jobs is to ensure future success by strengthening the farm system, which can be completely unrelated to big-league efficiency, especially for teams with high GM-turnover), but pray tell me this wasn't an engaging, enlightening article. Kudos Shawn.
Jays decade PER 1.16
Rays decade PER 1.13
Red Sox decade PER 1.01
Orioles decade PER 0.92
Yankees decade PER 0.87
If you look at the teams who averaged at least 10% better than average you get 11 teams. 2 in the AL East and 9 elsewhere. The 2 in the AL East made the playoffs just 1 time (Rays in 2008). The other 9 teams made the playoffs 33 times and each team made it at least 2 times.
So that is 3.5 playoff trips per decade of 10%+ baseball if you play outside the AL East and 0.5 playoff trips per decade of 10%+ baseball if you play in the AL East. In terms of fairness, something is wrong with the current model and the Rays, Jays, and Orioles are suffering!
It just seems as if the team first gets knocked around by being the wrong division at the wrong time and then gets kicked while their down by having more injuries to key players than most other teams.