The second of a three-part series.
One particular area of concern in my attempt to develop a new, more accurate formula for MORP is how to price wins above replacement level for superstars, players who add several wins above replacement level. The current alternative to MORP, found at FanGraphs.com, lists a player’s value using its win statistic, WAR. They do this by approximating a price of how many $/WAR teams pay on average on the free-agent market and multiplying it linearly by WAR to find that value. While I have a number of concerns about that methodology, this particular choice seems worth exploring.
As it currently stands, MORP is not linear with respect to WARP. In other words, if you take a player with 4.0 WARP, like Jason Bay, his MORP is more than twice as high as a player with 2.0 WARP, like Johnny Damon. This made some sense at the time the study was done, since WARP previously used a much lower replacement level. Thus, players who produced a WARP below 3.0 using the old replacement level were all probably paid close to the major-league minimum salary, as they were probably all pretty close to replacement level. For players between 3.0 and 6.0 WARP using the old replacement level, teams would start paying more for wins as WARP increased, since these players were actually above replacement level.
However, with a higher replacement level, this makes less sense as an assumption and is worth testing. Nate Silver‘s original formula for MORP logically had salary at the league minimum-when a player’s WARP was 0.0-as we see in blue below, but it also showed salary accelerating as WARP increases. If we imagine that the red points are sample data points of salary and WARP for individual free agents, it makes sense to graph it, with the curved shape we see below. Trying to draw a straight line through these points would not look like it accurately reflected the relationship. Instead, the blue parabola best fits the data:
However, Clay Davenport recently shifted replacement level upward to better reflect the players available at the league-minimum salary. Now, let’s see what happens to the same exact distribution when we shift replacement level upward and lower all those WARP levels by two wins.
If we continue the natural assumption that zero wins above replacement is worth the league-minimum salary, suddenly, the best fit is a straight line. To properly estimate whether the relationship between wins and the dollar value of a player’s wins is linear, we need to dig back into some theory and see what kind of assumptions would make that true and whether they hold in this case.
In a brief e-mail exchange with Silver on the topic, Nate said that one reason that it may still be worth keeping MORP as a non-linear estimator is that we are not estimating what teams should pay for wins but what they do pay for wins. We are stuck with the reality that we cannot determine how many extra fans are put in the seats by adding a player with a certain win value, and we are therefore stuck with using the inferred price of free agents that teams have shown us.
However, we are trying to approximate how much teams pay for that production. If we wanted to simply approximate how much a free agent would sign for, even if he was not necessarily worth the money, we would include a term in the equation to give the player extra MORP for high RBI totals. After all, some GMs certainly place a premium on this and the player will earn more as a result. A formula that simply adds extra dollars to the MORP value when a player has more RBI is hardly the production valuation that we are looking for from MORP.
Think of the implications: If we want to use MORP to evaluate signings, we cannot simply say, “Jason Bay got a great deal. Look at all those RBI that brought his MORP up! Clearly Omar Minaya overpays for RBI, just like Baseball Prospectus told him to!” After all, Bay’s placement in the potent 2009 Red Sox lineup led to a lot of RBI that we do not want to pretend is part of his value, since he would not have gotten the same total had he remained with the Pirates.
Similarly, we are also not going to include a dummy variable in the equation for “Was this a veteran relief pitcher signed by Ed Wade?” that adds a certain number of dollars to MORP when the answer is “yes.” We are figuring out what teams pay for production, so we should be using their WARP total as the measure of output. Similarly, we should not implement a system that adds extra $/WARP for higher WARP unless there is an actual basis for doing so in economic theory.
If baseball free agents were in a typical, perfectly competitive market like those you see in the first chapter of your introductory economics textbooks, the price per win would have to be linear. Basic economic theory of perfectly competitive markets would say that anything other than the same price for all wins would create arbitrage opportunities where teams could perpetually trade their way to the top of the league. This makes sense in something like a stock market. Say the dollar price of two shares of Microsoft was more than twice the dollar price of one share of Microsoft. People would simply buy up all the single shares of Microsoft, and then sell them in packets of two at a profit. The end result is that the demand for single Microsoft shares would go up, driving that price up, and the supply of pairs of Microsoft shares would go up, driving that price down.
Of course, strawman economists, who consider it their job to think really hard about the first chapter of the econ textbook, are always wrong. That first chapter is really supposed to be a baseline from which we can see what would happen as we change each assumption. In the case of baseball free agents, there are two main reasons why the baseline’s assumptions don’t apply. First, these markets aren’t thick enough that teams can sign and trade players so easily and quickly swap out players for others like investors can do with shares of Microsoft. There are only so many teams, and there are limits to making this kind of move in general. Second, you can’t employ 60 Garret Andersons on a team and suddenly become the best team in baseball. There are only 25 roster spots, and only so many players can realistically get enough playing time to realize their true value. Somebody needs to be playing above average for the team to make the playoffs.
The first issue of “thick enough” markets does not matter as much here. As the Mets could have simply signed Jon Garland and Johnny Damon, both two-win players, rather than just Jason Bay, one four-win player, it does not make sense to say that the MORP of a four-win player is any more than the combined MORP of a pair of two-win players. Since the money it would take to get four wins of production is better estimated through the price of two two-win players if there are spots for both, the price of four wins should be the combined price of Damon and Garland. Of course, that assumes that the Mets have spots in their rotation and lineup for both.
The issue of there being spots for both is curious. In the initial background study toward the general project of developing MORP, I decided to look at how many roster spots teams could realistically upgrade from a replacement player. I looked through each team’s roster at the beginning of the 2009-10 offseason and attempted to determine how many places a team would have been left with expected replacement-level talent without making a trade or free-agent signing.
The concept of what a replacement player means is hazy, but there certainly does seem to be a talent level that is common among teams who replace players. I used something roughly equal to that definition to do this analysis. This definition, for clarity, involves about 20 runs below average over a full season for position players with positional adjustments considered. For starting pitchers, it involves approximately a 5.50 FRA in a neutral league. For relief pitchers, it is approximately a 4.50 FRA in a neutral league. FRA, or Fair Run Average, estimates what a pitcher’s Run Average would have been if he had neutral performance on inherited/bequeathed runners. These are only ballpark figures, as I only have a few projection systems to work with, but these levels do make sense as there seems to be about three to six players who would have been slotted as starters at each position with no off-season moves. All of them were about 20 runs below replacement level and more than half of teams had a fifth starter who could be expected to put up a FRA near 5.50. Among relief pitchers, it’s tough. Although the average team employs six or seven relievers, not all of them get high-leverage innings. Looking through how teams have historically used relievers, a good estimate is that each team uses about four relievers in high-leverage situations regularly over the course of the season. Therefore, if a team had three relievers projected to have FRAs below 4.50, I considered that one opening in the bullpen where a new reliever could be signed and get high-leverage innings.
The analysis is relatively straightforward after that. In final part of the series on Friday, I’ll check how many spots the average team had, with special emphasis on teams who typically spend on free agents and therefore set the market. With these definitions and the understanding developed above, I’ll look through each position on the diamond (and DH in the American League) as of the beginning of November, and the top five projected starters and top four relievers at this time. If most teams have a few replacement-level openings at the beginning of the offseason, then we will know it is OK to go forward with a linear relationship between dollar value and WARP.
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.
Subscribe now
Specifically, are we talking about stuff like the Florida Marlins in 2009 replacing the outgoing Mike Jacobs at 1B by shifting Jorge Cantu over to 1B, and employing Emilio Bonifacio at 3B? So then Bonifacio's production would be in this talent level bucket?
(I suppose since Bonifacio in 127 games accumulated -19 runs below average, he'd be the perfect example).
I would still technically consider Bonifacio to be "free available talent", as he was acquired for a 29-year-old Jon Rauch and later a throw-in along with prospects for Willingham and Olsen-- the prospects being the "real" part of the deal. But I understand why you can't include guys like him in that bucket.
Regardless, it's hard to see exactly what it would prove if each team had 3 or 4 replacement-level starters heading into the offseason. This still means that a team cannot sign 6 low-WARP players for the same production as 3 better players; there is still a scarcity for higher quality players and they will demand a higher price. The only question is HOW non-linear this relationship is. Your graphs seem to imply that the answer is NOT VERY MUCH (presumably because the scarcity seems to provide extra contract security instead of a salary premium).
Still, i think we can look at the theoretical value of this scarcity a little more directly, in contrast to: "We are stuck with the reality that we cannot determine how many extra fans are put in the seats by adding a player with a certain win value, and we are therefore stuck with using the inferred price of free agents that teams have shown us."
In some ways, the increased value of better players can be measured by how far along the win curve their teams are, on average. That is, while marginal revenue is fairly linear in wins for the average team, valuable players make it easier to be further up on the curve, so that marginal wins are higher. By averaging the marginal revenue 4 win players provide (as measured by finding the difference between expected revenue at the actual number of wins - expected revenue for wins less the player's contribution), you'll get a better idea of what this advantage is (which should be greater than 4* the average marginal revenue that 1 win players provide; there's also one extra complication that needs to be taken into account-- the incentive to spend more money on your team when marginal revenue is greater --but this could be corrected for by using salary as a percentage of team payroll and then translating to an average payroll or some other fix). This would leave a similar scatter-plot of MRs and salaries which you could check for linearity.
Hope that was somewhat clear / made sense to anyone but me. I just wouldn't give up on attempting to use MR just because we don't have direct measurements.
The example you gave of 6 low-WARP players vs. 3 high-WARP players actually will illustrate the point of exactly how low and high WARP of free agents are. There are only a small handful of free agents that will get you more than 4.0 WARP reliably, while you can certainly see an abundance of guys in the 1.5-2.0 range. I guess it's possible that a team could try to sign 3 guys with 4.0+ WARP, but that's really rare and not common enough to affect the market price, mainly because you'd really need two bidders to push up the price. Maybe the Yankees signed Sabathia, Pettitte, Burnett, and Teixeira last year, but that's a rare year even for them, and they probably not have people competing with them to sign several of them rather than individual teams competing to sign one or two of them (e.g. the Red Sox were not probably also trying to sign three of these guys, probably just Teix and a pitcher at most).
The scarcity is reflected in the already higher price-- it's just that the scarcity comes in linearly.
The important distinction to be made with how far along their teams are along the win curve, on average, is an important one. The average team does not sign free agents-- the team that values the player the most does. Thus, teams that sign free agents are the ones that are likely to be the most competitive. For them, the marginal value of a win is already high since most of them are already in the 85-95 win range.
For your MR measurement, the problem is that we do not have counterfactuals. In other words, even ideally we could only know what the Phillies' revenue was last year given that they won 93 games-- we do not know what their revenue would have been had they not signed Ibanez and won, say, 89 games. Any attempt to solve this empirically would be muddled by the fact that the Phillies clearly thought 89-73 was unpalatable compared to 93-69, so even if we had the revenue of other teams who went 89-73 or even the Phillies in previous years where they won 89 games, those numbers likely understate the difference between 89-73 and 93-69 because in instances where the team won 89 games, they likely didn't have as huge a gap (and didn't want to spend the money), and in instances where a team won 93 games, they likely had a larger gap.
In reality, assuming the very rich individuals and companies who own baseball teams have at least a decent sense of their marginal revenue, we can use the Axiom of Revealed Preference to impute a decent approximation of marginal revenues.
Let me know if I left anything out, and thanks for giving such an incredible amount of thought to this. It's a very challenging job to redo MORP, and I encourage thoughtful comments like these to help guide me along and challenge my assumptions (and change them where necessary!).
@JDSussman: I would say that this is estimating what teams should pay for wins, conditional on the fact that the market is producing the correct price on average. For instance, if Omar Minaya overestimates the win-value of Jason Bay and overpays but Jack Zduriencik sneaks in at gets Chone Figgins at low $/WARP, that would not be an issue in the model as they would cancel out. Where the issue would lie is if winning actually doesn't add much to revenue but Steinbrenner just lets Cashman spend to a certain budget because he can afford it-- rather than because he can make money allowing him to do so competently-- then MORP won't know that this is happening. It assumes that, on average, teams are paying for wins at their marginal benefit.
And thanks for the clarification on your approach. I get what you're doing now, though i'm still a tad confused on the logic behind the last sentence of your article. Well-- can't wait for your article on Friday when perhaps things will clear up a bit!
I'm quite interested to see the next part of this series, because I think there's some confusion between the salaries being linear and the value of players being linear. With the appropriate replacement level and assuming 0 WAR players get paid the minimum you probably will get a linear relationship in observed salaries. However, I think there's an issue with market thickness that you haven't addressed (do certain teams have monopsony power over top free agents?) that may make observed salaries appear more linear. Furthermore, it may be the case that so few of the GMs or agents take into account lineup constraints when negotiating contracts that the value isn't accurately priced -- really smart guys like Tango didn't think about it either. Finally, I really do wonder about the shadow price of the lineup constraint. Take a team full of guys who are +1 win above replacement -- one +5.5 win player and one +0.5 win player are more valuable than two +3 win players because of who they're displacing to the tune of 0.25-0.50 wins each time you make this decision (full examples in the comments that I linked to above). So long as you're a team with significant depth of above-replacement players (i.e. with a great farm system or an unlimited budget), there should always be an extra return to concentrating talent.
Colin noted in those comments, rightly so, that I'm ignoring any aspects of risk hedging. That's absolutely right, and a series of articles on optimal roster/lineup construction would be a fan-freaking-tastic project for BP to tackle.
Last note -- I think it's worth deciding, and being clear about, what exactly you want the new MORP to represent. Is it the going rate of a win above replacement? Is it an estimate of marginal revenue product? Bid shading in a sealed bid first price auction would suggest that those are not the same thing. Similarly, if you viewed this market as a Nash bargaining game due to monopsony power (although I prefer your auction interpretation), it's not clear why players should be earning their entire marginal revenue product.
I do think that current estimates of replacement level-- 40 to 50 wins for a whole team-- look a little low unless you count displaced talent. Specifically, the first example I'll give Friday is the Phillies and Greg Dobbs. He's not quite 20 runs below average even if he had to hit off lefties, but the effective pinch-hitting changes by making the Phillies use a worse PH vs. RHP make it about 20 runs below average. So I think displacing people is part of the story.
Also, remember that calling up talent from the farm costs service time, so I don't know if that's really a preferable solution in many cases.
With respect to what MORP is trying to represent, it's a good question. I'm basically assuming that teams are setting marginal revenue to marginal cost on average, but that there is some noise in the estimates of win values of players. I would say that marginal revenue is really equal to marginal wins x marginal dollars per win. Further, I'm assuming that marginal dollars per win is probably negligibly different between the top few teams but that marginal wins is really where there is more variance. From there, I'm basically assuming a first price sealed bid auction. Due to the winner's curse and the fact that it's a first price auction, I assume teams probably due shade their bids down some but that those are negligible differences. Really, I think that on average, a model of ($/WARP constant)*(WARP) pretty much explains the market and can give you MORP but that teams can do things like the Phillies did with Polanco (check my earlier article on this if you haven't seen it) by adjusting the replacement level of the (N+1)th team where there are N free agents. But in general, I do think that the model as is explains it pretty well.
I'm not sure I like monopsony because I think of the 30 teams as being the labor demand side rather than 1 MLB, so that would put it at an oligopsony, and while I do think that the draft is an example of implicit collusion of the repeated prisoner's dilemma ilk, I'm not sure that the free agent market does that mostly because teams from different divisions bid, and so I'm pretty sure they bid based on the expected marginal revenue (conditional on winning, winner's curse thing).
Thinking more about it, I do see the possibility that there is still some fraction of marginal revenue product that the players do not capture in expectation, but I think in that case MORP still answers the question "What do teams pay for this number of wins on the free agent market on average?" so I think it's a valuable tool even without looking at actual marginal revenue.
I definitely liked your discussion with Tango, though. I think you saw something he missed, but I think the results make it such that it doesn't matter.
There are definitely a lot of complicated issues here, but I think that the main one is the roster construction issue of number of openings and probably these results are enough to ensure it's safe to use a linear $/WARP framework.
That said, this is a theoretical point, leaving the magnitude of the effect uncertain. If the shadow price of the roster constraint (and playing time constraints) is zero, then there would be no effect. If it's close, the effect would be negligible (which i believe is Tango's position here). It appears that Matt is presenting data here that back this up.
I've got to admit, that's pretty counter-intuitive to me, given all the hand-wringing and whatnot over getting enough arms in the bullpen, but if it's so, it's so.
I think given the uncertainty there, it's better to do a "free market returns" approach to valuing players, rather than to do an MRP analysis and assert that teams are incorrectly valuing players. (This hasn't stopped me from trying to do an MRP-style salary value estimate before, I should note.)
If the effect of your predictor variable (say X) is specificied as a quadratic function (b0 + b1(X1) + b2(X^2), you do allow for the possible curvilinearity (parabolic) but without necessarily imposing empirically.
This is true because if in fact the empirical relationship turns out to be strictly linear, then the coefficient b2 would be small and not statistically significant. In that case the linear coefficient (b1) would capture the effect of X on salary.
I agree with Nate that it may be preferable to use the quadratic function as a theoretically more compelling function in this case that would not in fact cause any serious distortion in your MORP estimates if it turns out empirically to be a linear relationship. The cost in degrees in freedom is also negligible.
My analogy is to the price that I paid per meal as a function of the number of stars that a restaurant gets in the Michelin guide. My experience in that case told me that while I might pay $50 for a meal in a 1-star restaurant, and $100 in a 2-star restaurant, it would cost me $200 in a three-star restaurant. The marginal pric of the additional star was a nonlinear function. (This based on a "45 days in Europe" tour some years ago. Your costs may be different!).
If b2 turned out to be significant, then that would indicate a market inefficiency or something missing like arbitration compensation or replacement level too low, etc. But if adjusting for arbitration and proper replacement level, I still find b2 is statistically significantly positive, then that means something is wrong with the market that can be exploited.
What Matt is calling a market inefficiency could also be understood in some other way. For one, a buyer (GM) is, after all, only making a forecast of the player's performance. would conjecture that the greater the GM's forecast on the wins produced by a given player, the greater the tendency to let hope come into play. Given that any forecast has a probability distribution around it, when you move into acquiring a star player you may tend to envision next year's performance as coming at the high end of that distribution (perhaps by picking out the player's "best" performances in recent years rather than making a more conservative PECOTA-like projection).
If you want to call such an "optimistic" projection inefficient, that's fine. But it still can lead empirically to "fairly valuing" slightly above replacement players and overvaluing the better players.
Also keep in mind that roster spots are also scarce (limit to 25), and there are transaction and maintenance costs for players (health care, uniforms and equipment, travel, etc.); so if you can get the same number of wins from one player as you get from two, then your "costs" per win may be reduced by hiring one 4-win player rather than two 2-win players. There can also be efficiencies when you get a "utility" 2-win player rathern than two one-win one-position players.
I like the transaction cost idea a lot, but I think short of health care costs, it's really hard to see other stuff playing a huge role (i.e. you get 25 uniforms and 25 plane tickets regardless of whether a schlub is wearing the uniform in the airplane seat or not). Health, maybe, but I don't know-- how much does Dr. Andrews charge?
I may have missed it somewhere in your explanation, but to be certain - I take it you assume that when a team makes a decision to sign a free agent, it is not just in the need to displace a replacement level player, but more specifically to replace a replacement level player at the position of the new signee.
I'm glad you were able to follow without taking econ. I always strive to explain the econ well enough that it makes sense as common sense rather than just empty theory :-)
I don't think this analysis is economic given the subjective theory of value, which very much is econ 101.
I'm also confident that in the past decade of work at Baseball Prospectus, including much of the pioneering work by Silver, there is little ability to control the probability of success once you reach the playoffs with notable exceptions like the Secret Sauce formula that you can find on the stat pages. Of course, teams put different subjective weights on these different outcomes, but regardless, that would not really change the fact that signing a superstar and signing several average players are both viable choices even for teams who have higher aspirations is still an option. I'll say that again for clarity-- the whole point of this article is that achieving the 100th win or the 60th win, teams who sign superstars will generally have the ability to add those win by filling in several average players as well. There are very few situations where a team has almost no replacement level players in important roles.
As far as your 71st win versus 89th win tradeoff, I addressed that in the last article, this one, and again in the comments-- the point is that the vast majority of teams who sign free agents are the ones with the highest value for a win. The reason is that those teams generally value the win more. You can figure that out from their behavior, which is both intuitively obvious and basic economic logic. No one is questioning Silver's marginal value of a win-- which isn't a subjective argument either, but an objective one with noise mixed in-- but rather approaching how the market works given that. Since teams who sign free agents are generally competitive and generally set the market, there is no conflict in those thoughts, what I'm saying, and economic theory.
I say that as a Jays fan with no real hope for this season - but some reasonable hope for subsequent seasons. Either way, even the Jays sign some guys - a bunch of MiLB contracts, John Buck, various MI waiver claims, re-upping John MacDonald... Even the Pirates signed Ryan Church