Ahead in the Count: Projecting Free Agent Performance

Before a team attempts to sign a free agent, they try to anticipate his value to them by approximating what they feel that player’s production will be over the coming years. Using that, they can then decide what to offer him. There are two prominent methods for evaluating free agents to come out of the sabermetric community in recent years, first here at Baseball Prospectus by Nate Silver in 2005 and later at FanGraphs in 2009. Both have made simple projections for the free-agent crop, and compared those to the salaries they received.

In moving into covering this topic as well, I have already noted that free agents who switch teams tend to decline more quickly than free agents who were not allowed to depart, which suggests that there is selection bias in who teams retain. Today, I'll look at all players who signed multi-year deals from 2007-09, and first demonstrate that these players generally fail to live up to their projections, and then delve into the fact that both re-signed players and those who sign with new teams underperform similarly in the first year of their contracts.

Players who re-sign for multiple years often agree to their deals outside of the offseason; indeed, some sign extensions pretty far in advance of a deal's period of service. For purposes of this piece, if a player signs a deal before he reaches six years of service time, the first year of this “deal” is the first year of the deal in which he began with six years' service time. So, for example, Joe Blanton and the Phillies avoided his last year of arbitration in 2010 by signing a three-year deal covering 2010-12, but the first year of his deal for purposes of this article is going to be 2011, the first year after he'd completed six seasons of service. To evaluate these deals, I used projected and actual VORP, since I wanted a counting statistic to allow for the fact that injured free agents turn out to obviously be bad investments. I specifically used VORP instead of WARP when projections were involved, since it was recalculated from a higher baseline beginning with the 2008 season, rendering 2007 and 2008 WARP projections too high and therefore working from a different scale. Although VORP does not evaluate defense or any balls-in-play performance metrics for pitchers, it will serve its purpose on the aggregate—the results are not ambiguous.

Of the 165 free agents who began multi-year deals in the last three seasons, just 72 managed to top their PECOTA projection in the first year of their contracts. The failure to get there was pretty extreme in some cases (I’m looking at you, Andruw Jones and Carlos Silva), leading the group to fall short of their projection by 14 percent; as a whole, they produced 2,483 VORP compared to 2,878.7 projected VORP by PECOTA. This was true for both re-signed players and players who signed with others teams, as re-signed players underperformed their VORP projection by 13 percent, compared to players who signed with other teams, who underperformed their VORP projection by 15 percent. This is consistent with Sean Smith’s finding that his CHONE projections were 16 percent too high for players on the free agent market.

Does this mean that PECOTA and other projection systems are universally optimistic? No. On the aggregate, they both predict about 81 wins per team each season, indicating that it is merely the sample of players who signed free-agent contracts that are the problem. This appears to be an issue with this particular sample of players who reach free agency. We'll get into why this is the case shortly.

It is important to couple this fact with my finding from February that players who sign with their old teams age far better than players who sign with new teams. I found players who signed contracts with their previous teams did not decline much if at all during the two-year and three-year contracts, while players who signed with new teams declined during similar deals; players who signed four-year contracts generally aged well. Here's the breakdown, and remember, breaking things down by age did not indicate any selection bias:

Pretty much all projections forecast decline for players old enough to reach free agency, so the implication of the information above is that teams who re-sign their players actually end up getting pretty good value, even if the first year of the deal does not look so hot, while players that sign with new teams tend to underperform their PECOTA projections and then decline further.

This is not meant to highlight a problem with projection systems in general. No projection system can take into account all of the qualitative information that goes into teams’ projections of performance or the actual performances; this is merely a biased sample. Just like looking at PECOTA projections of pitchers who had Tommy John Surgery the previous summer will yield a biased sample, and therefore produce an over-projection of innings if nothing else, looking only at players who signed with their previous teams and excluding the following year is a sample that is biased in a different way. Included in this sample are players whose previous teams chose not to retain their services—this group of players generally underperformed their PECOTA projection the first year of their contract, and then declined further as the contract went on. This sample also includes some players who did re-sign with their previous teams but whose teams initially did not extend them past that season. Of course, it also includes players who just became eligible for free agency for the first time.

It's important to remember that teams’ decisions against re-signing players are not always based on the ability to spend. Frequently they have inside information about the player that deters them from offering a contract. This includes information about the daily bumps and bruises and how well his body bounces back from them, how much medical treatment pitchers require between starts, and how diligent players are about their conditioning. Even players whose teams waited until the offseason to re-sign their players comprise a biased sample as well—these are players whose teams often chose to evaluate them further before tendering a contract. While sometimes this is because Scott Boras is just hell-bent on “testing the market,” frequently the teams simply had reservations and wanted to see how well those players held up.

The reason this matters so much is that those two samples of players are used to determine the old version of MORP and the FanGraphs’ “Dollars” estimate. Projections for these players are consistently too high, and yet they are being used to price the value of all players. Both of these methods will underestimate the actual amount of dollars paid per win, because they overestimate the number of wins these players will produce. This will be exacerbated when many of these players wind up declining beyond what these methods project.

As I move forward with MORP, I will take a different route to price talent. Knowing these biases, it is apparent that using all players with six years of service time will produce a better sample. This entire method will be revealed further in the coming weeks, but just like it was essential for me to rigorously test whether MORP should be linear in January and February, it was also essential to study the sample used. The results clearly show that using a projection system that is unaware of the context in which players were signed will not accurately price free agents.