Fantasy Freestyle: PFM vs. the ‘Experts,’ Part One (Hitters)

Sometimes in these pieces, I delve into a long explanation of what I’m going to write and what I’m going to set out to prove. That’s not going to happen today. I’ve got a lot of tables to produce and a tight deadline so I’m just going to dive right into it. Today’s goal is to take a look at the PFM, take a look at expert prices, and determine whether or not I should be using the PFM more as a tool to devise my bid limits or if perhaps I should chuck my bid limits entirely. How’s that for a attention grabbing lead?

Baseball Prospectus fantasy writers get asked about PECOTA a lot. Some subscribers don’t understand why they should bother with my bid limits when the PFM serves the same purpose as my bid limits but with a more mathematical bent. Others find my bid limits interesting but think that the PFM should be featured more as we approach Draft Day.

I talked about the differences between my bid limits and the PFM last year. That linked article was an explanation of why I do things the way I do. It was not an examination of whether or not what I do is worth doing. This piece you are reading will attempt to answer this question.

As I noted in last year’s PFM piece, one thing the PFM does very well is measure how the player pool is going to do on a macro level. What I did not realize is how exceptionally well the PFM does this.

Table 1: 2014 American League Hitter Earnings and Salaries in Groups of 10

You can quibble with individual hitter projections if you like (and I certainly will as we go along), but on a grand scale the PFM has a much better idea of how these players are going to perform in the future than the experts do. The PFM is $3 off in group 11-20, $4 off in group 31-40, and $4 off in group 41-50. This is consistency is pretty amazing.

The story isn’t any different in the National League.

Table 2: 2014 National League Hitter Earnings and Salaries in Groups of 10

The earning distribution is different—particularly in the first three groups—but once again, the PFM has reality surrounded with its recommended bids. Granted, it is pretty far off on the top group of hitters, but even the PFM has nothing on the experts.

The PFM’s aversion to the most expensive hitters is starker when you look at it on a case-by-case basis.

Table 3: Top 20 Salaries, 2014 AL and NL Hitters with PFM

For a description of what each column in Table 3 (and all subsequent tables) means, please read the descriptions from my retrospective player valuation series.

In Table 3, PECOTA and the PFM absolutely destroy the expert market. CBS, LABR, Tout Wars, Peter Kreutzer, and I are in lockstep while the PFM almost completely nails what these players are going to do as a group. It comes closer to predicting player earnings in 14 out of 20 cases, while the expert market only beats the PFM four times: on Jones, McCutchen, Braun, and Gomez. There are two ties.

The only top 10 AL- or NL-only hitter who turned a profit last year was Gomez. If you had followed the PFMs advice, you only would have walked away with Braun in the expert leagues. That’s still a loss for the experts as far as I’m concerned.

But (for me at least) this is all old news. The PFM is much better at predicting what the best hitters are going to do because it isn’t betting on any outliers. Nearly every player has some degree of failure built into his projection, which is reflected in the PFM. The experts, put simply, aren’t hedging their bets.

Should they be?

For the PFM to be definitively better than the experts, it has to tell me either when to buy or who to buy. I already know that it isn’t giving me any particular advantage in telling me when to buy, based on the evidence in Tables 1 and 2. In the 10 player groups where the PFM spends more money, the players in the AL earn $757 on an $848 investment. This is canceled out in the NL, where the PFM is more aggressive in groupings that earn $1,125 on $1,049 of recommended PFM bids. On the whole, the PFM’s passivity at the top of the player pool isn’t leading to a windfall of big profits on the players the PFM does like.

So this leaves us with the “who to buy” part of the equation. We kind of know based on Table 3 who the players are that the PFM doesn’t like, but are these the only types of guys that the PFM tells us to avoid or are there others?

Table 4: 20 Least Favorable PFM Projections vs. Expert Bids, 2014

There are a few others. But 12 of the 20 hitters on Table 4 are repeaters from Table 3.

For all of the knocks that PECOTA and the PFM have received over the years, Table 4 should be used in Baseball Prospectus’s publicity materials. Allen Craig at $12, Pedro Alvarez at $15, and Alex Rios at $19? These are the kind of predictions that win you leagues if you take them to heart and avoid these players. Abreu was a big miss, but with the exception of him, Moss, and Desmond, Table 4 is a big win for projection systems and staying conservative.

However, as I noted above we know that the market is too aggressive on the most expensive players. Does the PFM’s dour outlook on the best players translate to success on the players where the market is too soft?

Table 5: 20 Most Favorable PFM Projections vs. Expert Bids, 2014

The easy takeaway is that the PFM kicks ass and takes names because it is way ahead of the field on Gordon, Jay, Davis, and Uribe. If you bought Gordon, Jay, and Uribe in your NL-only, it probably didn’t matter what the rest of your roster looked like; you cleaned up even if you merely purchased the rest of your offense at par prices.

The problem is that on the whole this group of hitters stinks. Eleven of the 20 hitters on this list earned $5 or less, and only two of the 20 recommended prices in Table 5 are under $8. If the PFM was rightfully conservative with the top-tier hitters, here it looks like an inebriated businessman throwing money around at two in the morning on the Las Vegas strip. The optimistic viewpoint is that the PFM pointed you toward Gordon, Jay, Davis, and Uribe. The negative viewpoint is that if you take out the $90 these four players earned, the remaining 16 players earned a combined $79 and the PFM recommended paying $176 for them.

The PFM’s recommendation on a high level (don’t spend money on any of the top players) is a sound one. But the PFM has to spend its money somewhere, and isn’t all that much better than the market is at figuring out who the best players are.

Table 6: PFM and Expert Market: Cost and Earnings on hitters with +/- $3 differential

Table 6 lists the players in both the AL and NL where the PFM had a $3-or-greater differential than the expert market in either direction. The P/M/T columns show where the PFM came closer to the actual player earnings (P), where the market came closer (M), or where there were ties (T). I thought about creating a scorecard for every player but decided against it; a $0-2 difference on a player projection is marginal and it seemed arbitrary to score the PFM or the market on players where the valuation was this close.

In Table 6 we see another aspect to love about the PFM: it takes a stand on its preferences. There were a total of 336 hitters purchased in AL and NL-only leagues combined; the PFM has a $3-or-higher swing on 57 percent of those hitters. Unlike the expert leagues—which hardly have any variance whatsoever—the PFM has guts.

This doesn’t necessarily translate into better projections. The advantage that the PFM gains on the most expensive hitters (AL and NL Experts Favorable) is more than canceled out by the hitters where the PFM takes a stand (AL and NL PFM Favorable). Overall, the amount of money that the PFM recommends spending nearly matches what the expert leagues recommend spending. On a hitter-by-hitter basis, the market narrowly edges out the PFM, 93-92. Put simply, when you get past the most expensive hitters, the PFM’s precision isn’t any better than the market’s guesses.

This isn’t a problem with the top hitters, but for the hitters in the middle and at the bottom the PFM is giving all of its gains away. In a perfect world, the PFM would tell you to stop overspending on the top hitters and point you to the right hitters much more frequently than they do not. But it doesn’t. In fact, when you take the hitters with $30-plus salaries out, the market comes closer to predicting earnings than the PFM does.

So what does this all mean? Should we throw the PFM out because of this?

Of course not. The advice to stay away from the top hitters in auction formats is sound, particularly as we continue to see offense lag behind pitching. The odds of a hitter cracking $40 is very low, and many of the $30-plus salaried hitters are going to lose money. We should stop pushing $30-plus on the elite hitters. The separation isn’t there anymore. Mike Trout and Miguel Cabrera are sinking along with the rest of the pool.

But where the PFM is failing is at the bottom. Part time players don’t have enough of a ceiling to justify $8-10 prices but the PFM continues to insist paying in this range for players like Mayberry. The failure rate on these guys is too high to pay par. If Mayberry disappears, you have budgeted $9 for nothing…and players like Mayberry often do disappear.

The auction market recognizes this and doesn’t pay these guys. But the lesson we should take out of this is not to distribute the money to the top as the expert leagues always do but instead to move it to the middle. A healthy redistribution of your dollars solves both for the problem of the market (overpaying the guys at the top) and the PFMs problem (paying good money for bench players).

How does the PFM do with the pitchers versus the market? I will examine this in Part 2 of this series next week.