Lies, Damned Lies: PECOTA Takes On Prospects, Part One

I’ve spent the better portion of the last two months thinking about how we can do a better job of evaluating the futures of baseball players and, particularly, the futures of young baseball players. It goes without saying that I’ve done this thinking in the vehicle of PECOTA, and essentially every aspect of the system has been rethought and reevaluated. But above and beyond anything PECOTA might have to say, one conclusion has jumped off the page:

When it comes to prospect analysis, a lot of disagreements result from improper valuation, rather than true differences of opinion about how well a particular player plays the game, or how well he is likely to develop.

The Baseball America scouting report on Chin-Lung Hu, a shortstop in the Dodgers chain, claims that he’ll become a player who hits .270 with 10 home runs in the majors, complementing that production with excellent shortstop defense. What does PECOTA think? It has Lu hitting .282 with 7 home runs as a 26-year-old major leaguer in 2010, saving 8-10 runs above an average shortstop with his glove. Suffice it to say, those aren’t the sorts of differences that are going to help to sell Moneyball II.

But exactly how valuable would that sort of performance be? If we accept that performance as gospel, how should we rank Lu among Dodgers prospects? Among shortstop prospects? Among all prospects? Which players would we trade for Chin Hung Lu, and which wouldn’t we?

Michael Bourn is a center fielder who played in the Eastern League last year. BA offers the following scouting report:

Bourn offers the quickness, aptitude and offensive approach required of a leadoff hitter. He’s the system’s best defensive outfielder and also has an above-average arm.

Here too, PECOTA largely agrees with BA’s assessment. Bourn’s speed scores are off the charts, and he rated as a +9 in center field last year. It doesn’t think Bourn’s a sure thing–his strikeout rates are high for a player with his profile, and he almost certainly won’t develop home run power. But it gives him about a 25% chance of getting up to an EqA in the .265/.270 range, heavy on the OBP side and coupled with good baserunning, which would in fact make him an adequate leadoff hitter.

But how does Bourn rate compared to Shane Victorino, the International League MVP who will inherit the fourth outfielder’s job in Philly next year? PECOTA would seem to have reasonable enough expectations for Victorino. It thinks he was playing a little bit over his head at Scranton last year, where he put up a .282 EqA, and pegs his 2006 at an EqA of .266 with plus defense in center field. It doesn’t expect Victorino to get any better from here on out, nor does it see much in the way of star potential–his 90th percentile forecast only gets him up to a .289 EqA.

So we have:

1. A 23-year-old center fielder who has little star potential, but about a 25% chance of developing into a solid regular.
2. A 25-year-old center fielder who has little star potential, but who could be a solid regular right now.

Which of these players is the more valuable commodity? The answer to that seems obvious enough, and PECOTA’s various long-term valuation metrics say that Victorino is perhaps two or three times more valuable than Bourn. But BA has Bourn ranked as the #3 player in the Phillies system, whereas Victorino doesn’t crack in their Top 10 (in one of the weakest systems in baseball).

This is what I mean by improper valuation. There’s relatively little difference of opinion about what sort of prospect Bourn is, and what sort of prospect Victorino is; PECOTA doesn’t have Victornio going all Jonny Gomes on the league or anything like that. But in the rush to put players in some sort of ordinal ranking, we’ve lost sight of the bigger picture.

I’m picking on Baseball America because they can afford to be picked on. BA employs dozens of extremely bright and hard-working people who scout thousands of baseball games across the country and have Rolodexes the size of a small Walmart. They’ve earned their position as the most influential voice in the scouting community.

Besides, this sort of problem is just as pervasive among analysts. In fact, it may be more so; BA, at least, has the advantage of primary source information, whereas the rest of us are left to fight for the scraps and some video clips from the Futures Game. And so there’s an awful lot of debate about whether Dustin Pedroia is going to hit twelve home runs a year in the big leagues or fifteen, and relatively little notice given to the fact that it doesn’t matter very much in the grander scheme of things. The basic technique has been–and I’m not saying we’ve been immune from this–take BA’s list, move some guys upward or downward ten slots, and call it a day. There’s been relatively little attempt at serious, econometrically oriented valuation of prospects, and the overall tenor of the debate resembles an investment bank picking its clients by reading tea leaves and two paragraph business descriptions.

This is why I think PECOTA has significant potential as a prospect ranking tool. It can consider a large number of factors at the same time. It can translate those factors into some quite tangible notions about player value. And it’s completely immune from groupthink. It’s a primary source, albeit in a very different way than scouting information, and it’s objective.

Just how close PECOTA is to fulfilling that potential I’ll leave for you to decide. But, as a result of the improvements I’ve made to the system, I think it’s finally developed to the point where it can produce a credible ranking of prospects, the kind of ranking that we can put into a time capsule, and come back in ten years and compare against other artifacts of its day. I’ll go without describing all the changes that I’ve made to PECOTA, which are discussed at much length in this year’s book. But the most important fixes, from a prospect evaluation point of view, are as follows:

• PECOTA now considers minor league level when selecting its comparables. That is, a Double-A player should ideally be compared to other Double-A players. Sometimes the system fudges; a Double-A player might be compared to a Triple-A player or a guy from the Florida State League. Sometimes, the Double-A player is Delmon Young, and he’s so far ahead of his league that he’s earned the “right” to use some major league comparables. But level is considered (above and beyond the Davenport Translations themselves) and it turns out to make a large amount of difference.

• PECOTA now treats a professional plate appearance as a professional plate appearance, regardless of whether it occurred in the majors or minors. This is a decided improvement over the sort of three-fifths compromise we’d made in the past, which gave partial credit to minor league plate appearances, introducing all sorts of biases while at the same time needlessly compressing the projected playing time of minor league players. We now leave it for the Twins to decide whether Francisco Liriano plays in Rochester or Minneapolis; our assessment of Liriano’s value will place him on a level playing field either way.

There are a lot of presentational improvements too. We now provide an explicit forecast for each of a player’s next five seasons: how many home runs is Prince Fielder going to hit in 2009? We’ve introduced a couple of new charts, most notably the Stars & Scrubs and (my personal favorite) Career Path Analysis. We’ve even made an attempt to quantify a player’s upside. I’m going to be leaning heavily on these new tools in this series of articles, so I’d encourage you to browse through the new cards, as well as the updated glossary, before reading further.

Methodology

I’m going to be ranking prospects by combining two metrics from PECOTA’s peak forecasts, which are designed to assess a player’s medium-to-long term value. The first of these is a player’s peak WARP score, and the second is his peak UPSIDE score. The WARP forecast is the more conservative of the two. It’s focused on value above replacement, and therefore gives somewhat more emphasis to certainty and playing time.

UPSIDE, on the other hand, is focused only the possibility that the player develops into an above average major leaguer. It doesn’t care whether a player winds up riding the major league bench, gets stuck in Double-A, becomes the new Luis Rivas, or goes off to Australia to smoke ganja with Ricky Williams. Each of these outcomes is equally undesirable, and UPSIDE recognizes that.

Because UPSIDE is denominated in runs, and WARP is denominated in wins, we use the following formula to combine the two:

Combined Ranking = Peak WARP * 10 + Peak UPSIDE

That is, we assume that ten runs are equal to one win, the intention being to give equal weight to the two metrics.

You may also notice that, for position players, the WARP rating accounts for his projected defensive prowess, whereas the UPSIDE rating does not (although UPSIDE does account for the player’s position). This is an intentioned feature of the combined rating–essentially, we’re giving half-credit to a player’s defensive forecast. I have a lot of trust in the defensive ratings that Clay turns out for minor league players, and the studies we’ve done internally suggest that they are in fact quite good predictors of major league defensive performance. But considering that a large number of minor league players undergo position changes, that they have to contend with field conditions of varying quality, and that defense is hard enough to get a hold of to begin with, this seems like the fairest thing to do.

Caveats and Limitations

Although PECOTA is very good at what it does, it has some limitations:

• PECOTA is not especially good about accounting for injuries. Although it can make some reasonable inferences based on playing time metrics, it can’t know, for example, if a player missed the first month of the season because his hammy was bothering him, because Scott Boras was his agent, or because he’d been suspended for steroid use.

• PECOTA, since it depends on player comparables, may understate the forecast uncertainty for highly unique players. I will attempt to point out when these guys come up.

• The PECOTAs are very heavily influenced by the Davenport Translations. If you’re unhappy with someone’s forecast, you should look at whether we don’t have him developing as much you think we should (PECOTA jurisdiction), or whether his initial translations weren’t as favorable as you’d like (DT jurisdiction).

• Because the DTs do not translate college or high school performance, or performance from the Arizona and Gulf Coast Leagues, PECOTA cannot produce forecasts for players who have not yet performed above these levels. Thus, no Elvis Andrus. And no Alex Gordon or Justin Upton, either.

• Similarly, although PECOTA is smart about handling players with somewhat limited sample sizes, it is not really designed to handle players with extremely limited sample sizes, particularly position players with fewer than about 150 translated PA in their professional careers, or pitchers with fewer than 50 translated IP. Because the limited playing time guys tend to emerge with unfavorable rankings, I’ve left them alone if they’ve managed to achieve a particular slot anyway (congratulations, Andrew McCutchen). But we shouldn’t hold it against a player like Troy Tulowitzki if he hasn’t.

• Finally, we can only rate a player if we’ve run a PECOTA for him. This means that an author made a request for his forecast in conjunction with this year’s annual book (98% of the time), or I ran his forecast separately after coming across his name elsewhere (2% of the time). Our coverage should be very good–we’ve run more than 1600 PECOTAs this year–but we’re probably missing a handful of unheralded players toward the bottom end of these lists.

Qualifications

In keeping with BP tradition, we’re using Rookie of the Year standards to determine which players are classified as prospects and which aren’t. There’s some confusion about what this means; a RoY candidate must have:

• Fewer than 130 career major league at bats

• Fewer than 50 career major league innings pitched

• Fewer than 45 days spent on a 25-man major league roster in his career, EXCLUDING time on the DL and any days after expansion to 40-man rosters in September.

I don’t want to promise that we’ll bat 1.000 on these classifications. The roster time information can be hard to track down, and I can understand why Baseball America doesn’t bother with it. But we’ve been reasonably careful, and with the possible exception of Bobby Jenks, this year doesn’t feature a lot of controversial cases.

Besides that, I’ll be running supplemental lists of players who have lost their RoY classification, but are (i) no older than 24, or (ii) split time last season between the majors and the minors. Personally, I think the entire distinction between "prospect" and "good young player" is silly, especially when applied to evaluate an organization’s scouting and development capabilities–why punish a team that successfully graduates a player to the big league level? (It also seems that players who make a mid-season debut–guys like Curtis Granderson and Edwin Encarnacion–tend to be systematically underrated by prospect hounds.)

Catchers: Real Prospects

Player                                  WARP       Upside      Combined
1.  Jarrod Saltalamacchia, C, ATL (21)  16.9        137.0       306.4
2.  Josh Willingham, C, FLO (27)        15.3        126.2       279.3

3.  Kenji Jojima, C, SEA (30)           14.0        94.7        234.9
4.  Jeffrey Mathis, C, LAA (23)         14.0        86.3        226.1
5.  Michael Napoli, C, LAA (24)         15.3        71.8        224.7
6.  Ron Paulino, C, PIT (25)            16.1        59.4        220.4
7.  Russell Martin, C, LAN (23)         16.3        52.3        215.4
8.  Neil Walker, C, PIT (20)            14.0        60.9        200.6

9.  Miguel Montero, C, ARI (22)         11.5        59.2        174.5
10. John Jaso, C, TBA (22)              12.0        49.6        169.2
11. Curtis Thigpen, C, TOR (23)         13.6        28.8        164.4
12. Jeffrey Clement, C, SEA (22) **      9.2        59.6        152.0
13. Kelly Shoppach, C, CLE (26)         11.2        39.6        151.2
14. Brandon Snyder, C, BAL (19)         10.7        41.6        148.4
15. Taylor Teagarden, C, TEX (22) **     9.4        39.8        133.4
16. Carlos Ruiz, C, PHI (27)            10.4        20.8        124.9
17. Justin Knoedler, C, SFN (25)         9.8        26.3        123.9
18. Eliezer Alfonzo, C, SFN (27)         9.6        26.8        122.7
19. Kurt Suzuki, C, OAK (22)             9.1        31.1        121.6
20. Francisco Hernandez, C, CHA (20)     9.9        21.3        120.6
21. Christopher Iannetta, C, COL (23)    8.6        30.0        116.1
22. George Kottaras, C, SDN (23)         8.3        28.3        111.0
23. Shawn Riggans, C, TBA (25)           7.6        29.4        105.4

** Fewer than 150 translated PA
Line breaks indicate ratings differences of at least 25 points

I recognize that these numbers, new as they are to all of us, will not mean very much in the abstract. But broadly speaking, it takes a combined rating of about 300 to attain the status of an elite, Top 20 prospect. By that definition, only Jarrod Saltalamacchia qualifies among backstops this year. Salty is a pretty well known commodity, and PECOTA doesn’t buck the consensus, finding a player who projects to hit 20-25 homers per season in the big leagues supported by .350ish on base percentages. The issue, apart from the de rigeur concern about a lack of foot speed, is that catcher almost certainly won’t be his position in the big leagues. Not only is there the Brian McCann problem, but both Saltalamacchia’s translations and reputation rate him as a poor defensive catcher, and he’s tall (6’ 4”) for the position. I’d venture to guess that, if the prospect is a catcher, shortstop, or center fielder, the odds of him playing that position as a 27-year-old big leaguer are quite long unless he enjoys a truly premium defensive reputation.

Josh Willingham is an extreme representative of the Shane Victorino group of players: someone who is not so much a prospect as a player who could be an average-to-good major leaguer in the present tense, without further development. Statheads have been touting this sort of player for a long time. Along the way, we’ve made a few missteps. It’s nice to say that minor league statistics predict major league performance as well as major league statistics do, but it isn’t quite true. Certain statistical profiles–position players with extremely high walk rates unsupported by other skills, soft-tossing pitchers with poor strikeout numbers, players who have a "breakout" year when repeating a level–may reflect an ability to exploit weaker competition, rather than true big league talent. Willingham doesn’t really fit into any of these profiles. While his walk rates are high, they’re supported by abundant power, and while his contact skills are not good, his speed metrics come out surprisingly well. PECOTA has him posting the highest EqA of any National League catcher next year.

I’m fairly comfortable that, on the question of players like Willingham, the scouting side of the aisle does have it wrong. But, to reiterate, the error is not so much in failing to define the commodity appropriately–nobody doubts that Willingham will hit his home runs and draw his walks in the big leagues, and not do much else–but in failing to put it into its proper context. It’s nice to talk about a player’s upside, but by definition, relatively few players will realize their upside. And sometimes, like in Michael Bourn’s case, that upside just isn’t all that exciting to begin with. By contrast, players who have already realized their upside are liable to be underrated, provided that upside means being an average-to-good big league regular, warts and all.

Speaking of warts, it’s hard to identify a player who has more of them than Michael Napoli, a player whose power hitting ability may be even better than Willingham’s, and contact skills may be even worse. Unlike Willingham, however, Napoli is a born-and-bred backstop who fields his position well, and flaws that might end the career of another player can come to be tolerated when a guy plays a legitimate catcher. The $64,000 question is whether they’ll be tolerated by The O.C., which usually prefers eye candy to substance.

Otherwise, this is a pretty boring list. I don’t have any particular feelings about Kenji Jojima’s ranking, other than that this isn’t really an exercise designed to evaluate 30-year-olds. Jeff Clement is a sample size victim. He was able to make up some of the lost ground with a good UPSIDE score, but in practice he belongs somewhere between #2 and #4 on this list. John Jaso is a little bit of a sleeper, but has moved slowly and is harmed a lot by the level adjustment; it says something when Ross Gload is your number one comparable.

Catchers: Almost Prospects

Player                        WARP     Upside    Combined
Joe Mauer, C, MIN (23)        33.9     283.8      622.3
Brian McCann, C, ATL (22)     24.7     210.6      457.5
Dioner Navarro, C, LAN (22)   18.9     131.7      320.2
Ryan Doumit, C, PIT (25)      15.5     120.6      275.2
Yadier Molina, C, SLN (23)    14.7      28.8      175.7
John Buck, C, KCA (25)        11.3      51.1      164.5
Guillermo Quiroz, C, TOR (24) 10.1      56.2      156.8

Before we get on too much of a "where have all the catchers gone" lament, let’s remember the two young players who could represent their leagues in the All-Star game for the next decade to come. It’s rare that PECOTA renders this unambiguously favorable a verdict on a player, but that’s what we get with Joe Mauer, whom this metric regards as the sixth most valuable commodity (prospect or otherwise) in all of baseball. PECOTA is almost as keen on Brian McCann.

Another of the biases that creeps into prospect analysis is what we might call Shiny New Toy Syndrome. Russell Martin excited a lot of people with a good season in Jacksonville, and by most assessments he’s now #1 in the eyes of the Dodgers brass. But Dioner Navarro is a year younger than Martin, and posted a .354 OBP in 200 plate appearances in Dodger Stadium. We’ll get into more discussion about the importance of strikeout rates later, but not to be overlooked is that Navarro almost never fans: PECOTA thinks he’ll be able to sustain batting averages as high as .280-.290, even without much speed down to first. If Navarro can develop power on top of that, we could be talking Ted Simmons. If not, he could still have Mike Scioscia’s career.

On the docket for next week: outfielders and first basemen. After that, the middle infielders, and the real controversy begins.