PECOTA Takes on Prospects: Introduction: I Now Pronounce You Scout and Stat

When I reread Nate Silver’s PECOTA Takes on Prospects series, three themes emerged. One, minor-league statistics are pretty damn good at predicting future performance. Two, so many factors can derail a prediction, particularly for young prospects. Three—which doubles as a disclaimer for this series—I’m not Nate Silver. Apologies in advance.

In the two-year run of his series, Silver, using stats alone, attempted to quantify the future value of prospects with rookie eligibility. He rated each prospect with a metric named UPSIDE and parsed his rankings, explaining why, for example, Dustin Pedroia deserved no. 4 prospect status in 2006, after just 538 (whoa) plate appearances above High-A. Or why UPSIDE projected Alexi Casilla for an excellent five years at shortstop and Troy Tulowitzki for just a decent future. We’ll be doing the same during the next few weeks, aided by Rob McQuown’s revival of Silver’s models.

UPSIDE was conceived to provide a complementary perspective to scouting rankings. It’s not necessarily a better perspective (in fact, it’s not a better perspective)—it’s simply a process that chooses to rely solely on rigorous calculation. Obviously, we should avoid numbers-only approaches when evaluating players—prospects are far more nuanced than their stats and quantifiable attributes. But this extreme approach does provide some benefits. Scouting valuations, while comprehensive, might overweight certain factors. They might say, misadjust on league and level effects. We know the California League is a hitter’s league, but how much of a hitter’s league is it? How much better is the competition in the Eastern League? And consequently, how much do we debit and credit batters and pitchers within our rankings?

This isn’t the fault of scouts, of course. It’s just humanly impossible to consider park, league, and competition-level effects for every player. We know the effects exist, but the adjustments we apply in our head are rarely precise. This is the primary argument for a statistically driven prospect ranking. PECOTA weights such factors objectively, while incorporating additional calculi like position, aging curves, and major-league equivalencies (Davenport translations). PECOTA digests all of these inputs, generates an UPSIDE value for each prospect, and forms a ranked list.

This stat-driven approach not only offers a different perspective, but prompts further questions that may lead to our better understanding of a prospect. The presentation of UPSIDE will be a single figure—which, by the way, is available on the player cards—and when that figure defies the prospect’s scouting diary, we’ll delve into the machinations behind UPSIDE. Is he performing especially well for his age at a high minor-league level? Will his skillset falter against better competition? Does PECOTA project a flat development curve for him?

Models have flaws too. They miss certain attributes. They are emotionless machines that do not discriminate between players (or more accurately, player IDs). This is where scouting comes in: Why, for example, do 185 players rank above Austin Hedges in UPSIDE, who ranks 18th on Jason Parks’ Top 101? While PECOTA considers age, body type, handedness, past performance, environment, and much more, variables like pitch mix, position flexibility, mental acumen, injuries, and makeup have yet to be incorporated. Models can be powerful, but they’re just simplified versions of reality. We can reference BP’s prospect experts to explain the complexities that PECOTA misses. Together, we hope they’ll provide the most complete view of today’s prospect landscape.

Prospects have upside, but how much UPSIDE?
UPSIDE measures the player’s likelihood of above-average performance while under his parent club’s financial control. For players aged 24 or higher, that’s their next five years. For younger players, it’s the peak projected five-year window (PEAK) through their age-28 season (i.e. whichever is highest between age 20-24, 21-25, etc. to their 24-28 window). Why age 28? As Silver explained in 2007, “This is the real fruit of the unforgiving labor of scouting and development: getting impact performances from players who are still cheap under the reserve clause, or in arbitration.”

With PECOTA as its backbone, UPSIDE calculates these peaks by first finding the prospect’s top 20 Comparable Players. The grittiest calculations arise from here: Each prospect is compared to every baseball player season in our database based on baseball attributes and performance. Each comparison is given a similarity score; we take the 20 highest. The higher a player’s overall Similarity Index, the more confidence we can have in his UPSIDE rating.

Once UPSIDE determines those 20 comparables, it takes the sum of their PEAK above-average WARP values, doubles it, and weights them by similarity (higher similarity score, higher weighting). Basically, UPSIDE determines prospect upside by looking at similar players’ peaks. It disregards below-average performances as they contradict the definition of upside. Here’s Nate again:

The cost of employing a prospect is relatively low, both in terms of financial outlay and opportunity cost (a player can simply be left in the minors if he’s not good enough for MLB), so assigning negative points for a below-average or below-replacement level performance isn’t quite fair.

We’ve modified the definition of UPSIDE from our initial release, which used the sum of PEAK non-negative WARP values. Above-average reflects the UPSIDE definition better than above-replacement: It takes much more skill to clear an average WARP than zero WARP, and that skill would indicate the presence of upside.

With the methodology covered, let’s turn to the rating scale. We interpret UPSIDE by the following chart, borrowed from Nate (and Kevin Goldstein) once more:

With that in mind, let’s jump into PECOTA’s Top 100. This ranking is generated solely by PECOTA and its UPSIDE algorithm, with no qualitative adjustments applied. It’s comprised of rookie-eligible players under age 27 and excludes Japanese imports. Jason Parks’ rating of the prospect is listed for comparison, and pitcher rows are shaded.

PECOTA rates seven prospects as “Excellent” and 52 as “Very Good,” cutting off the fallen Jonathan Singleton as its first “Good” prospect. No surprise up top: Byron Buxton takes the number-one UPSIDE rating, which tends to happen when Mike Trout is your primary comparable. Oscar Taveras follows him, with Colby Rasmus, Wil Myers, and Adam Jones as his first three comparables. As indicated by the rating scale, PECOTA expects long major-league careers out of Buxton and Taveras.

Overall, PECOTA’s top 100 contains 60 position players and 40 pitchers. Of the position players, UPSIDE, as it has in the past, rates shortstops and center fielders quite well. It may actually be overrating them, if we contemplate the trajectory of minor leaguers. PECOTA tends to compare players to others at the same position, but a shortstop in Low-A today might not be a shortstop in the major leagues, because he might not have the range for the position. Nevertheless, teams will keep that potential shortstop at that position in the minors, in hopes that he’ll develop at the position—he has positional upside, after all. That raises the UPSIDE of some shortstops and center fielders, because PECOTA doesn’t necessarily know who will eventually be forced to second base or right field. Attrition at the position happens slowly, and those who make it—the Starlin Castros, the Austin Jacksons—become comparables for current minor leaguers.

PECOTA projects 213 prospects, starting with Singleton, as “Good,” giving them a “reasonable chance at a meaningful major-league career, but only an outside chance at stardom.” Forty-one of them close this top-100 list, and the names are a mix of young and old, hitters and pitchers, scout-adored and not. By its definition of “Good,” PECOTA tells us that a few of these prospects will become stars. Scouts have their eyes on a few, but PECOTA has its inclinations too—prospects it nearly pushes into “Very Good” territory without much scout support, for instance. Here, UPSIDE becomes a diamond-mining tool: Of these scout-neglected prospects, who have the best arguments for stardom based on their stats? This is perhaps the most interesting application of UPSIDE—to speculate about which of the relative no-names will eventually become big names.

But aside from analyzing PECOTA’s likes and dislikes, we’re also interested in comparing its rankings to qualitative ones, such as Jason Parks’ Top 101. This exercise can reveal what the model misses, which prospects may be underrated scouting-wise, and which both scouts and stat agree on. With two lists, let’s observe how they overlap:

• 34 appear on both lists
• 66 appear exclusively on PECOTA’s top 100
• 66 appear exclusively on Parks’ top 101

Here we observe the nature of a purely stat-driven prospect ranking: It disagrees with a scout-based ranking two-thirds of the time. I appreciate this large figure—it shows how naïve a stat-driven list can be, given that scouts have access both to stats and to their own insight about players. It stresses that PECOTA thinks certain players aren’t getting enough credit, leading us to ask why. It also pushes us to consider how to improve PECOTA as a model.

We’ll examine the disagreements between PECOTA and BP’s Prospect Staff throughout the series, position by position and player by player.

The Hybrid Top 60
We can synthesize the Baseball Prospectus minds by combining both PECOTA’s and Jason Parks’ lists into one “BP” rank. Below, I’ve taken the geometric mean of the PECOTA Top 100 and Parks Top 101 ranking for each player. If the prospect didn’t make Parks’ 101, I assigned him a default Parks ranking of 150.

I also capped PECOTA rankings at 150, so anyone exceeding that number receives a PECOTA ranking of 150. This minimizes any outlying factors PECOTA couldn’t account for. For example, Lucas Giolito has a PECOTA ranking of 3,409 because of limited playing time; we’d rather not dock him for that. So for the purposes of the Hybrid Top 60, he sits in a 6,669-way tie as 150th on PECOTA’s rankings (that’s about how many players we ran PECOTAs for).

The formula in mathematical terms:

While the inclusion of Parks’ grades balances this list into a respectable ranking, the influence of PECOTA is hardly lost. Note, for example, Mets’ catcher Kevin Plawecki, whom PECOTA quite likes; he retains the no. 11 ranking despite missing Parks’ top 101. Small-sample pitchers like Giolito and Mark Appel surface on this list too.

We’ll talk more about Plawecki when PECOTA takes on catching prospects next time.