Prospectus Toolbox: Reliever Tools

Welcome back to Prospectus Toolbox. I spent the first two editions talking about the replacement level, focusing on two stats-VORP and WARP-primarily as they apply to position players. Both statistics can be and are also used to measure pitching performance, and while this time out we’re going to focus on pitching metrics, we’ll be particularly looking at the ones that deal with relief pitchers.

Relievers have played an increasingly large role in the game. This year alone, relievers have pitched 33.8 percent of the innings played in Major League Baseball. Back at the dawn of our play-by-play database, 1959, relievers only contributed 27.7 percent of the total innings pitched. By 1979, the figure had risen to 28.9 percent, and by 1999 it was 33.7 percent, or around where we’re at today. Despite the growing emphasis on relief pitching, the mainstream statistics used to describe relievers’ success remain rather crude. On Sunday, John Perrotto bemoaned the lack of consensus as to what number of saves should open the gates to Cooperstown. Part of the reason that there is no consensus is because the save is a severely flawed statistic. Despite the statistic’s name, saves are not necessarily awarded to the reliever who contributes to his team when it needs the most help, but rather to the reliever who finishes the game with a small lead. The statistic’s influence has resulted in managers sometimes using their best relievers in accordance to the rules governing saves, rather than the rules governing common sense.

If saves are such a bad statistic, then what tools should we use to evaluate relievers’ performances? When evaluating offense, we tend to emphasize components (hits, walks, home runs, outs made) over results (runs, RBI). That’s because we’re usually trying to isolate one hitter’s performance from those of the batter before him and the batter behind him. The pitcher, on the other hand, stands alone on the mound, and for better or for worse, we tend to blame him for anything that happens in the game while he’s up there. The best-known pitching stat, earned run average, is results-oriented: it doesn’t matter if a pitcher’s earned runs come on solo homers or in a soft barrage of singles and walks, all that matters is how many earned runs have scored. However, ERA isn’t the ideal statistic for relievers, either, in large part because relievers often come into the game with another pitcher’s runners already on base, and they sometimes leave runners of their own on base for the next reliever on their team to deal with. Any good reliever evaluation stat would have to account for how well the reliever prevents those inherited runs from scoring, and what sort of situation he tends to leave for the next reliever on his team to clean up.

The advanced relief statistics we use here take an extremely result-oriented approach to a relief pitcher’s contributions. Using different methodologies, these statistics look at the game situation when the reliever enters the game, and compare that to the game situation when the reliever leaves the game (or the game ends), without much concern over how we got from Point A to Point B. That comparison creates a positive or negative value that represents how well (or poorly) the reliever helped his team by preventing runs from scoring.

One reliever statistic which you might be familiar with because it’s used extensively on this site is WXRL. WXRL is a metric developed by Keith Woolner which is based on a Win Expectation framework. Win Expectation has its own statistical report (the Win Expectancy Matrix), and is a pretty complicated topic, so I’ll give you a bare-bones, no-math explanation. Win Expectation breaks down each game situation-inning, score, number of outs, number of runners on base, and which bases they’re on-that occurs in the major leagues, all to measure how the transition from one situation to another alters a team’s chance of winning the game. So, within this framework, a pitcher who enters a game in a classic “save situation”-ninth inning, three-run lead, bases empty-increases his team’s win expectancy if he makes it through the ninth without giving up the lead, albeit by a small amount, since the chances of victory with a three-run lead were pretty good to begin with. If the pitcher comes into the ninth with a one-run lead, no outs, and two men on, and gets out of the jam, his contribution to the team’s win expectancy is considerably greater.

In WXRL the raw change in Win Expectation (the “WX” in WXRL) from the reliever’s entrance to his departure is adjusted for the replacement level (the “R” in WXRL) and for strength of the opposing lineup (the “L”). Here are the current WXRL leaders:

Pitcher           W   L  SV  BS  IP     RA+    WXRL    LEV
Takashi Saito     1   0  16   0  24.0   3.07   2.904   1.86
Francisco Cordero 0   0  21   0  23.1   2.11   2.702   1.49
Al Reyes          1   0  14   0  25.2   2.21   2.627   1.73
Rafael Soriano    1   0   5   0  25.0   1.85   2.600   1.53
Scott Linebrink   1   1   1   1  25.2   2.14   2.268   2.23

The qualities of WXRL to keep in mind:

• WXRL is a counting stat, measured in wins.
• All WXRL values are above replacement level-any below replacement-level performances have negative values.
• WXRL accounts for inherited and bequeathed runners.
• Leverage matters. A reliever who comes into high-leverage situations will accumulate more WXRL than a player who pitches just as well in garbage time.
• WXRL can be found in the Relievers Expected Wins Added Report, or in the Pitcher Season, Pitcher Team Year, or Team Pitching custom reports.

Now, WXRL is quite popular around here, so much so that it has overshadowed a previously favored in-house relief pitcher statistic, Adjusted Runs Prevented (ARP). Originally developed by Michael Wolverton, ARP measures a reliever’s contribution using the Run Expectancy Matrix. For each base-out situation, the Run Expectancy Matrix tells us how many runs are expected to score in that inning. So far in 2007, if you have runners on first and second and no outs, it’s expected that the team on offense will score 1.45354 runs. If the team on offense successfully sacrifices, that creates a situation with men on second and third and one out; at that point, the number of runs expected to score increases slightly, to 1.45951. A relief pitcher who takes over that second and third, one-out situation, and only allows one run to score, would be considered to have prevented 0.45951 runs from scoring over what an average relief pitcher would have done. Those prevented runs are adjusted for park and league to make ARP.

How is this different from WXRL? The Win Expectancy and Run Expectancy matrices are calculated in completely different ways, so the two statistics are measuring different things. The biggest difference is that in a Run Expectancy context, every inning the reliever works is created equal-it doesn’t matter if the pitcher’s team is tied or trailing by ten runs, the value of the reliever’s performance is based on the base runners and outs when he takes the mound that inning.

You can see the difference by checking out the league leaders in ARP. While the top of the WXRL list is dominated by closers, the ARP list is a little more diverse:

Pitcher         W   L  SV  BS  IP     RA+    ARP    LEV
Pat Neshek      3   0   0   0  27.2   3.56   16.0   1.30
Heath Bell      0   2   0   1  33.1   4.16   15.2   1.34
Matt Guerrier   1   2   0   0  33.2   2.90   14.4   1.10
Jeremy Accardo  1   0   7   1  25.1   4.39   13.9   1.42
J.J. Putz       0   0  14   0  25.1   3.16   13.6   1.14

As you can see in the last column, marked “LEV,” all the ARP leaders have been used in lower-leverage situations than the WXRL leaders. Some qualities of ARP to remember:

Further Reading

Michael Wolverton, What the “R” Column Doesn’t Tell You: Evaluating Relievers by Prevention of Expected Runs: This article sets out the technical details of ARP and reliever run expectancy.

Keith Woolner, “An Analytical Framework for Win Expectancy,” in Baseball Prospectus 2005 (Workman, 2005): This article introduces the Win Expectancy framework used by Baseball Prospectus, and the sets out the technical aspects of the WXRL statistic.

Keith Woolner, “Are Teams Letting Their Closers Go To Waste?” in Baseball Between the Numbers (Basic Books, 2006): This article discusses both the run expectation and win expectation frameworks, putting both in the context of the optimal time for teams to use their ace relievers.