There is no group of players whose performances are more volatile than relievers’. Because of their small workload and variable handling, all but the most reliable of them vary wildly in effectiveness from season to season. They are also sensitive to the way they are deployed by managers. Two relievers with identical numbers are going to make differing contributions to the winning effort if one is used in close games and the other only when the team is ahead or behind by a large margin.
Despite the almost random nature of the subject group, PECOTA forecasts relievers each season in the form of Win Expectation above Replacement, Lineup-adjusted, or WXRL. Put simply, WXRL is a measure of wins added by a reliever. As history has long shown, reliever ERA isn’t a helpful guide to their performances because of the brevity of their appearances and the way that inherited runners that they allow to score are accorded to other pitchers. Nor are saves a proper measure, as not all saves are created equal-the textbook save, getting three outs with a three-run lead, isn’t as difficult or important a task as getting three outs with a one-run lead or keeping a tied game in stasis. Therefore, we use WXRL as a measure of a reliever’s contribution to the winning effort.
Since we’re at the season’s half-way point, a quick and dirty way to compare PECOTA‘s WXRL forecast to actual performances from the 2009 season is to take each reliever’s WXRL and double it. In order to screen out small samples, we’ll focus on those relievers who have thrown at least 30 innings this season. Because the group of relievers is vast, we’ll also take a sampling of those at the top of the rankings, those at the bottom, those in the middle, and last year’s leaders.
Pitcher Team G WXRL WXRLx2 PECOTA Diff. David Aardsma Mariners 38 4.1 8.3 0.5 +7.8 Jonathan Broxton Dodgers 37 3.4 6.8 3.2 +3.6 Heath Bell Padres 34 3.4 6.8 1.8 +5.0 Ramon Troncoso Dodgers 38 3.2 6.4 0.4 +5.9 Jonathan Papelbon Red Sox 37 3.1 6.2 3.4 +2.8 Ryan Franklin Cardinals 32 3.1 6.1 0.6 +5.5 Rafael Soriano Braves 38 3.0 6.0 0.9 +5.1 Francisco Cordero Reds 36 2.8 5.7 1.4 +4.3 Mariano Rivera Yankees 34 2.8 5.6 3.7 +1.9
There are many relievers having great seasons right now-last year, only six relievers exceeded a WXRL of 5.0. PECOTA was a bit surprised by this, as well as the composition of the group. It figured that Mariano Rivera would have a good year, but as he was entering into his age-39 season, it anticipated a slight decline from his pace of a year ago. The miss here, the narrowest in the group, is unsurprising; PECOTA is a bit flummoxed by Rivera, whom it rates as a historically unique pitcher. The system generates its forecasts by looking for comparable pitchers, and it ranks these based on a scoring system; a player with a score a score of 50 or higher has a very common typology, while a player with a score of 20 or lower is historically unusual. Rivera’s score is eight.
The other misses in this group are attributable to changes in usage. The rise of David Aardsma from journeyman first-round bust to closer was unanticipated, even by the Mariners. Ryan Franklin was looked upon as a middle reliever. PECOTA saw Rafael Soriano as pitching effectively, but did not project him as the closer and worried about his durability. With Francisco Cordero’s age and so-so 2008, PECOTA didn’t foresee a dominant performance.
We turn to the bottom of the list, this year’s least effective relievers with a substantial number of innings.
Pitcher Team G WXRL WXRLx2 PECOTA Diff. Jensen Lewis Indians 28 -0.4 -0.8 0.7 -1.5 Peter Moylan Braves 45 -0.4 -0.8 0.6 -1.4 Joel Zumaya Tigers 27 -0.4 -0.9 0.5 -1.4 Manuel Corpas Rockies 33 -0.5 -0.9 1.1 -2.0 Luis Ayala Twins 28 -0.5 -1.1 0.3 -1.4 Joel Hanrahan Nationals 34 -0.6 -1.2 0.6 -1.8 Brandon League Blue Jays 37 -0.7 -1.4 0.8 -2.2 Cla Meredith Padres 33 -0.7 -1.5 1.0 -2.5 Miguel Batista Mariners 32 -0.8 -1.5 0.6 -2.1 Brad Lidge Phillies 33 -1.5 -3.0 1.9 -4.9
Here the volatility of the reliever population is even more pronounced. While PECOTA didn’t anticipate great things from any of these pitchers, it did generally figure that they would make mildly positive contributions. The most dramatic miss of the group is with Phillies closer Brad Lidge. The system foresaw a steep decline in effectiveness for Lidge-perfection doesn’t last-but didn’t anticipate the degree of his fall. Otherwise, we have a collection of mostly non-closers, middle relievers and set-up men; these pitchers are the most inconsistent of performers within an inconsistent class. Note the sensitivity to context of WXRL; most of these relievers have not been pounded. Miguel Batista, for example, looks to be superficially strong with a 3.22 ERA and 8.9 H/9. It’s when we look at his performance with runners on base that Batista’s WXRL starts to make sense: Batista has inherited seven baserunners from other pitchers this year, and five of them have scored, a rate more than twice as high as the major league average-relievers allow about a third of inherited runners to score, but Batista has sent 71 percent of them home.
A sample around the median WXRL marks:
Pitcher Team G WXRL WXRLx2 PECOTA Diff. Manny Delcarmen Red Sox 32 0.7 1.3 1.1 +0.2 Evan Meek Pirates 26 0.6 1.3 -0.1 +1.2 Brandon Medders Giants 32 0.6 1.2 0.2 +1.0 Brad Ziegler Athletics 34 0.6 1.2 0.6 +0.6 Burke Badenhop Marlins 20 0.6 1.2 0.2 +1.0 Ryan Madson Phillies 43 0.6 1.1 1.0 +0.1 Jason Jennings Rangers 29 0.5 1.1 0.4 +0.7 Shawn Camp Blue Jays 29 0.5 1.1 0.6 +0.5 Justin Miller Giants 27 0.5 1.0 0.4 +0.6 Mark Lowe Mariners 37 0.5 1.0 0.4 +0.6
On the whole, PECOTA was right on this group, expecting them to be somewhere in the middle of the reliever pack, and getting what it called for. The biggest gap here is that of Pittsburgh’s Evan Meek-PECOTA looked at Meek’s meek performance last year and didn’t see much to love about the former Rule 5 pick. Given Meek’s rate of 6.6 walks per nine innings, it seems likely that Meek will be catching up to his projection in the course of time.
Finally, the aforementioned group of relievers that were over 5.0 wins added in 2008. We’ve already accounted for the top two, Brad Lidge and Mariano Rivera, which leaves four others:
Pitcher Team G WXRL WXRLx2 PECOTA Diff. Joe Nathan Twins 35 2.7 5.4 3.1 +2.3 Francisco Rodriguez Mets 38 2.1 4.3 3.3 +0.9 Carlos Marmol Cubs 44 1.8 3.6 2.3 +1.3 Joakim Soria Royals 21 1.7 3.4 2.6 +0.9
Speaking broadly, we’ve seen consistency from Nathan and K-Rod, while Marmol and Soria have taken a step back, the former due to infection with the wildness bug, the latter due to injury, which has truncated his playing time. The broadest miss, Nathan’s forecast, is caused by a similar effect to what happened with Rivera: he’s been so good for so long that PECOTA has difficulty finding pitchers to compare him to. Indeed, Rivera is seventh on Nathan’s list of comparables, though the system doesn’t seem them as being close relatives in its taxonomy of pitchers. Nathan’s similarity score is 13, so the system casts a wide net and takes its best guess.
Taken in the large view, WXRL is a terrifically sensitive statistic, and at the end of the season its range tends to be fairly narrow. In more than 50 years, there have only been 49 individual pitcher seasons in which a reliever exceeded a WXRL of five and change, and most years the league leaders can be found in the 5.50-7.50 range. This means that as the season proceeds it wouldn’t take much of an improvement by pitchers underperforming their PECOTAs, or much of a regression by those over-performing them, to bring the projections and performances into alignment. Relievers have little margin for error, and each success or failure swings the statistics dramatically. PECOTA would seem to have done a good job so far in aiming darts at a board as big as the universe.
A version of this story originally appeared on ESPN Insider 
.
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.
Subscribe now
A high or low strand rate isn't sustainable, for example, so I don't see why we bother measuring it.
WXRL rewards pitchers for good defense behind them, and for lousy strategies executed by the opposing manager: those are luck; the pitcher didn't cause those.
Since wxrl measures how the pitcher has changed win expectancy, it tells us how effective a reliever has been. If a reliever comes in with the bases loaded and no outs, and then strikes out the side, shouldn't he get credit for that effectiveness? We shouldn't say, "His strand rate for that inning was 100%, so he just got lucky."
And if we can measure his past effectiveness, that gives us a stat with which we can project future effectiveness, no? And if we project future wxrl, we can take into account the luck factor you are mentioning.
It depends on why you're looking at him. If you want to know, in hindsight, how much he contributed to team wins and losses, then yes. If you want to predict how useful he's going to be in the future, then not much. If you want to know whether to draft him for your Diamond Mind or Strat-o-Matic team, then not at all.
I'll echo the original poster's plaint, which I have aired here myself in the past: BP doesn't publish any pitcher metrics that aren't either context-dependent (WXRL) or outcome-based (VORP, ERA, SNWL, etc.). There's nothing that is based solely on the outcomes of individual plate appearances.
Those aren't systematic cutoffs?
I would love to see some form of a phylogenetic tree based on PECOTA's similarity indices. It'd indicate nothing, as we already know that pitchers didn't develop their skills through natural selection across the generations, but it would be fun. I don't suppose you have any bioinformaticians on the staff?