Between The Numbers: Fact-Checking Scott Boras

Combing through Scott Boras' statements for inaccuracies is a little like tilting at windmills (or so I assume—it's been years since I've seen a windmill, let alone tilted at one). That’s because Boras' greatest ambition isn't impeccable candor; it's getting the most money for his clients (and by extension, of course, himself). Telling the truth is often a good way to get paid, since no general manager likes to be lied to. Sometimes, though, the best way for an agent to stretch his wallet is to stretch the truth. That’s why every offseason, each team can expect to receive a hefty booklet about the latest big Boras free agent, explaining why Oliver Perez is the second coming of Randy Johnson or how Prince Fielder isn't fat, he's just big-boned. There's nothing wrong with these tall tales, so put down your pitchforks. Boras is just doing his job, and he’s doing it better than anyone else. (Just ask Fielder.)

Still, it doesn't hurt to hold him accountable for some of his more glaring leaps of logic. That’s why alarm bells went off in my head when I read a quote of his from a couple weeks ago:

As many as eight major league teams have taken great risk in the closer role. The game has shown many times that teams need closers with the efficiency to (convert) 85 percent of their save opportunities and the durability to make 60 appearances. Numerous teams didn't follow those metrics as a criteria for a closer this offseason. They turned their back on the closer role.

Conditioned as I am to believe that the importance of closers is generally overemphasized, this stat struck me as unlikely. Boras was still steaming about Ruben Amaro’s version of the events that led to Ryan Madson being deprived of a four-year, $44 million deal from the Phillies and settling for a one-year, $8.5 million “pillow contract” from the Reds instead. It occurred to me that he might've gotten a little hot-headed and twisted the truth, so I investigated. And as it turns out, Boras was completely right. Well, maybe not completely—teams don’t need to have a closer at all, let alone one meeting Boras’ specifications. But the ones with a player who met his criteria were much more successful. From 2001-2011, teams that had a pitcher who earned at least 20 saves*, converted at least 85 percent of his save opportunities, and made at least 60 appearances won at a .530 clip. The others collectively finished .479.

*I made up the “20 saves” part myself, based on the fact that it was hard for me to imagine a modern reliever earning a reputation as a “closer” with fewer saves in a season than that. A bit arbitrary, I admit, but no more so than that 85 percent conversion rate and those 60 appearances.

He almost got me. For a minute, I questioned everything I’ve learned about how much closers matter. I wondered whether they were worth those much-maligned multi-year deals. But only for a minute. Then it dawned on me: when you really boil it down, what Boras is saying is that teams with a good player at a particular position tend to be better than teams without a good player at that position. Well, yeah. This is a lot like one of those meaningless stats we tend to hear every now and then on baseball broadcasts: “in games when so-and-so hit a home run, so-and-so’s team went 23-7.” The stat seems to imply that so-and-so’s homers have some magical ability to propel his team to victory, but the team’s success in those games doesn’t have much to do with the fact that so-and-so hit the homers. All that matters is that someone did. All else being equal—which it is, as far as we know—it shouldn’t come as a surprise that a team does better when a player hits a home run than it does when he doesn’t, regardless of which player it is.

Boras’ stat makes it sound like having a good closer is particularly vital, but theoretically, a team with a good player at any position should possess the same edge. To test that, we just have to pick a position and come up with a suitably arbitrary statistical baseline to go with it. I picked second base, since no one ever thinks about how important it is that a team have a good second baseman (or whether a successful second baseman needs a short memory or a “second baseman mentality”). There were 137 teams from 2001-2011 with a Boras-approved closer, so I played around with the statistical cutoffs for second basemen until I came up with a group of similar size. Over the same span, 128 teams had a second baseman with at least eight home runs, an OBP of at least .315, and at least 500 plate appearances. You know, the famous 8-.315-500 club. And guess what? Teams with a second baseman meeting that description went .522. All others went .483.

That looks a lot like the gap we found between teams with and without a Boras-brand closer. But that's not our only clue that Boras' stat might be suspect. There’s another reason why those closers with high success rates tended to play for good teams: closers on good teams enjoy easier save opportunities. The correlation between a team’s winning percentage and its average lead when its leading save-getter enters the game in a save opportunity is 0.2. That’s a weak correlation, but not a nonexistent one, and what it tells us is that as winning percentages climb, leads get larger and save opportunities get easier. Naturally, as save opportunities get easier, conversion rates rise. Good teams aren’t necessarily good because their closers have high conversion rates. But their closers have high conversion rates in part because they play for good teams.

So what have we learned? First, never try to catch Scott Boras in a lie, since you’re probably wasting your time. Boras’ facts check out. I doubt Scott sat down and ran the numbers himself; more likely, some flunky buried deep within the bowels of the Boras Corporation tinkered with the parameters until they spit out something that seemed compelling. Regardless, Boras deserves plenty of credit for having persuasive facts at his disposal. But that doesn’t mean we should always subscribe to his interpretation of what those facts mean. That way lies overpaying for free agents.

Thanks to Andrew Chong and Rob McQuown for research assistance.