Skewed Left: The Good and the Bad News About Instant Replay’s Spring Trial

I give it one year until there’s a fundamental change to Major League Baseball’s new replay system.

I’d give it even less time if we weren’t less than three weeks from baseball season. While MLB certainly isn’t opposed to going seat-of-the-pants on this, the league probably won’t want to make another major change so soon.

But in a year, I’m counting on it.

What we’ve learned from watching the first two weeks of replay review trials in spring training is that the lack of a deterrent is going to be a problem. “Don’t overanalyze spring training stats” might apply here too, but we already saw this coming. Tuesday’s Mariners-Angels game saw the 21st challenge of spring training and the end to a streak that you could see as either embarrassing for the game or good for the umpires, who were 20-for-20 in upholding the other 20 challenged calls.

And why not issue challenges that are unlikely to be upheld? In their January piece at Baseball Prospectus, Dan Brooks and Russell Carleton showed that there’s almost no reason to hold back from challenging anything. The only deterrent is running out of challenges, which would require three close calls in six innings to work against you, and that never happens.

If baseball was looking for an encouraging soft launch of its product, this hasn’t been it. Not even so much for the sideshows or for Joe Maddon already trying to beat the system, because you knew there’d be some of that, but for the managers’ 0-for-20 start, which was also somewhat predictable.

So I will begin here where Dan and Russell left off in January and examine the possible deterrents to challenging every play that there’s even a minuscule chance of getting reversed.

In football, which was the first major American sport to institute challenge-based replay, there is a clear deterrent. Lose a challenge, lose a timeout. You get three per half, and they can be extremely valuable on a final drive, saving up to 40 seconds apiece. But there’s no real equivalent in baseball, so one has to get creative, and that’s what Dan and Russell did in their last paragraph.

We suggest an out, either added to the current inning or added to the next one. Even this penalty would strongly favor managers challenging plays that they were convinced had been called wrong, correcting obvious mistakes, which was the intended point of the challenge system in the first place.

An out is a lot. According to our stats, it works out to be worth, on average, 0.28 runs, which is definitely a deterrent. The obvious issue is how to handle the out in box scores and such. Baseball is prepared for something like this. There is a batting-out-of-order penalty of an out, which is credited to a pitcher who didn’t really record it.

But here, do you give the out to the next batter up and make him then go to the end of the line? Would that change whether a manager would challenge a play, not wanting to take a chance at burning a good hitter but calling an out for a bad hitter expendable? Or is it just a phantom out that leaves the batting order intact?

And would something worth a quarter of a run or more be too punitive, especially for a play where the only thing preventing a reversal might be a camera angle?

So here are some deterrent alternatives less drastic than a full out.

A ball or a strike
The natural place to go if you feel an out is too much would be to one ball or one strike on the next hitter. Sure this would unfairly punish a hitter or an individual pitcher, but not by putting him into a hole so deep that he couldn’t climb out and make it look like nothing happened.

But is it enough?

Again, Harry Pavlidis came through with some numbers for the ball and strike in addition to his stat on the value of an out. On average, going to 1-0 costs the pitching team 0.036 runs, while going to 0-1 costs the hitting team 0.043 runs, though this could fluctuate in specific cases depending on who the hitter is, how he does with two strikes, etc. (As an aside, these numbers are put together for some of the work we’ve done on catcher framing, hence the .08 maximum value for framing a ball into a strike in Table 2 of the Pavlidis/Brooks must-read on the subject.)

There’s a small bit of asymmetry there, which would make different sides slightly more or less interested in challenging, but the overall takeaway is the penalty. It’s about one-seventh that of an out, with much less messy housekeeping. I’d argue that if all you’re looking for is some deterrent, this is some deterrent without being overly punitive.

Losing a player
No, we’re not talking about a power play situation where you have to play with eight fielders, which would be something straight out of the Effectively Wild emailer dreamscape. But as Russell suggested, lose a challenge, and your opponent could pick a player on your bench who wouldn’t be able to come into the game. Is making the challenge worth perhaps losing your closer if it’s a close game?

My idea in this category comes from the not-too-distant future of the universal designated hitter, where the unsuccessful challenge would cost you your DH spot for the rest of the game. As the game got later, that would mean less and less, but it would still be a large deterrent, especially if you feared extra innings.

These things, like the timeout in football, are much less measurable in terms of run expectancy. But there is a problem you don’t run into with changing the count or the innings: Somebody could get hurt. The second you enact a move that leaves a pitcher in for an extra couple of batters because the team is minus its LOOGY, and the guy forced to stay in winds up on the operating table a few months later, it will be Hot Take City. And the second a reliever is batting because the DH has been challenged away and that reliever goes Chien-Ming Wang on the basepaths, that will be the end of accountability for your challenges.

Use the third category of challenge results
We consider a challenge to be of dual outcomes—you win it or you lose it. But the language and MLB’s system of keeping records don’t classify them that way. There are actually three possible results, and therein could lie another way to enforce this.

We obtained from MLB a copy of their records from this spring’s challenges through the close of play Tuesday, and they are actually divided into three categories: “Overturned,” “Confirmed,” and a third category—“Stands.”

Of the 23 challenges that got off the ground (one was nullified due to a power failure in the machinery), one was overturned, seven were confirmed with visual evidence to support the umpire’s call, and the other 15 simply stood, meaning that there was a lack of evidence to overturn it.

Within this separation is a chance to punish what we want to punish. Perhaps leave it as no penalty or just a ball/strike for a call that simply stands—you don’t want to be screwed if you’re right and there’s just no good angle to show that you’re right. And make a severe penalty like an out for plays on which the replay clearly shows confirmation of the call.

This would allow for the deterrent without being punitive when a play really is close and worth challenging.

Remove one challenge
This is the simplest possible deterrent. Right now, the thinking goes that the disincentive is the possibility of running out of challenges, but this is an extreme scenario that’s not really worth fearing, as the research showed. But if instead of two in the first six innings, it went to one that you got to keep if you were right, there really would be some second thought about frivolous challenges.

***

There is some very good news in the spring training data that we have: Instant replay hasn’t been much of a time-suck on the game. From the MLB document, we have good time data on 21 of the 24 challenges (omitting one with no timing because of broadcast problems, one where the timing was an outlier because of an ongoing pitching change, and the aforementioned power failure). The challenges so far have averaged:

35 seconds from call to challenge (standard deviation 19 seconds)

1:20 on the headset, (s.d. 31 seconds)

2:59 to the next pitch, (s.d. 37 seconds)

The ubiquitous comparisons of a challenge time to the argument time are extremely silly without making the leap of a 1-to-1 correspondence between times we have replay and times we would have had the argument. But we do know that if we assume that 2:59 to the next pitch represents a delay of about two minutes, we’ve lost about 48 minutes to challenges so far this spring. That’s over the span of exactly 200 games from which the data was pulled. So we’re losing between 14 and 15 seconds per game to replay, which is nothing.

One would think that in spring training, we’d see challenges only when it’s most egregious (except for the time when a manager was begged to do it). After all, in the regular season, challenging is no risk, high reward, and in spring training it’s no risk, no reward. Yet managers are 1-for-23, and while this is probably just a hot streak from umpires or some experimentation by the skippers, there will be downward pressure on the percentages caused by higher potential gains in the regular season, and still no consequences.

In the NFL, the coaches’ success rate in getting calls overturned is roughly 40 percent. Baseball’s doesn’t need to be that high to be effective, especially if umpires can get more efficient. But if it’s down in the single digits or the teens, and we get even short amounts of time wasted for what feels like nothing, then it will be time for one of those deterrents very soon.