Best of BP: The Banging Scheme Hurt the Astros As Much As It Helped Them

This article was originally published on Jan. 30.

Some of the biggest lingering questions in the Astros sign-stealing controversy have circled around its efficacy. The excellent reporting by Ken Rosenthal and Evan Drellich, as well as MLB’s investigation, established beyond any reasonable doubt that Houston cheated, but we were left in the dark about whether that cheating helped. To date, every attempt to study the Astros’ banging scheme has been beset by doubts about when and where they were employing the system, which makes comparing splits a futile exercise.

New data gathered by Tony Adams shows that the Astros scheme was a fantastic aid to their hitters when it was accurate, but a powerful stumbling block when it was wrong, as it frequently was. Overall, the net impact of the cheating seems to have been almost nothing: The downside perfectly counterbalanced the upside and the Astros ended up being about as good as they were from the start.

Back in November when news first broke about the Astros’ banging scheme, many people (including me) cued up various old Houston games stored on YouTube and dove in, trying to isolate exactly when the Astros had been using it. I noted that you could use a spectrogram—essentially a digital fingerprint of what sound frequencies were active—to isolate the peculiar twin peaks that the Astros usually employed to signal a non-fastball on its way. After surveying about a dozen games’ worth of audio using the spectrograms, I noted that Houston used the scheme often and accurately.

The next logical step would have been to commit to digitizing a whole season’s worth of audio and then relentlessly hunting through it, either with a computerized approach or a painful, manual hunt. The former path would have involved teaching a computer to identify the sounds (very hard to do), the latter days of intense listening to an entire half-season’s worth of audio (81 games x 1.5 hours per game = I don’t have time for that). In no hurry to subject myself to at least a few dozen hours of torture with either route, I put aside the idea of gathering data for a few slow weeks in the offseason.

Enter Tony Adams, a self-described Astros fan who took on the task. Using a combination of ears and eyes (by looking at the spectrograms), Adams listened to all 58 Astros home games with available videos and marked every single pitch—there were more than 8,000 of them—with whether he heard the trash can a-banging or not. He estimates that it took him 40 or 50 hours to gather the information, listening to accelerated playback using a custom application he built. He kindly provided this data free of charge to the public on his site.

With Adams’ data, it’s possible to do a much more thorough analysis of how the Astros’ banging impacted the team’s offense. Instead of assuming that the Astros were perfectly accurate after they started the banging, we can tell exactly when they cracked the signs and when they turned out to be inaccurate.

Previous analysis of the audio suggested that when there was trash-can banging—any amount, but often a pair of hits—the code breakers in the tunnel thought that the next pitch would be a non-fastball. Conversely, when the can was silent, it could be a two-seamer or a four-seamer, but either way heat was incoming. By and large, the Astros tended to get the signals right, but it was hardly perfect.

They were most accurate when they whacked the can: When they did so, a non-fastball (curve, slider, change, etc.) was on the way 93 percent of the time and they were wrong seven percent of the time. But a lack of banging didn’t necessarily predict a fastball. It’s likely that at least some of the time, players knew going up to the plate that they hadn’t stolen the signs yet or that the code was unreliable.

More stunning is the fact that the Astros seemed to be lacking any banging information in a large portion of the plate appearances they faced. Excluding plate appearances where they only saw fastballs, hitters got bangs for at least one pitch in only 70 percent of the times they batted. Many of the at-bat bangs that were missing were late in the game against relievers, who presumably debuted new codes when they came in.

Based on Adams’ data, the Astros used the trash can frequently and they were relatively accurate, moreso when banging than when silent. Let’s turn to whether the banging scheme actually helped the Astros.

Here, it’s important to split the data by whether the banging was correct or not. They batted an impressive .282/.346/.478 overall throughout the year and .289/.346/.503 on pitches that were correctly identified (+.007/+.000/+.025). For reference, in 2019 that would have been roughly the difference between an entire team hitting like Francisco Lindor or Trea Turner vs. hitting like Franmil Reyes or Christian Walker. Especially when spread over the nine hitters on the Astros lineup, 25 points of slugging is a material boost.

But this neglects an important part of Adams’ data: all the times that the Astros signaled wrongly for a breaking ball when a fastball was incoming or were silent when a breaking ball was on the way. When this happened, hitters were at a severe disadvantage. They thought they knew a curve was going into the dirt, but actually a fastball whizzed by them. These misidentifications were significantly more damaging to Houston’s fortunes than the correct identifications were helpful. They batted .237/.294/.385 when the signal was wrong, roughly equivalent to 2019 Khris Davis or Leury García and certainly much worse than the incredibly-talented 2017 Astros lineup should have been able to hit.

So when the fix was in, the Astros received a substantial advantage. But when the players in the tunnel thought they had cracked the code and it turned out they hadn’t, it harmed the batters at the plate more than knowing the incoming pitch helped them. Considering that the trash can was relatively accurate and in use at least two-thirds of the time, how does it all shake out? Did the Astros gain or lose by stealing signs?

The ultimate test of the efficacy needs to price in the fact that one of the Astros’ most-skilled hitter, José Altuve, was not receiving bangs during his at-bats. It also needs to account for the opponents they faced when the banging was on or off and the fact that all the banging occurred within Minute Maid. It’s possible to slice and dice the data to deal with all of those confounding factors, but rather than take the data into the realm of small sample size, I opted to fit a statistical model (a generalized linear model, with random effects for hitter and pitcher) instead, measuring the outcome in linear weight value of every single pitch. This approach not only gives us an idea of how big of an effect the banging had in total but also how confident the model is that there was an effect at all.

And when you do all of that, the net effect of the banging comes astonishingly close to being zero. Nothing. Statistically, for all the work and effort that went into the cheating scheme, the grand result of it, at least as measured in this way, turned out to be no runs at all. 

This won’t be the last word on the Astros’ banging scheme. It can’t be: MLB’s investigation notes that they used other methods like whistling, including in the 2017 postseason. None of those alternate methods are represented here in this dataset, solely the banging scheme. As a result, I don’t think of these conclusions as definitive. However, they point to a major issue with the efficacy of any sign-stealing system: The penalty for bad signals is larger than the benefit to be gained by accurately relaying the incoming pitch. As a result, any method has to be extremely accurate for it to actually boost a hitter’s productivity.

If the other signaling methods were as inaccurate as the trash-can thwacking, they likely had no significant impact, in aggregate, on the Astros’ success. If the Astros refined their scheme or had better reads in the postseason than in the regular season, it’s possible that it was a major advantage; if teams got wise to their tricks and better at switching signs mid-game, they might have won their championship in spite of the banging rather than because of it.

And there’s no way to tell whether a well-timed bang or whistle or shout swung a decisive playoff series. None whatsoever. All the forensic audio analysis in the world won’t be enough to rule out every possible way the Astros might have sent signals to the plate, whether via buzzing bandages or electromagnetic signals beamed directly to the hitters’ neurons. At this point, however, the loud banging scheme for which the Astros got caught seems to have done at least as much harm as it did good.