Baseball Therapy: Is Oakland’s Run Differential for Real?

As of the moment that I write this, the best run differential in baseball is owned by…the Oakland A’s. Raise your hand if you saw that coming. Also, please raise your hand if—since I mentioned the A’s, you can work the word “moneyball” into this paragraph. I’ve run out of ways to do it.

Sure, like everything else around this time of year, we can yell “small sample size!” and use that as a convenient excuse to ignore the patently obvious. The A’s have scored a few big wins so far in 2014, including an 11-3 triumph over the Astros, a 10-1 win over the Astros, and a 12-5 win over the Astros. (In fairness to the Astros, the A’s also beat the Rangers 12-1 one night. Maybe they just like playing against teams from Texas.) That’s fueled a lovely run differential, and also the most wins in the American League. But is that run differential a reality or a small sample size mirage?

BP alumnus Jonah Keri, who edited the book that turned me into a sabermetrician, tapped me on the shoulder and provided the appropriate rainbow sprinkles to cajole me into running a few analyses on the subject. Additionally, since I’m of vaguely French-Canadian ancestry (although about all that’s left is that extra “e” in my last name from when my ancestors changed the family name to honor this guy), there’s an honor code to uphold.

How long does it take until a team’s run differential isn’t just random noise anymore?

Warning! Gory Mathematical Details Ahead!
The answer actually depends on how you ask the question. For example, the real question that most fans want answered is something like whether a team’s current run differential is a good predictor of how things will be at the end of the season. They’re trying to peek ahead to the end of October. To test that, I gathered together all team-seasons from 1962 (when the schedule expanded to 162 games in both leagues) to 2013, except for the strike-shortened seasons of 1981, 1994 (sorry, Jonah), and 1995. I started on Opening Day and calculated each team’s total run differential (divided by games played). I did the same for the team after Game 2, then Game 3, etc. At each step, I looked to see how well that run differential correlated with their end-of-season run differential. After the first game, looking at run differential is pointless. After Game 162, the correlation is perfect. When does it reach a point where it’s believable?

In general, I like to look for the point where the correlation reaches .70. For the uninitiated, 0 means that there’s no relationship between the two numbers, and 1 means there’s a perfect relationship between them. It’s a matter of how close you want that number to be to 1 before you feel comfortable, and you can see a chart below of how the correlation approaches 1.0 as the season goes on. The correlation hits .70 after 39 games. So, around the 40-game mark—mid-May—run differential starts to be a good predictor of what things will look like at the end of the year.

But those who read that closely are probably thinking, yeah, the reason that the correlation is so good is that those 40 games are baked into the end-of-the-season numbers. In some sense, we’re comparing something to itself, which usually produces a pretty strong correlation. What if we wanted to know how long it took until we could be independently sure that the team’s run differential is real? For that, we need a more complicated method called Cronbach’s alpha.

The idea behind Cronbach’s alpha is similar to work that I’ve done on when stats for individual players stabilize. For individual players, I’ve taken groups of, say 100 PA per player, and split them into equal 50-PA bins. If a stat is stable—say strikeout rate—then we’ll see a good correlation between the two groups of 50 PA each. If it’s a big enough correlation, we’ll say it’s stable at 50 PA. Cronbach works on the same basic principle, except that mathematically it slices and dices those 100 PA into every possible way to split into groups of 50 and averages out the correlation between them.

Again, I took all team-seasons from 1962 onward. For each game, in sequence, I figured out what contribution to the team’s overall run differential it made. That’s a nice way of saying I figured out how many runs the team won or lost by. I entered these numbers into the Cronbach’s alpha formula and looked to see how deep into the season I had to go before Cronbach’s alpha hit that magical .70 number. It turns out that after 140 games, you get there. Now, that means that you have two samples of 70 games per team-season being compared to each other. So, we can say that after 70 games, a team has a sample that’s independently big enough to make statements about its true talent level. If the season started over for some reason, we could expect that those 70 games would give us a good idea of what to expect over the new 162-game season.

Of course, the season doesn’t reset like that. The 70-game threshold is for when you want to know some deeper truth about the team in question, rather than just to peek ahead to what the final standings will look like. Still, it suggests that we can make some reasonable conclusions about a team’s true talent level by the All-Star break.

Run Differential: The Key to Understanding Life, The Universe, and Everything
The reason that run differential is so important is that it’s actually a better predictor of how well a team will perform in the future than their record at the moment. If your favorite team is under .500 but they are above water and have scored more runs than they’ve given up, try not to worry too much. However, on the flip side, if they are above .500, but have been getting by on squeakers and getting blown out when they lose, consider this a warning. The good times will not last.

And as for those A’s, check back in a week and a half to see whether they still have the best run differential in the league. At that point, we’ll know whether we can expect it to last through the rest of the summer.

Moneyball.