Baseball Therapy: Should I Worry About My Favorite Pitcher?

Of course. He's a pitcher.

I've gotten a few requests for this one. Almost a year ago, upon returning to Baseball Prospectus, I posted an update to the work that I had previously done on the reliability of hitting statistics. I had originally written one on pitching stats, as well, but never updated it similarly.

Warning! Gory Mathematical Details Ahead!
As with my piece from a year ago here at BP, I'm updating the methodology that I had originally used. I'm using Kuder-Richardson (formula 21) reliability for binary outcomes and Cronbach's alpha for non-binary outcomes. Data set is 2003-2012 (Thanks, Retrosheet!), with a minimum of 2000 batters faced during those years for each pitcher (unless otherwise noted), meaning that I can see reliability up to sample frames of 1000 PA. For stats that refused to stabilize by a sample size of 1000 PA, I used the Spearman-Brown prophecy formula to estimate the stabilization point.

These numbers represent the point at which each stat reaches a reliability of .70 or greater according to the relevant formula.

What Do I Really Know About You?
Well, we quickly have an idea of strikeout rate, ground ball and fly ball tendencies, and (somewhat less quickly), walk rate. Over a season, you can get a pretty good idea of a pitcher's single and HBP rates. Strangely enough, singles stabilize a lot faster than the alleged "true" outcome of HR rate. Some of the classic one-number rates (OBP, SLG) can stabilize over the course of a year for a full-time starter. And yes, BABIP still needs a lot of data (roughly 2000 balls in play), but that number is actually about half of what I had once estimated elsewhere.

All Numbers Tell a Story, But it Might Not Be The Story You Wanted to Hear
I'm well aware of the fact that most of the requests for these analyses came from people who were trying to get a feel as to whether their favorite pitcher having a bad year (read: Roy Halladay) was just having a bad couple of games or whether his performance was "real." I want to (again) point out that the way in which I most often see these numbers used is not exactly what they’re meant to show.

When I say that strikeout rate for pitchers stabilizes at 70 batters faced, what I mean is that we can be reasonably sure that his strikeout rate over those 70 batters is a good reflection of his talent level over those 70 (now past) plate appearances. This is different from saying that once a pitcher has gotten to 70 batters, we can assume that he will perform this way for the rest of the season. That's an assumption. It's not a bad one, but it is an assumption. Instead, what it means is that if his underlying skill set has changed in some meaningful way, we'll know in 70 plate appearances.

Also, I'd caution people against treating these numbers too dogmatically. 70 plate appearances is not a magic number. It's the point where a measure of reliability slowly crosses an only-somewhat-arbitrary line in the sand. At 139 PA, the reliability for strikeout rate is just shy of .70, and you need to have just a shade less confidence in any proclamations that you make using those 139 PA.