When you think about Baltimore cuisine, you probably think about seafood, and more particularly crab cakes. The Chesapeake offers a rich harvest of the ocean blue, but it isn’t all high quality. For example, a common cheap eat in Baltimore is known as lake trout, which has three surprising characteristics. One, it isn’t actually lake trout (although that is a real fish). Two, it isn’t actually from a lake (it’s from the Chesapeake). Three, it isn’t actually trout (it’s Atlantic whiting). For the uninitiated, the surprise of being served two filets of whiting and some white bread might be rather severe. That being said, if you don’t have much money and you’re not really used to better, get yourself some lake trout. Trouble is, lake trout isn’t very healthy, since most of the taste comes from the deep fat fryer, and a steady diet of it would get monotonous—it is, after all, a mostly bland white fish. Every once in a while, you’re going to want some steamed crabs.
The Orioles are bad again this year. When is the last time you couldn’t have said that about the team? That is, when was it that not only were the Orioles good (or at least not bad), but that they had created reasonable expectations that they would be good? Here are the Orioles’ records for each season of the last decade:
|
Year |
Losses |
|
|
2009 |
64 |
98 |
|
2008 |
68 |
93 |
|
2007 |
69 |
93 |
|
2006 |
70 |
92 |
|
2005 |
74 |
88 |
|
2004 |
78 |
84 |
|
2003 |
71 |
91 |
|
2002 |
67 |
95 |
|
2001 |
63 |
98 |
|
2000 |
74 |
88 |
The last time the Orioles had a winning record was 1997, when they were 98-64. Do you know how long ago 1997 was, in baseball terms? Put it this way: the only contributor to that team who is still playing major-league baseball is Arthur Rhodes, a fact that is revealing particularly because it demands some serious respect for what Rhodes is currently doing for the Reds at age 40. The last time you couldn’t really say “the Orioles are bad again this year” was during the Clinton administration.
It has not really been a decade of soul-crushing, monochromatic gruel (you know, the kind intentionally low on protein to keep you docile); that’s the diet Pirates fans have been on. Rather, it has been an awful lot of lake trout. There have been moments where Orioles baseball kind of hit the spot—at least while it lasted. There was 2005, when the team darted out to a 42-28 record on the strength of its .285/.342/.481 batting line—oh, Chris Gomez, how we never tired of your BABIP shenanigans. The year before that, the team actually outscored its opponents 842-830 and finished in third place, despite the fact that Sidney Ponson yielded an astonishing 136 runs.
That it was third place in 2004 with 78 wins demonstrates the real problem. Lake trout is nobody’s first choice of seafood. If you had a little more money, or were in some other town, you’d probably opt for something else. But that’s because other seafood can be absolutely delicious. The hegemony of teams atop the American League East is a ceiling impervious to the Orioles' charms. That they stood by while the Rays darted past them puts a particularly bitter spin on the state of affairs, but it doesn’t change the basic calculus. Since the ascent of the Red Sox and Yankees in the late '90s, the Orioles have been forced to eat fried Atlantic whiting from corner stores that insist on calling it something other than what it is.
But Hey, At Least They Get Some White Bread, Too
It was supposed to begin to turn around. Sure, it took a little wishcasting and some magic beans, but it wasn’t too hard to see a 2012 Orioles roster that featured front-line starters Brian Matusz, Chris Tillman, and Jake Arrieta to complement a middle of the order featuring hitters Matt Wieters, Adam Jones, Nick Markakis, and Josh Bell. Pretty soon, the Orioles were going to buy themselves a nice condo on the Inner Harbor and it would be bye-bye lake trout, hello crab imperial.
That plan has not worked out. Before their current four-game win streak, the Orioles had a winning percentage below .300. According to our postseason odds, the Orioles wouldn’t make the playoffs this year even if they had 100,000 shots at doing so. Even as it stands, they are on pace for a 50-112 record, which would be by far their worst season since the truly miserable 1939 St. Louis Browns went 43-111. Matusz has an ERA close to 5.00 and a SIERA that isn’t that much better—thanks, bullpen! Tillman, after finally getting called up, was torched in four starts and has a 5.91 SIERA. Arrieta? He's at 5.69. The hitters aren’t faring much better. Wieters (and his .233 TAv) has as of yet been neither deadly nor accurate. It took Jones a June on fire to reach a .255 TAv, and he still isn’t OBPing over .300. Bell is hitting a disappointing .263/.307/.438 with a 24 percent strikeout rate in 296 Triple-A Norfolk plate appearances. Only Markakis and his .295 TAv have stood above, and even his power ceiling is much lower than it once was: his .427 slugging percentage is the lowest of his five major league seasons.
Of course, a great deal of it has to do with their rough schedule. They’ve gone 8-25 against teams from the AL East, but their 15-27 record against all other opponents is considerably better. They’ve also gotten a little unlucky this year, and reading from left to right on the BP Adjusted Standings report is like washing away a little bit of awful from the 2010 Orioles. Their third-order record is 29-46, which translates to a .387 winning percentage. The Postseason Odds report projects the Orioles to win 57 games, which is some small consolation to the dignity of George McQuinn and the '39 Browns.
Things get dramatically better (on the order of six wins) when we do this because the Orioles have faced better hitters than any other team and have faced better pitchers than any other team. “Yeah, yeah,” you’re thinking, “we all know the AL East is good. But is it really that good?” I’m glad you ask, because I like to use numbers. The hitters the Orioles’ have faced have a combined .264 TAv, which is the highest mark in baseball. The pitchers they have faced have allowed a combined .254 TAv against, which is the lowest in baseball.
Doesn’t it just strike you as unfortunate that the same team that has been miserable for a decade would face not only the best hitting so far but also the best pitching? It makes sense, since they are the only bad team in the AL East, meaning they are the only team that doesn’t get to play themselves. And the AL East has played a disproportionate number of interleague games against the also-strong NL East. But, really, lake trout again?
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.
Subscribe now
Hahaha - great potshot at how badly PECOTA missed on this guy. PECOTA is just as bad as he is these days.
The problem isn't PECOTA, the problem is the inherent unpredictability of player projections. Every system, no matter how accurate, is going to have quite a few big misses.
PECOTA not only is less accurate compared to other systems now, it even has a lower accuracy than previous years. Considering that this change accompanied the loss of Silver from BP, this seems like it is NOT a coincidence.
Feel free to disagree, but you could look into it instead of just following along like the rest of the unquestioning masses while I look for better content elsewhere.
Obviously, you decided not to look it into it and just assumed that your uneducated opinion is correct. Typical idiot behavior and another sad decline, as this used to be a community of thinking people.
http://www.insidethebook.com/ee/index.php/site/comments/evaluating_the_2009_forecasts_chone_zips_fantastics_win/
As shown here, PECOTA in 2009 was about as accurate as just using 2008 numbers and by far worse than any other system. Now I've even done the work for you. Anything else, genius?
Make a real point instead of bitching and I'll reply cordially.
Ahhh, sweet irony.
Now, as regards the Steamer forecast evaluations for hitters you reference.
The claim that PECOTA was not significantly more accurate than the '08 numbers is based off the Standing Gains Points analysis. What I want to emphasize here is that looking at SGP looks at two elements of forecasting simultaneously - projecting a player's talent and projecting a player's usage.
If you look at the evaluation for OPS listed there, you find that PECOTA was, in fact, more predictive than the '08 totals. Converting the RMSEs listed to a more intuitive scale, using some numbers provided by Jared in that Book Blog thread you list:
'08: .094
PECOTA: .085
Marcels: 0.083
ZiPS: 0.080
CHONE: 0.079
What this means is that, for instance, 68 percent of CHONE forecasts tested were within .079 points of the observed. What you tend to see is that there is little differentiation between forecasting systems at projecting OPS. They're all better at predicting results than single-season stats, but there's not much room for differentiation.
[I do want to register a methodological complaint here, while I'm at it. The results you see are RMSE between projected OPS and observed OPS. The average OPS of players in sample was 0.769; the average OPS projected by PECOTA was 0.763, compared to 0.770 for ZiPS and CHONE. But we don't care about a player's OPS forecast in the abstract, but his OPS relative to other players. If you correct the forecasts to have the same average OPS as the observed, you should see the spread of forecast accuracy tighten up further.]
So, if PECOTA is predicting the rate of performance roughly as well as the other forecasting systems, why did it show up as doing so poorly in the SGP analysis? Because PECOTA did a terrible job "projecting" playing time. Of course, the values in the weighted mean are a projection of how much a player could play at all levels, including the minors.
If you're looking for projections of playing time, we provide those in the form of the depth chart values, available through the Player Forecast Manager. (The PFM values are also available for download in CSV format.) But the Steamer evaluation you linked did not use the depth chart playing time, it used the weighted mean playing time.
So what that result is really telling us is that PECOTA is substantially worse at projecting playing time than other systems, if you ignore our playing time forecasts altogether. I don't know - is that a meaningful result? I don't think it is. (If you'll note, the Steamer forecasts DID use the PECOTA depth chart results, and those PA forecasts grade out as among the best, substantially better than CHONE or ZiPS.)
My comment was voted below the viewing threshold for criticizing BP despite the fact it originated from a BP author, which continues a trend of other subscribers automatically supporting the BP brand without thinking. I find that incredibly depressing, and I'm not going to play nice just because people want to assume that everything is great because they'd rather not look into it themselves.
While your take on PECOTA doesn't show as drastic of a difference between the systems, it still shows that PECOTA performed the worst out of the systems. Some criticism of PECOTA is warranted and hopefully that would lead to improvement.
Overall, I'd say that the criticisms of PECOTA, the Baseball Prospectus brand, and other comments is more than warranted.
He assumed that PECOTA was just fine regardless of how many misses or by how much, instead of questioning it, so I'd say he's clearly part of the "unquestioning masses."
*Clay has taken over PECOTA from what I understand, so it might seem odd that I'm basically criticizing him and praising him at the same time. I guess that's true but I just think he does some things better than others.
In any event, I'm done with this one. Frankly, you are just wasting your time here and the time of everyone else who actually enjoys reading the site.
And frankly, again, I'll reiterate my point about if it's that bad here, why even read? Even if the subscription itself is free, it's still a massive waste of your time on something that is apparently worthless; time which you could otherwise be using to read ever-more profound baseball analysis.
One final point: if indeed you are receiving the subscription for free and not paying like the rest of us, that was something done by one of the authors here as a good-faith gesture in an attempt to win you back over. If you still don't like the final product, fair enough, don't read the site. However, if someone made a good-faith gesture to you to give you six months of a free subscription, the least you could do would be not publicly bash the authors and their work product on the site itself. That's not you having objections to the work product as much as you just having the need to be an ass in a public forum.
There was a time not too terribly long ago when the motto of PECOTA was more ambitious than what sounds a whole lot like: "not that much worse than the average projection system." Any impartial observer reviewing the data would quickly realize that PECOTA was a relative disaster in 2009; and yes, I am talking about rate stats, not counting stats. I am also talking about standings predictions: http://vegaswatch.net/2009/11/evaluating-april-mlb-predictions-2005.html.
The additional problem with the "not much worse than the others" argument is that the others are far simpler systems and are freely available. I.e., the systems' authors would readily admit there is not a ton of scientific rigor behind the algorithms used. These systems should be used by PECOTA as the benchmarks to beat -- rather than shooting only to beat using last year's stats.
Nate Silver used to pride himself on beating other projections. I sense none of this drive to outperform in this post or any other by a BP author with regard to PECOTA over the past year. After the pre-season debacle this season, one would think there would be an in-depth article (or ten) on the steps BP was taking to improve PECOTA. You know, cool research articles on possible ways to improve forecast accuracy -- the former bread 'n' butter of BP. One would think BP would be putting its top minds on the project *now* -- not next Jan or Feb.
As a final note, how can you praise the PECOTA depth chart playing time forecasts, when among the four systems designed to project playing time, it finished third?
Um, I thought that was what Eric and Matt spent a series of articles on... namely, introducing/explaining/using SIERA. Nor have those been the only ones. And they did it last January/February as I recall...
The main beef people have about PECOTA is it is proprietary and a bit black box, so it's hard to critique it or know why it does/doesn't work.
Then again, I don't subscribe to BP solely, or even primarily, for PECOTA.
PECOTA ain't perfect. That's why you should use several of them. But if you're not using PECOTA, you're missing out. It informs its projections with player comps using things like body size, injury history, etc. That certainly has value.
No, it doesn't always nail down Ichiro or Javy Vasquez. But other systems have similar shortcomings as well. Multiple tools are almost always better.
I promised to give them that opportunity, which is why I'm still reading these articles.
My real complaint about PECOTA is more related to the annual, which is no longer as reliable a reference guide because the player comments no longer go out on a limb and the projections are less accurate.
I actually thought that the decline in PECOTA was somewhat public knowledge. It seems like many people still believe it's working at the high levels it posted years ago. There's a reason why all of the favorable quotes on the cover of the annual are 5 or more years old.
I will say this... watching the futile rebuilding efforts in Baltimore makes me appreciate even more the freakishly awesome job Tampa has done. And, in turn, that makes me profoundly depressed. The fact that a team has to absolutely nail it on every front--mdrafting, development, free agency, trades, management--just to compete with the Yankees and Red Sox... I may just stop cheering for a team and start rooting for realignment.
But deep down there are some fundamental problems. The scouting is fatally flawed - players don't seem to adjust their approaches to the opponent, they pretty clearly recommend signing players who're done (Atkins, Lugo). The coaching and development system has long had a problem with the jump from the minors to the majors. And their international "efforts" barely exist.
I go back to a Bill James' quote that MLB teams do tons of little things to improve all the time, and that he can see from the inside that the Red Sox are clearly better in numerous ways than even 5 years ago. The O's seem to be getting better 5% a year, while the competition moves up 10%, 15%, 20% a year.