2010 PECOTA Projection Analysis
As promised, here are the results of our preliminary analysis of the 2010 PECOTA projections. We took the methodology for multiple builds of the projections (which will be easier to identify when we start using versioning, which we will do when we've got everything else cleaned up) and ran all the inputs for the 2009 version of PECOTA through it. Then we compared those projections to the actual results of the 2009 season.
In all cases, lower numbers are better.
RMS Error results
|
System |
Release Date |
R |
H |
2B |
3B |
HR |
RBI |
BB |
K |
SB |
CS |
TOTAL |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2009 PECOTA | 3-28-2009 | 10.30 | 12.91 | 4.86 | 1.46 | 4.16 | 10.12 | 8.78 | 12.25 | 3.87 | 1.47 | 70.18 |
| BP2010 PECOTA | 1-5-2010 | 8.92 | 12.21 | 4.28 | 1.50 | 4.22 | 10.35 | 9.81 | 14.52 | 3.68 | 1.77 | 71.26 |
| Current PECOTA | 3-12-2010 | 9.12 | 14.16 | 4.30 | 1.50 | 4.43 | 9.98 | 9.09 | 10.42 | 3.68 | 1.50 | 68.18 |
Bias-adjusted RMS Error results
|
System |
Release Date |
R |
H |
2B |
3B |
HR |
RBI |
BB |
K |
SB |
CS |
TOTAL |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2009 PECOTA | 3-28-2009 | 8.10 | 9.50 | 4.16 | 1.46 | 4.06 | 8.94 | 7.73 | 9.73 | 3.72 | 1.43 | 58.53 |
| BP2010 PECOTA | 1-5-2010 | 7.11 | 8.68 | 4.10 | 1.46 | 3.93 | 8.46 | 7.83 | 10.07 | 3.67 | 1.50 | 56.81 |
| Current PECOTA | 3-12-2010 | 7.16 | 8.90 | 4.12 | 1.47 | 3.93 | 8.15 | 7.88 | 9.91 | 3.56 | 1.45 | 56.53 |
BP used the 2010 PECOTA projections as the basis of our LABR draft strategy this weekend.
PECOTA Ten-Year Forecasts and Hitter Cards
We have diagnosed the main problem with our ten-year forecasts as reported on the PECOTA beta hitter cards. Nate Silver generated one set of comps in the original PECOTA process and used those comps to generate the long-term projections. We were trying to re-generate new comps in year n+1 based on the player's career thus far with his projections for year n included, and repeating. In addition to introducing considerable extra complication, this process generally created much less favorable long-term projections, as many readers noted.
We've adjusted the long-term projection process to work the way Nate originally designed it. Once we've got everything stabilized and released, we'll be revisiting this topic.
Another problem we had was that a player projected to be out of baseball entirely had a value of zero, while if he was good enough to be projected to get some playing time but bad enough to be below average, he'd have a negative value. For example, in a player's tenth percentile, he might be out of baseball entirely, returning 0 WARP, but in his fiftieth he performed well enough to stay in baseball, and rated -1.0 WARP in the playing time he was projected to get. Because we didn't distinguish between a 0 WARP in baseball and a 0 WARP being out of baseball, this enabled some very weird results for players with this condition. We've changed the process to differentiate between the two, and the values should now be much more reasonable.
The K bar in the player profile graph has been reversed, and now works as it did previously.
We still have a problem with some players having their higher percentile projections zeroed out, while the lower are filled–we've identified this internally as "the Koyie Hill problem". This is a similar issue to the reversed projections problem above, and it'll be fixed this weekend.
PFM Settings Update
We've made some small changes to the position adjustment setting of the PFM:
"Level 0" is now "OFF"
"Level 1" is now "ON".
Levels 2 and 3 are disabled. We are looking into bringing those levels back into play–lets us know how much you used and miss them.
The default is now "ON", not OFF.
Depth Charts, Weighted Means Spreadsheet, and PFM
We've pushed two updates to the Depth Charts, Weighted Means Spreadsheet, and PFM this week.
The first one was sent out late March 9. With this update, we attempted to address some of the issues people had noted with Depth Chart team statistics that Clay mentions in this post. While this might have made the team projections more satisfactory, it quickly became apparent that it did so at the expense of the individual player projections, which are what the vast majority of subscribers are using PECOTA for this time of year. If you downloaded a spreadsheet or ran PFM between the evening of March 9 and now, please do it again and use the individual player stats you get from the current data.
As an aside, the modifications that were made to the March 9 data were more-or-less applied on a league-wide basis, so draft order and dollar value from the PFM probably won't change very much. The raw statistics that were predicted will be fairly different, though.
This morning, we pushed out another update which puts things largely back to their previous state. The RMSE analysis above is run on this version of PECOTA.
Depth charts will be updated at least every other weekday through the start of the season.
Weighted Means versus Fiftieth Percentile
Traditionally, PECOTA has used weighted means projections for it's default projections–the Depth Charts, PFM, Weighted Means Spreadsheet (obviously), and player cards have all used or highlighted the weighted means stat line.
This year, we've been using the fiftieth percentile for these applications instead. Until recently, we haven't had the weighted means at all for PECOTA in 2010. We now have the weighted means in the cards–see the bottom row of the 2010 projections table–and the weighted means are being used for the ten-year projections in the cards. Everywhere else, though, we're sticking with the fiftieth percentile projections for now, so you'll see the projections in the fiftieth percentile line of the cards match the Depth Charts, PFM, and Weighted Means Spreadsheet (which I realize means the spreadsheet is now misnamed).
We're working on the pitcher cards now. They'll be available next week.
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.
Subscribe now
Is there a way for you to footnote new additions/eliminations to the PFM/Depth Charts/Weighted Means (in terms of players added/removed)?
It would make it easier to match up prior iterations which have been transferred into spreadsheets.
Our crew has posted an updated about some of the recent changes to PECOTA. You can find it at
http://www.baseballprospectus.com/article.php?articleid=10226.
We hope this is helpful and we're sorry for the delay in communicating with you. Thank you for your continued support of BP."
Seriously?! This is the response I get from your customer service when I complain that I paid for access to "just the numbers", but find most of those numbers are not available for my draft tomorrow.
It mid March and there's no long term projections, no pitcher weighted mean, no pitcher cards, the hitter cards are "beta", the PFM is using different stats this year because the proper ones are not available. This is a giant problem for fantasy subscribers.
Sorry but a link to this post is not good enough, and does not address the problem.
Allow me to boil it down: this post is insufficient. BP acknowledged a huge problem with your core product, then went silent for two weeks. When you posted the above, an analysis of that problem, readers immediately identified further significant problems that BP hadn't considered. BP acknowledged that those problems should be examined.
What is missing, at this point, is an abject apology for soliciting customers on the basis of a core product that was never delivered, and a gigantic banner on the home page that tells it as it is: "We fucked up. We'll get it right next year. Enjoy our high-quality in-season content over the next nine months."
His Eq rate stats are .305/.370/.509 and .298/.360 /.509 in his 90th and 80th percentiles, respectively. But the adjusted rate stats are .306/.371/.519 and .308/.370/.535 for those same percentiles. Why is this happening (for him and others)?
Staying with Adam Jones, his Weighted Mean Projections have him at .291/.323/.483. This OBP is lower than that for his 10th percentile. It seems that a similar OBP-suppressing phenomenon is happening for most players. However Dexter Fowler's Weighted Mean OBP is .380 which is higher than his 60th percentile projection.
Just a couple of observations.
Fowler's 80th percentile slash stats: 0.295/0.393/0.458 are better than his 90th percentile stats: 0.285/0.390/0.448
What a joke... BP, you guys have lost it I think. Two weeks after the initial problem is diagnosed, this is what you come back with?
Is that setting reliable at the moment?
Thanks in advance!
If you want to add in propensity to overperform, consider making one of your categories "Upside."
At least I've finished in the money every season I've used it...
Thanks for setting me straight!
http://www.baseballprospectus.com/article.php?articleid=4793
So you would use it to inflate the values of the studs and devalue the other players, i.e. my Albert Pujols example.
If you want what I would consider a true dollar value, use the conservative values without position adjustment turned on. Those will compare more closely with the magazine dollar values, IMO.
The depth charts now show the AL at 61 games over .500, which would be impressive, all the more if not for the fact that the NL is projected to be right around .500.
--JRM
I suggested this a couple of weeks ago. I'll suggest it again now. Ask some of your subscribers to be Beta-testers for the Depth Charts, PFM and Weighted Means, PRIOR to release.
Nevertheless (and I say this with all due respect), I think you should be offering some sort of refund to all of us. I can't speak for anyone else, but 90% of the reason I subscribe to BP is for PECOTA stats to prepare for my fantasy draft. In particular, I'm most interested in UPSIDE, as I am drafting almost exclusively prospects in a 100% keeper league. I pay $40 for a year's subscription so that I can have access to these statistical tools for basically one to two months of the year. I tried to delay the start of my draft further and further in hopes that PECOTA would be fixed before we started making picks, but the other guys in my league understandably wouldn't wait any longer. So while I will still enjoy reading the columns for the rest of the year, I feel like I just wasted $40 (or, to be fair, about $36) on this subscription.
I suggest a partial refund and/or an extension of our subscriptions.
You took the words right out of my M(cL)outh.
I have long been concerned that PECOTA is one of the latest projections posted every season. I can get Shandler's stuff in December. If BP wants to make this right, let's get an good schedule for the fall where we can have access to this much sooner. One suggestion was Jan 15. I think that's too late.
PECOTA is the primary input into my draft list every year. That may have to change.
1. Get the Pecota Beta on line January 15 next year.
2. Consider getting customer input before posting.
3. Review the Pecota at basic levels so at least the math is correct. Not +60 wins in the AL and -15 in the NL.
4. Make a big public apology.
5. Offer a discounted subscription rate for renewals.
THIS. I understand making mistakes, but I feel like nobody at BP has come close to acknowledging the extent to which this process has been screwed up this year.
Dave: with all due respect, this post ain't it.
I've been working on that message in a larger format, with more background included. In the meantime, I understand and agree with many of the points people are making here. We're working hard to make things better than ever, and I'm sure we'll get there, but it's been much more difficult than we anticipated.
If you want to send me an email about PECOTA, or anything else, I'm at dpease@baseballprospectus.com.
You may also want to address *why* you've released a new update today, when looking at the projections both on the site and the spreadsheet, it seems abundantly clear that there are still a number of bugs in the system. While your post above seems to get at this - although, to me, it doesn't move the needle at all - why should we trust the current updates rather than the 2/25 projections?
For me, having access to another set of projections that likely contain a number of errors - different ones than prior versions - only further muddles the process of trying to identify what production I can reasonably forecast for players this year. In short, in my opinion providing the current updates (presuming that they're wrong, which I believe they are) only makes the matter worse.
I may only speak for myself, but I recommend getting it right and then giving us the final, ACCURATE product.
Thanks again.
Dave, if you're writing a chronology, explaining why facially defective updates were put up is important. If that explanation is, "We're idiots," that's better than no explanation or an implausible or excuse-ridden one. The latest Depth Chart snafu is one of many that are obvious to moderately mathy folks. I've articulated some of the many other problems elsewhere.
I realize I have a pitchfork and a torch, here ("Some problems are best solved by angry mobs" - Homer Simpson, and if it wasn't him, it should have been) but I think the pitchfork and torch crowd are right.
And I didn't come with an axe to grind (to mix my metaphors); you can see my number, I've been to at least a half-dozen BP events, my Scoresheet league is called AL_BP_NorCal, I've bought every book since number two, y'all have kindly paid me for a couple of articles, and I've been acknowledged in the book (which I found quite flattering). It took *a lot* to get this torch lit.
But I think it's important to shed light on the unsupportable continued releases and continued reliance on PECOTA. Further, it's important to show that what you're saying now doesn't explain nearly enough, doesn't make up for continued obvious errors, and doesn't speak in a properly meaningful way to PECOTA 2010 v. 0000011's accuracy.
No more releases of Depth Charts until wins average 81 (or slightly under, if you want some rainouts.) No more releases of PECOTA that facially don't make sense. No more using 2009 data to determine if it accurately predicts 2009 outcomes. No more tinkering. Back-engineer it to 2008 PECOTA if you can; that was the last well-performing iteration. If not, well, there's 2011. But enough of this course of conduct, which people are rightfully quite angry about.
--JRM
First, back-testing on one year of data is insufficient.
Secondly, was the back-testing done after filtering out the known information (the 2009 season?) That is, doesn't the 2010 PECOTA have some 2009 season information in it? If you did filter it out, you should have clarified this. If not, the data's value is damaged.
Thirdly, what bias exactly was filtered out?
Fourthly, the BA/OBP/Slg are all worse now than they were with the book's method. You used a measuring system that a cynical unpleasant person might think is designed to hide this.
Fifthly, the changes keep coming. I have to say, if I kept changing a system, I'd think I could fix it up to predict the 2009 players right well by this time.
To quote myself from the Depth Charts thread, I don't believe you when you say PECOTA 2010 is fine now. More very simple errors are being generated with each iteration, so I don't think you believe you either.
--JRM
We've seen some evidence, however, they they were also changing the formulas -- not just trying to replicate Nate's original code in a different system. For example, in this article Dave mentioned using a different way of making 10 year forecasts than Nate did -- which, however, led to a host of unanticipated problems. He also talks about sometimes using the weighted mean PECOTAs and sometimes the 50th percentile. In an earlier article Clay also talked about using a different, larger base of historical player data for purposes of identifying similar players.
With so many moving parts, it's really hard to know what went wrong. But again, their looking at 2009 is a good thing to do.
No it's not, you just gave the answer. Too many moving parts. You can't port software to a new platform AND make changes at the same time. Port it. Test it. Then change it. If you only change one thing at a time, and constantly test after making each change, you'll know exactly what broke it. It's when you get in a hurry, and try to cut some corners that you end up screwing yourself. Unfortunately, I know this from experience.
I have slightly older runs of the new PECOTA system for three years of data, which I have tested against the originally published PECOTAs. Old and new PECOTA both return identical RMSEs for OPS, at least in the version of the test I'm currently looking at. (I'm currently working on ID mapping for some other projection systems, to give some basis for comparison.)
Would like to see RMSEs for SLG and OBP (not counting stats), new vs. old, for at least the last five seasons.
Thanks very much for this information. Comparing against actual PECOTA's for 2007-2008 (under the old Silver system) would be very, very helpful, too.
A full discussion of various iterations of current PECOTA vs. 2007-2008 PECOTA vs. (say) Marcel and CHONE, with complete explanations of the methodology and a direct comparison of the slash stats would be welcomed by many.
Combined with a full explanation of how we got here, and (critically) a good PECOTA performance in 2010, and that's dirt in the hole rather than dirt out.
--JRM
The results remain the same. The current iteration of the PFM data - what is in the PFM and the 'MjLgHitters' portion of the has a lower RMSE than the 2008 Pecota on 8 of the 10 categories (losing triples and CS); the sum of RMSE across all 10 categories was 6.5% lower (better) than the 2008 system. In bias-adjusted terms, it was 1.5% better on the sum of all RMSEs.
Second, when back-testing for 2009 projections, all data was restricted to data available in spring 2009 (likewaise for 2008). The target for the league and parks settings was always taken from previous year's data, and a section of code in the program prohibits any comparison data from an equal or later year than the test player from being used.
Third, the bias in question is the ratio between projected numbers and actual numbers. For the first set of numbers, all forecasts have been pro-rated to the actual number of plate appearances - we're testing the performance of the program, which means our real target is the rate of each stat. The second set of numbers pro-rates again for systemic bias - if forecast HR were 10% high across the board, we knock 10% off everybody and recompute.
Four is correct. The book version was entirely restricted to forecasting 2010 data, but I will certainly look into taking back what worked there - assuming it can identified.
Five, as long as the system has identifiable improvements, changes will continue. I don't recognize any point to say "done, stop now". And more drastic improvements to 2009 were definitely possible - but not without overfitting and killing performance in other years.
We've had serious execution and communication problems this year, and I must again apologize for those, but I'd sooner go out of business than resort to something like that.
It's the same as developing a stock trading system using all data through 2009, and then testing it on 2009 data. And then comparing to a system that was develped using only data through 2008. Ideally, you want to test the system on data that has never been used in the development process.
*No one* is suggesting that the new PECOTA somehow has access to future results as it makes it projections. Rather, there is simply some interest in using more rigorous methods when trying to assess accuracy. In this case, to compare the old vs. new PECOTA, it would only make sense to look at past years that neither system had access to during *devlopment*.
I'm going to lay out my impressions about what's been going on, reading between the lines of the reports we've received.
Getting the various parts of the PECOTA system working on a new platform is very complicated. One could probably expect some differences in precision by integrating the estimates on one platform. Nate did some analysis (specifically the identification of comparable players) using a stats package and imported results from that to his Excel spreadsheets. So there could be some loss of information from using such a "disintegrated" system of equations and data management.
However, contrary to the remarks by previous commenters here, there was no need to do split-half comparisons or find new data in order to determine whether the new PECOTA platform is achieving what the old PECOTA platform achieved. Instead, making sure that the new and the old are consistent with one another, using past years (e.g., 2007, 2008, 2009) and the same input data would be perfectly sufficient -- the best way to go in fact. (The purpose of this work was NOT to prove whether PECOTA did better than other forecasting systems -- but rather to get the existing PECOTA working on the new platform.)
That said, as I mentioned in response to a query earlier, Nate made changes to PECOTA pretty much every year except perhaps 2009 (e.g., using GB/FB ratios, league adjustments, platoon splits, etc. -- none of which were in the original 2003 version). So replicating PECOTA estimates for earlier years could be a challenge -- requiring the programmers to REMOVE features that were in the 2009 version. That probably wouldn't be a reasonable expenditure of time. And I don't imagine that they actually tried to do this.
For that practical reason I think it was perfectly reasonable to focus on first trying to replicate the 2009 Excel-generated PECOTA's on the new platform, and if that replication proved successful (as Dave's and Colin's entries suggest it was), then seeing whether the "2009 PECOTA" algorithm on the new platform worked well retroactively on the 2007 and 2008 estimates.
Leaving aside the peculiar problems with the PFM -- which may not be integral to evaluating PECOTA (or any other systematic projections), then once Clay and others were convinced that their translated code on the new platform closely matched those that Nate obtained in 2009 and did AT LEAST as well for 2008 and 2007 (when Nate's formulas may have been different from what they were in 2009) they could address the PFM issues.
These, unfortunately, proved to be more complicated than was anticipated. And introducing some changes to the system, such as in the multiyear forecasts, added further complications to producing the PECOTA cards and getting the PFM to work right. In addition, as Clay has noted here, unlike the PECOTA's generated in December (and publised in the annual book), the later ones get refined by taking more information into account about lineup changes, batting order, and playing time -- reflected in the depth charts.
Users of PECOTA expect such information to be taken into account as it comes available up til opening day; and in its depth charts BP also tries to anticipate playing time for the entire season (including players not on the 25-man ML roster on opening day). BP subscribers know from experience that the post-January PECOTAs are therefore subject to change as the depth charts and batting orders are refined.
Like everyone else -- including Clay, Dave, and the BP team -- I wish this would have been resolved before now. But I am happy to see them working through these issues as well as that checks of the system (which are at the top of this article) are favorable.
Age 26: .305
27: .294
28: .292
29: .290
30: .291
31: .293
32: .292
33: .291
34: .301
35: .292
Other examples (pulled quickly by spot checking random players) include Sizemore, Miguel Cabrera, Adam Lind. Basically almost every good player is projected to experience no decline between age 30 and age 35. This does not correspond to observed reality (http://www.baseballprospectus.com/article.php?articleid=4464 and http://www.baseballprospectus.com/article.php?articleid=9933), or to the way past PECOTA projections looked.
Jesus Montero's weighted mean OB of 315 is only 20 pts above his BA of 295, and lower than any of his percentile forecasts. Clearly a miscalculation. But the final nine years of the ten year forecast are built on that number.
Clearly, they have broken some things during the learning process.
This is why I cannot fathom the thought process that resulted in BP's strategy to implement the changes in the manner they chose. PECOTA is, after all, the foundation for all projection analysis done on the site and the signature product for the company. At the least, BP should have run one year of the new system completely behind the scenes while evaluating the performance. As someone relatively risk averse fiscally, there is no way I would have put the brand at risk to this degree.
I've written some fairly extensive excel calculators before, though nothing on the scale of PECOTA (which used to take -days- to actually do all the calculating!), and when I've needed to change or update them, it's a task that takes hours even knowing where you put every equation. For people who didn't write the spreadsheets to come in and learn how everything is arranged and where all the math is done and adjust it is something that can only lead to many mistakes and a lot of wasted time.
That's what happens when something that was intended as a personal toy turns into a major financial asset. PECOTA wasn't created to do what it does for BP: I'm sure it was far from optimally coded in the spreadsheets, and to update it and possibly migrate it to a better calculating framework has got to be an incredibly complex task requiring thousands of man-hours.
(we now return you to the PECOTA/BP-flogging, already in progress)
Unless BP transparently addresses and fixes the PECOTA meltdown, none of the data they have published this year will inform my fantasy drafts for 2010. Just impossible to determine if any of the BP iterations is better or more accurate than the others.
HQ, CHONE, hell even mlb/cbs/espn projections are providing more value to me this year. Particularly the no-charge ones.
As someone who has used PECOTA data to draft for several years with varying levels of success, I do not see anything in the current iteration that renders that data worthless. Even during the Silver years, PECOTA managed to produce individual projections that, in retrospect, whiffed badly. I believe that the data have always needed to be interpreted based on whether you make the same assumptions about a player that PECOTA does. I can name several players from past seasons for which PECOTA has seemed to have a multi-year blind spot (Chris B. Young, Jay Bruce, and Kelly Johnson off the top of my head) and after some bad blunders, I learned to adjust accordingly. The problems with internal consistency in this year's numbers are frustrating, but I don't see PECOTA churning out a steady stream of outlier projections. While some may believe that CHONE has a better read on (say) Kendry Morales simply because its team standings add up correctly, I'd argue that this alone does not inherently make it more accurate. I still feel that there is plenty of accuracy left in PECOTA, and that if it is less useful this year than in the past, the differences are not as significant as is sometimes maintained. (Of course, I draft in a traditional 5x5 league, so problems with UPSIDE or 10-year projections do not affect much of what I do.) As I see it, BP's biggest mistake this year (and last, admittedly) appears to be that they did not recognize the size of the problem back when there was sufficient time to deal with it. However the "problem" to me has more to do with getting PECOTA from 'good' to 'the best it can be', instead of from 'useless' to 'OK'. Your mileage may vary, but I still say the numbers can be used, and used successfully. Good luck with your draft.
beyond that, y'all could've communicated to your audience a little more regularly, and broadcast in a different way than has been done. if i were reliant on this information for my draft, i'd be furious.
that being said, i'm extremely sympathetic to the BP staff for the heat they're taking, especially considering the years of quality that have gone before 2010. i'm hoping the staff is learning from their experience to ensure history does not repeat itself.
We're stuck with the name PECOTA for branding reasons, but the product is no longer the same. It isn't clear whether it should be exactly the same or not--certainly asking that question is the correct thing to do. PECOTA was *not* a fantasy baseball tool at first. My impression from the beginning is that the focus is on identifying prospects before they identify themselves or prognosticating breakouts and collapses before they happen. It's too bad that building a successful business in baseball analysis requires turning to the fantasy world. PECOTA wasn't and isn't well built for fantasy--it's an aggregate tool being shoehorned into an individual game.
Basically, 2010 PECOTA == Microsoft: Pay us, then identify/fix our bugs for us!
That said, I continue to be dumbfounded by this year's comparables. Is Everth Cabrera really most comparable to the young Hanley Ramirez and Jimmy Rollins, and also very similar to the young Derek Jeter? Can you comment on whether the comps perhaps need to be tweaked along with the 10-year forecasts?
--------------
Everth Cabrera, 2009: .276 EQA
Derek Jeter, 1996: .279 EQA
Jimmy Rollins, 2001: .266 EQA
Hanley Ramirez, 2006: .291 EQA
This is not a defense or robust analysis or anything quite like that. Rather, I found it pretty cool. :)
Articles keep getting posted, analysis keeps getting churned out, readers keep reading.... But upon what are the articles, analysis and readers depending upon? Are we really to expect some point of clarity which allows us all to move forward with reliable data, predictions and analysis?
I can pick my own fantasy baseball team on my own just fine. Thats on me. But PECOTA.... That's on you guys, BP. I wish you the best and truly hope that you get it all sorted out.
Presumably it was clear during last season that a switch over needed to be made from the Nate Silver cobbled-together sheets into something more dynamic (and by the way - I've never understood the Silver hagiography that goes on here, nor been impressed by the next site he's gone on to run, but this makes me wonder if he isn't in fact the only person with a clue around these parts).
With that in mind, why not work on the program throughout the 2009 season and have a 2010 projection ready to roll out around October 1st? You guys shot yourself in the foot by waiting until February to get this going.
And while I've been a defender of the site in the past, you guys reap what you sow when you tell people that "Everything they know about baseball is wrong" and that your forecasts are "deadly accurate" ... and then you deliver a stinker like this.
The reality is, the product you are offering for a price is inferior to those being offered elsewhere for free.
Not a great path to prosperity in my view.
The following are a random sampling of projections. One comes from the current Depth Charts at BP. The other comes from a popular free system. Without identifying which is which, can it really be said that one is demonstrably inferior? Or that based on these numbers, that one should wonder whether anyone at BP has a clue?
Grady Sizemore
Brand X: .274/.375/.491
Brand Y: .272/.369/.484
Pablo Sandoval
Brand X: .310/.366/.504
Brand Y: .325/.368/.526
Elvis Andrus:
Brand X: .264/.327/.362
Brand Y: .266/.328/.367
I can't tell which of these brands is 'delivering a stinker,' or 'permanently impaired', and I suspect that most of the commenters on this article can't tell either. You will get no disagreement from me that the roll-out for PECOTA this year has been flawed at best. Certainly the site has taken a hit for that, and they've acknowledged their mistakes. However, instead of simply massaging the numbers to make it work out right, at least the site has been transparent about its efforts to correct the programming, and apologetic for the delays. That does not excuse their errors, or make it OK that PECOTA is still not 'locked,' but I think the scorn being heaped upon Dave and the rest of the BP team in this comment and others of its ilk is unwarranted. I also believe that the errors are not 'fundamental' ones, and that the fact that Adam Jones' 80th and 90th percentile projections don't add up, doesn't make BP's .283/.344/.459 inherently inferior to CHONE's .294/.349/.497. It doesn't mean that Christina Kahrl can't say anything intelligent about the Blalock signing. It doesn't mean that Will Carroll can't explain an arm injury, and it doesn't mean the Kevin Goldstein doesn't know how to evaluate a prospect anymore. It is an annoying logical error, but that on its own does not signify to me that BP no longer knows anything about baseball, or that I can get better analysis/information from Yahoo, Fangraphs, or MLB.com.
If you are in a non-keeper league, as far as I know there's no reason you shouldn't be able to make use of the projections in the book, or the ones released on the site in late January, to inform your draft strategy.
The long-term projections (and, by extension, the upside), the comps, the weighted means, the percentiles--we've had problems with all of those to some degree, and we're still working on fixes. But if what you'd generally use is the output from the PFM, none of those are relevant.
Seriously, I know there's been a discussion of McLouth's projection, but the one for Jones looks low across the board.
it LOOKS cool ... but,
My main frustration with BP.com this year is the lack of PECOTA cards linked from the PFM. Believe it or not, that really affects my draft prep.
I guess I'll just have to work with what I've got...
Although the McLouth projection screams 'proceed with caution' to me, I'll take a stab at answering your question. First off, McLouth's Depth Chart projection lists his 5x5 counting stats as 109/25/91/23/.267. This matches well with the 113/26/94/23/.276 he put up in 2008, so if the system simply sees this as a bounce-back year, then the numbers line up. Although McLouth's flyball rate dropped significantly last year and he struggled more against LHPs than in years past, his walk rate also increased and he joined an Atlanta team that should be far superior to the one he left in Pittsburgh. He missed time with a hammy injury that should be healed now, and rates a 'green' on Will's THR. If you look at your other prognostications, I think you will find that PECOTA's triple slash projections for McLouth are not far off these other systems at all. The big difference in counting numbers comes because PECOTA sees 713 PAs for McLouth, while others I've seen have him well below 600. As I said before, I plan to proceed with caution, but I don't know that I'd discount the projection completely.
The fact that PECOTA has some kinks is not ideal, but who here is drafting blindly with these projections? Marc's guidance, Will's THR's and damn near every article on this site should help all of us form opinions and make the calls.
The book is great, the site is great, the BP team is being upfront about the issues. Let's roll with it and move on.
PECOTA is data, not decision.
And a lot of that goes beyond PFM. I look to PECOTA to get an idea on upside, as I'm in a long term keeper league. And I don't have any faith in the long-term projections this year. They simply still do not even pass the smell test.
Is that enough to make me cancel my subscription? No. BP still has great content all over the place. It's just kind of irksome that I spent all winter waiting for this stuff to come out, and when it did, it didn't have any more reliable data than MARCEL or some other such one-year projection. Luckily my main draft isn't until April. I'm just hoping the kinks are worked out by then.
A), get those projected records down somehow; or,
B), put up a message box stating that you know they don't add up and you're working on that.
It just looks awful, putting such an obvious error out there for all to see.
As for the 10 year projections they still don't pass the sniff test. Nate McLouth is not going to show any meaningful decline as a player until age 35 ? Give me a piece of that action. Meanwhile, Adam Jones, who arrived in the majors at a much younger age than McLouth shows a drop off at age 33. Doesn't this fly in the face of what we know about career arc's ? Typically, it's the later arrivers who fade first, right ?
Having the PFM based on the depth chart is logical and not confusing. It does, however, make the accuracy of the projections dependent upon the accuracy and timeliness of the depth chart. We can’t expect PECOTA to determine who plays and who doesn’t. I suspect every fan has suffered through a manager who keeps a deserving player on the bench because of (insert frustrating excuse here).
I don’t know who currently updates the depth chart, but I would love to have it assigned to John Perrotto. The notes at the bottom of his On the Beat article often have real insight into who is likely to get playing time and who isn’t. NL and AL only fantasy leagues are usually won by late round picks, and playing time has a huge role in determining which players should be chosen in the later rounds. Having the chart updated more frequently (at least once/week) would be wonderful.
We do have internal discussions about playing time that become part of Clay's updates, and John provided a bunch of info earlier this month on that front.
The bottom line is that the frustration of myself and others relates directly to what we're used to seeing from BP - innovative thinking and great writing. We don't always get everything we want, but the fact that BP provides a forum for their customers to comment/complain/offer feedback is appreciated. Then taking the time to listen and address said feedback is even more appreciated.
I have all the confidence in the world that when you put together this many talented minds, that issues (like PECOTA) will be both addressed and resolved.
Merely solving the PECOTA issues will not be enough. BP also has to get PECOTA released in time to be useful, and they have flirted with that timeline for years. That it's the most accurate system out there is useless if it's not ready in time.
Indeed the Rangers, who have been consistently projected as having the highest slugging average in the majors, are now predicted to be 6th, tied with Toronto. This is also the first projection which shows them under 800 runs scored, and its significantly below that level. That doesn't pass the sniff test.
My guess is that the projections at the top are pre-thyroid diagnosis (e.g., 97 runs) and the forecasted numbers in the 2010 Forecast section post-thyroid diagnosis (e.g., 79 runs at the 90% level).
This also raises the question as to which of these numbers, if any, PFM uses, which upon examination appears to be the projections at the top of the page. Unfortunately, these appear to be sorely out of date.
in PFM, McLouth is hitting .267. I don't see this AVG in any of his PECOTA percentiles. I do see it in the Weighted Means Spreadsheet though
Guys, I love BP to death and have been a long and ardent follower, but if this is transparency, then I'd hate to see the "wool pulled over our eyes" by you guys.
How does this article explain why just a couple of weeks ago Alex Rodriguez had a .700 SLG% in his 90th percentile and now has a .630? That's not a playing time correction, and it sure as heck isn't defined in any of the changes you outlined above.
PECOTA was worthless weeks ago, and for us -- the subscribers -- to have any sense
of it having been reconciled, it would be nice if you guys would actually fess up to what in the world you did to screw these numbers up so badly and what you did to fix them. Sweeping it under the proverbial rug would not be a good first step, I would think.
I think it's telling that in 2003, when I was first introduced to the site, Baseball Prospectus told me everything I needed to know about statistics and value and the great world of sabermetrics in baseball. Now? I only come here for write-ups on prospects who are evaluated almost solely by scouts. I think the message has been lost somewhere, gentlemen.
I'd note that any comparisons to past iterations needs to include slash stats or (or preferably and) total value stats, like virtually every other forecasting comparison ever done, including by BP. Failing to include that basic comparison renders such an exercise of little value.
--JRM
Colin has reported some of his unpublished findings regarding this year's PECOTA.