Welcome to our hastily-implemented beta tester program. If you can read this, you've been vocal about quality in the past, and we want to know what you think, so I've added you to our beta tester user group, and you'll have access to data early. (Please pardon my presumption if you aren't interested; you can either ignore this whole thing altogether, or shoot me an email at dpease@baseballprospectus.com and we'll remove the designation on your account.) Thank you to dianagram and other users for suggesting this program.
The first order of business is the PECOTA cards. As I said last week, we're generating the pitcher cards and we should have them out in the next couple of days. In the meantime, we've got the third and hopefully final revision of the hitter cards ready for you to look at.
Please click here to access the beta testers only version of the cards.
Notes on this version:
-
We haven't set up a beta-tester-only header, so any links to the PECOTA cards in the header, and the search box in the header, will still take you back to the public PECOTA beta cards. Sorry–we'll flesh out the program to handle this more gracefully as we get it established. In the meantime, use your browser's "Back" button, or just head straight to the beta tester index to visit other players.
-
We've fixed the OBP bug for the 2010 percentiles projections.
-
No Kyle Gibon and Dustin Ackley yet. We might be able to get them–I'm not sure.
-
Only 20 comps per for these, but all 100 will be there once we switch these over to their final resting place.
-
The stats in the "biographical" box are the playing time projection adjusted stats at 50th percentile, if a player has playing time projected. Any differences in rates should be the result of roundoffs. For example, consider Albert Pujols. At his 50th percentile, Pujols is projected to go 164 for 511, for a .321 batting average. His playing time projection (used in the depth charts and PFM) changes that from 608 PA to 679, which results in (679/608*511) = 571 at bats and (679/608*164) = 183 hits, but 183/571=.32049, rounding down to .320. If a player does not have any playing time projected, the stats in the box are his PECOTA 50th percentile projection.
- We changed the color of the totals and weighted means lines to be easier to read.
The beta testers only version of the hitter cards is available at https://legacy.baseballprospectus.com/ddjp.
Please leave any comments you have right here, or feel free to contact us with them. Thanks again for your help.
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.
Subscribe now
Looking at Josh Hamilton, under his 10 year projection I see a TAv of .296 for 2010, but when I look at his 2010 projection by percentiles, that looks like it would be a level of performance around his 85th percentile. So, something doesn't quite make sense.
I'll be busy tonight, but hope to take a good look at this tomorrow.
I gave a look at Matt Kemp just for starters:
1: I love the layout, it is much cleaner and easier to read.
2: I like the added tAv data for comparables
3: The links to nearly everything ever said about the player are wonderful
With that said, I am moderately surprised by PECOTA's read of Matt Kemp: That age 24 was his career year by a fair margin. That might be because so many of his 20 comparables are downward arrows, though. Nonetheless, it is awesome to be able to see at a glance how those comparables performed in the past, and I know I will really appreciate that feature. It makes the context of comparable players so much more useful.
Things that look wrong:
1. Some of the Weighted Means issues I've discussed earlier look fixed (Druby, for instance.) But the TAv weighted means still look way wrong (see, for instance, Nolan Reimold.) Adding the percentiles and dividing by nine should come close; it doesn't and in fact no matched pair (90+10/2, 80+20/2) comes close. (I see tbwhite points this out also.)
2. The ski-slope playing time problem in the 10-year PECOTA's still exists. Jesus Montero's not going to be a half-time player in six years hitting 300/357/519. Jason Heyward, same deal.
3. ccmonster's right.
4. Something's wrong with the EqA translations. For 2009, St. Louis reads as a pitcher's park for Holliday (Eq's: 380/437/658 on raw data of 353/426/604) but a neutral park for Glaus (172/250/241 for both). This may be yet another rounding issue due to low PA's by Glaus in St.L, but if so, it's a good reason to not round; there's especially no reason to round to generate EqA's. Nate did not round for anything; the display of homers was decorative, but the actual Excel numbers had plenty of entries like 11.975 homers. Going to a rounding system has generated errors. If you are measuring old P with the rounding, you're going to get the projected slash stats wrong.
Good luck. Off to work.
--JRM
The full Heyward projection for 2016 has TAvs between .324 (550 PA, comping to Juan Gonzalez, Miguel Cabrera and Adam Dunn at the 90% side) to .232 (113 PA, comping Chris Snelling, Jeff Reed, and Darnell Coles) at 10%. The weighted mean will tilt the rate stats strongly towards the high end, but the PA still get held down.
It isn't immediately obvious, but anyone making serious use of 10 year weighted means is informed enough to understand it, I think.
Without reading all the other comments below, I'd like to chime in on a potential anomaly I'm seeing in the 10-year projections of both Heyward and Montero, both age 20. I think we can all agree that these are guys who we would like Pecota to nail. And I am just not confident that is entirely the case.
Looking at their 10-year projection SuperVORP and TAv charts, both of them peak at age 23 - in every one of the projection percentiles. Age 24 and 25 are declines - across the board in Heyward's case, and with some Age 25 stability in the upper projections for Montero. After Age 25, and into their so-called peak years, we see a slight spread in their projections. The upper percentile projections give them some respectability, but the red-50% line and other percentiles resign them to medicority or worse by age 27.
Mike Stanton, another 20 year old, is seeing a similar profile. His upper two projections show .290 TAvs well through his age 27 peak, but his 50%TAv stays under .250 for from age 25 onward.
I suppose it's possible that these three players all share the characteristics of an early peak, with only a slim chance at being star players from age 24 on. But it just seems odd to me that each player has such a decline in their 50% projection from Age 23 to Age 27 seasons:
Heyward: .289 TAv to .265
Montero: .283 TAv to .265
Stanton: .252 TAv to .229
These seem to be 'better' projections than the wide spread projections we had seen in the first round of beta cards. But something still strikes me as being odd, that these guys are all going to peak at age 23 across all their projections, and have their 50th percentile trend down afterwards.
Is this happening because of the "out of baseball" projections holding down his 50% projection? Just doesn't seem right to me. What happens if you toss those out of baseball percentages out? Does the 50% line more closely resemble a career path with peaks in the age 26 through 29 seasons?
Again, thanks for including me as a beta-tester, and I'm sure I am going to learn more from this than you will. But hopefully you get something useful from my inclusion.
I looked at a few players (Carlos Gonzalez, Dexter Fowler, Cameron Maybin) and was extremely surprised at the differences, which I don't recall typically having been so large in past years.
I then looked up a few players from last year - 2009 cards are still available if you search for a player and choose to see their PECOTA card - and sure enough, the differences just don't *seem* nearly as large in the past (I checked Ian Stewart and Matt Kemp for 2009).
Obviously not a ton of empirical evidence there, just an observation. As I have time, I'll keep trolling to see if anything else stands out.
Player ('09 diff, '10 diff)
Jay Bruce (.026, .007)
Carlos Gonzalez (.031, .009)
Matt Kemp (.018, .004)
Just a few examples.
Generally, the median and weighted mean should not vary based on the variances at all, *assuming* that the distributions are identically distributed. However, in this case, even if the rate statistic distributions were identically distributed, the playing time distributions are NOT - higher percentile projections are accompanied with more playing time, so therefore a projection with greater variance should (in my opinion, based on the above) lead to a greater difference between median and weighted mean. (So I think we disagree on this point.)
More interestingly: are this year's projections more *skewed* than prior year projections? Because - aside from the difference being caused by the playing time as noted above - a difference in skewness would be a far more interesting explanation....
The EqA displays for comparables are helpful but of limited use: since the up/down is based on whether that player was over or under what pecota wouldve predicted for him, the EqA stat without context doesn't tell us if that red arrow means someone had a .277 eqa vs a predicted .280, or a predicted .308.
It would be useful to do one of the following:
1: list the predicted EqA and let us click the name to see the actual result.
Or
2: list both predicted and actual EqA, with one color-coded so we can easily identify it.
This would let us gauge the magnitude of a comparable's impact on the prediction.
I think this would improve our ability to understand comparable impacts.
--JRM
Also, to reiterate what some others say, I love the new format. It's a great idea to link to every mention of the player's name and the graphics look good.
One very minor point that only very slightly bugs me: why not have a link on each PECOTA card to the BP home page? As it is now (and always has been), you have to go back to the PECOTA home page before you go back to the BP home page.
As far as I can remember, the comparables adjust the projections last (yes?), which would mean that there aren't really comparable effects on breakput and collapse rates. Some player types, especially younger, are probably predisposed to exceeding their projections gigantically if they meet x criteria (one imagines those toolsy guys who finally get it with plate discipline) so I could see being able to note that comparables were only beating the projection on breakout leaps as valuable added information, for instance.
Typing this on a blackberry makes for some runon sentences, apologies.
Its possible if comparables are adjusting breakout rates etc that this is extraneous data but from my understanding of pecota, the comparable effects are not that advanced. In that case, I would shoot for aesthetic effect. However, just the +/- info would satisfy a lot of my curiousity and I am not even a fantasy player. I just like trying to get better perspective on the development and evolution of my team and their competition.
I do see the note about changes being made tonight, so this might be another fluke.
I see there are some Beta changes coming tonight; that's good.
On an aside, can I suggest that things like prior OBP problems and weighted mean problems be acknowledged in a public post at some point? Some note that there were systemic problems with the 2010 PECOTA's seems like it would help regain trust, not just from the pitchfork-wielding mob, but also from those who saw what was illuminated by the torches.
The step forward in opening the drawbridge to the mob in this exercise is a positive.
Good luck on tonight's update.
--JRM
One minor quibble - and this may just be a function of the beta pages being temporarily housed at the moment - is that on last year's pages, the header of the internet window was the player's name, whereas on the beta cards it's "Baseball Prospectus - Your Source For All Things Baseball". It's much more convenient to have the player name in the header, where it's visible both on the top of each internet window, as well as on the taskbar. I've found myself on numerous occasions while perusing the current beta cards asking myself..."whose card am I looking at again?" And answering that question involves scrolling all the way to the top of the page, or inferring from the player comments at the bottom of the page. This issue doesn't take place with the 2009 cards.
Again, this may be a temporary issue, but I wanted to point it out.
I used to like using MLVr to sort position players by offensive value for Scoresheet purposes.
I would be interested in knowing why the stat has fallen to disuse, and if there will ever be an attempt to make it more relevant if its felt to be inadequate or uninformative.
Player (2009) (2010)
Marlon Byrd (.088, .088) (.072, ,017)
Aramis Ramirez (.074, .070) (.073, .052)
Adam LaRoche (.077, .085) (.088, 056)
Kelly Johnson (.075, .064) (.095, .035)
So, I think this is why there is a larger difference between median and weighted mean forecasts in 2010 - the distributions are skewed upward in 2010 versus 2009.
I'm heading out for the evening. I apologize for the volume of posts!
I just looked at 15 2009 cards and 15 2010 cards; I think the effect you cite is real, and substantial. I think there's something wrong there at the low-end percentiles.
Great catch, worthwhile post, and I'd love to know the reasons this is happening. A strong piece of evidence for this Beta Blog and wisdom of at least part of a crowd (or mob, as the case may be.)
I might be wrong; this might not be as pervasive as it looks like. But it looks pretty damn pervasive; I found a handful of players with similar differences this year, no one with a long 10% tail, and lots with big 90% performances. I'm guessing this stems from the same issues as the 10-year projections - out of baseball isn't considered a performance decline, and that's one of the causes of being out of baseball.
Are we handing out Great Catch Awards? One to Jivas, assuming he's right, seems in order.
--JRM
However, the differences are larger for players who should have significant collapse rates (like Marlon Byrd).
Is it possible that collapse rate is not properly being factored into hitter percentiles, but breakout is (or breakout is being over-emphasized and collapse is functioning properly), creating an upwards bias?
Just from browsing some cards myself, it does look to me like these gaps are larger with players who have disproportionate collapse to breakout rates, for example:
Casey Blake (breakout 5%, 50th to 90th vector .099 slg, collapse 27%, 50th to 10th vector only .048 slg).
James Loney (breakout 18%, 50th to 90th vector .075 slug, collapse 10%, 50th to 10th vector only .034 slg).
I picked those two players because they have identical median slugging projections (.424/.425) and identical park effects (one could surmise that parks like GABP would jack breakout seasons up even further than they would help mitigate collapse seasons, so I didn't want to pick players with different park effects).
I wanted to quote the BETA for both players but that statistic does not seem present on the cards anymore; I really did miss this measure of prediction reliability and would like to see it re-implemented in some fashion.
Also, prior year cards had a Similarity Index figure to help identify the extent to which the comparable players matched with the target player (e.g. Ichiro's 17 indicating that his comps were less meaningful than average). Is there a reason why this has been omitted this year?
Thanks....
My limited understanding of PECOTA is that the system makes predictions based on performance of previous players. It follows that knowing how good a fit a current player has to previous data is more valuable than knowing who to whom the player is being compared.
It then determines the most similar players at position and age and, independently, generates a projection for them based on that data. It then compares the projection to what that player -did- do in the next season. If they under-performed, PECOTA slightly reduces its projection. if they over-performed, it benefits the projection across the board. These effects are slight and their weighting is dependent on the strength of the similarity score, so guys like Ichiro who always had very low similarity scores got less from their comparables than did players with more traditional profiles.
PECOTA is not solely based on specific comparables and most of its body-type and statistical analysis that establishes the bulk of the projection are based on much more meaty statistical analysis of what things are significant factors in player performance. The comparables help tune PECOTA's accuracy, but they are a fine-tuning mechanism, not the bulk of the projection system.
Being able to see the similarity scores is helpful because for players like Ichiro you are basically informed that the comparables are specious at best (Ichiro's best comps have traditionally had similarity scores that other players might see with their 150-200th best comps). The impact of those comparables on Ichiro's projections is very small, and is part of why PECOTA has a harder time with players like him, Randy Johnson, or Barry Bonds in the past. In those situations, it's forced to go with only its baseline weighting of factors, for players who have always succeeded in the face of overwhelming trends against them. The comparable system helps control that within the normal bounds where there are comparables. For the really extraordinary outliers, PECOTA struggles a little more because it has so much less context to work with.
Some of the 5x5 categories have changed a bit ...
As an example:
Mariano Rivera (old,new) 5/40/58/3.19/1.14,5/40/64/2.56/1.03
I don't have a PECOTA card to reference for him, but SOMETHING is different.
Haven't seen any massive changes yet, but looking. With the settings of my league entered, McLouth still rates just above Kemp and Werth in CF, but behind Sizemore.
Mariano jumped ahead of Papelbon in the rankings of the relief pitchers, but Papelbon is still #2 (not that I draft top closers, but I try to keep an eye on what the competition might be looking for).
It would be nice to have historical VORP data available as it was in previous PECOTA cards.
In sampling a number of the hitters, the weighted mean numbers seem to be consistently higher than the 50th percentile data. I can understand this if a player is expected to be a starter, but this shouldn't be the case in a reserve player, unless I'm not understanding the weighted mean concept correctly. For example, Alberto Callaspo's 50th percentile VORP is 14.8, his playing time adjusted VORP under the bio data is 10.3 and his weighted mean VORP is 18.5.
If the 50th percentile data is more accurate for hitters than the weighted means, why are the ten year forecasts based on the weighted means?