tag:blogger.com,1999:blog-1106308362982890799.post8119460212072388842..comments2014-09-22T02:26:11.298-07:00Comments on Inherent Uncertainty: Prediction and the Meaning of ProbabilityJake Abernethyhttp://www.blogger.com/profile/15806900044617621815noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-1106308362982890799.post-63190111464264652852010-10-10T16:59:45.447-07:002010-10-10T16:59:45.447-07:00Thanks to anonymous commenter for those last two p...Thanks to anonymous commenter for those last two papers, they seem quite interesting!Jake Abernethyhttp://www.blogger.com/profile/15806900044617621815noreply@blogger.comtag:blogger.com,1999:blog-1106308362982890799.post-31181084999916767152010-10-06T09:23:45.229-07:002010-10-06T09:23:45.229-07:00This interpretation gives the Game Theoric Probabi...This interpretation gives the Game Theoric Probability of Shafer and Vovk (http://www.probabilityandfinance.com/). For the online learning setting this gives Defense Forecasting (http://arxiv.org/abs/cs.LG/0505083)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1106308362982890799.post-24576880328834831212010-09-22T16:51:27.449-07:002010-09-22T16:51:27.449-07:00Regarding the last couple of comments: it's no...Regarding the last couple of comments: it's not just being unbiased. It's being unbiased on *all* predictions. For a stationary distribution of outcomes, that's easy as mentioned, simply predict the empirical mean! But what if the outcomes are adversarially chosen? In this case calibration can still be achieved, which i think is still pretty surprising.Jacob Abernethyhttp://www.eecs.berkeley.edu/~jakenoreply@blogger.comtag:blogger.com,1999:blog-1106308362982890799.post-42476326751782836792010-09-22T16:41:24.018-07:002010-09-22T16:41:24.018-07:00Exactly. In that sense, "calibrated" is ...Exactly. In that sense, "calibrated" is a bit like "unbiased." An estimator that always guesses the mean is unbiased, but pretty useless.jneemhttp://www.blogger.com/profile/15498452240988618234noreply@blogger.comtag:blogger.com,1999:blog-1106308362982890799.post-57391927617722057302010-09-22T03:35:43.753-07:002010-09-22T03:35:43.753-07:00Correct me if I am wrong, but doesn't this not...Correct me if I am wrong, but doesn't this notion of calibrated imply that a weather forecaster who simply makes predictions using only the correct prior (while ignoring the current state of the weather) can claim to be calibrated? If that is the case, then an uncalibrated forecast is clearly bad, but a calibrated forecast isn't necessarily all that good either.Bayesian Empirimancerhttp://www.blogger.com/profile/18110557307446065462noreply@blogger.comtag:blogger.com,1999:blog-1106308362982890799.post-18220663722578239972010-09-21T20:07:45.869-07:002010-09-21T20:07:45.869-07:00I'm not sure I completely understand your noti...I'm not sure I completely understand your notion of calibrated forecaster but your motivation and examples remind me of proper scoring rules in economics (or what Bob and I called "proper losses" in our ICML paper last year). If you look at Savage's "Elicitation of Personal Probabilities and Expectations" the derivation of proper scoring rules uses a similar trading/gambling framework.<br /><br />For a more modern take, you may want to look at Lambert, Pennock, and Shoham's "Eliciting Properties of Probability Distributions" from EC'08 (and the follow-up paper for classification in EC'09) if you haven't already seen them.Markhttp://mark.reid.name/noreply@blogger.com