Research Prediction Markets and the Wisdom of Crowds David Pennock Yahoo! Research NYC

Research Outline Prediction Markets Survey What is a prediction market? Examples Some research findings The Wisdom of Crowds: A Story

Research Bet = Credible Opinion Which is more believable? More Informative? Betting intermediaries Las Vegas, Wall Street, Betfair, Intrade,... Prices: stable consensus of a large number of quantitative, credible opinions Excellent empirical track record Hillary Clinton will win the election “I bet $100 Hillary will win at 1 to 2 odds”

Research Example: March Madness

Research Screen capture 2007/05/18 More Socially Redeemable Example

Research A Prediction Market Take a random variable, e.g. Turn it into a financial instrument payoff = realized value of variable $1 if$0 if I am entitled to: Bird Flu Outbreak US 2007? (Y/N) Bird Flu US ’07

Research Why? Get information price  probability of uncertain event (in theory, in the lab, in the field,...more later) Is there some future event you’d like to forecast? A prediction market can probably help

Research A Prediction Market Take a random variable, e.g. Turn it into a financial instrument payoff = realized value of variable = 6 ? = 6 $1 if  6 $0 if I am entitled to: US’08Pres = Dem? 2008 CA Earthquake?

Research Aside: Terminology Key aspect: payout is uncertain Called variously: asset, security, contingent claim, derivative (future, option), stock, prediction market, information market, gamble, bet, wager, lottery Historically mixed reputation Esp. gambling aspect A time when options were frowned upon But when regulated serve important social roles...

Research Getting Information Non-market approach: ask an expert How much would you pay for this? A: $5/36  $ caveat: expert is knowledgeable caveat: expert is truthful caveat: expert is risk neutral, or ~ RN for $1 caveat: expert has no significant outside stakes = 6 $1 if  6 $0 if I am entitled to:

Research Getting Information Non-market approach: pay an expert Ask the expert for his report r of the probability P( ) Offer to pay the expert $100 + log r if $100 + log (1-r) if It so happens that the expert maximizes expected profit by reporting r truthfully caveat: expert is knowledgeable caveat: expert is truthful caveat: expert is risk neutral, or ~ RN caveat: expert has no significant outside stakes = 6  6 “logarithmic scoring rule”, a “proper” scoring rule

Research Getting Information Market approach: “ask” the public—experts & non- experts alike—by opening a market: Let any person i submit a bid order: an offer to buy q i units at price p i Let any person j submit an ask order: an offer to sell q j units at price p j (if you sell 1 unit, you agree to pay $1 if ) Match up agreeable trades (many poss. mechs...) = 6 $1 if  6 $0 if I am entitled to: = 6

Non-Market Alternatives vs. Markets  Opinion poll  Sampling  No incentive to be truthful  Equally weighted information  Hard to be real-time  Ask Experts  Identifying experts can be hard  Incentives  Combining opinions can be difficult  Prediction Markets  Self-selection  Monetary incentive and more  Money-weighted information  Real-time  Self-organizing

Non-Market Alternatives vs. Markets  Machine learning/Statistics  Historical data  Past and future are related  Hard to incorporate recent new information  Prediction Markets  No need for data  No assumption on past and future  Immediately incorporate new information

Function of Markets 2: Risk Management  If is bad for me, I buy a bunch of  If my house is struck by lightening, I am compensated. $1 if $0 otherwise

Risk Management Examples  Insurance  I buy car insurance to hedge the risk of accident  Futures  Farmers sell soybean futures to hedge the risk of price drop  Options  Investors buy options to hedge the risk of stock price changes

Financial Markets vs. Prediction Markets Financial MarketsPrediction Markets PrimarySocial welfare (trade) Hedging risk Information aggregation SecondaryInformation aggregationSocial welfare (trade) Hedging risk

Giving/Getting Information What you can say/learn % chance that –Hillary wins –GOP wins Texas –YHOO stock > 30 –Duke wins tourney –Oil prices fall –Heat index rises –Hurricane hits Florida –Rains at place/time Where –IEM, Intrade.com –Intrade.com –Stock options market –Las Vegas, Betfair –Futures market –Weather derivatives –Insurance company –Weatherbill.com

Screen capture 2007/05/18

Research Intrade Election Coverage

Research

Screen capture 2007/05/18 Screen capture 2007/05/18

Play money; Real predictions

Cancer cured by 2010 Machine Go champion by

Research Yahoo!/O’Reilly Tech Buzz Game

An Incomplete List of Prediction Markets  Real Money  Iowa Electronic Markets (IEM),  TradeSports,  InTrade,  Betfair,  Gambling markets? sports betting, horse racetrack …  Play Money  Hollywood Stock Exchange (HXS),  NewsFutures,  Yahoo!/O’REILLY Tech Buzz Game,  World Sports Exchange (WSE),  Foresight Exchange,  Inkling Markets  Internal Prediction Markets  HP, Google, Microsoft, Eli-Lilly, Corning …

More Prediction Market Games BizPredict.com CasualObserver.net FTPredict.com InklingMarkets.com ProTrade.com StorageMarkets.com TheSimExchange.com TheWSX.com Alexadex, Celebdaq, Cenimar, BetBubble, Betocracy, CrowdIQ, MediaMammon,Owise, PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds

Catalysts Markets have long history of predictive accuracy: why catching on now as tool? No press is bad press: Policy Analysis Market (“terror futures”) Surowiecki's “Wisdom of Crowds” Companies: –Google, Microsoft, Yahoo!; CrowdIQ, HSX, InklingMarkets, NewsFutures Press: BusinessWeek, CBS News, Economist, NYTimes, Time, WSJ,...

Does it work?  Yes, evidence from real markets, laboratory experiments, and theory  Racetrack odds beat track experts [Figlewski 1979]  Orange Juice futures improve weather forecast [Roll 1984]  I.E.M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002]  HP market beat sales forecast 6/8 [Plott 2000]  Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’03][Schmidt 2002]  Market games work [Servan-Schreiber 2004][Pennock 2001]  Laboratory experiments confirm information aggregation [Plott 1982;1988;1997][Forsythe 1990][Chen, EC’01]  Theory: “rational expectations” [Grossman 1981][Lucas 1972]  More later …

Does it work? Yes... Evidence from real markets, laboratory experiments, and theory indicate that markets are good at gathering information from many sources and combining it appropriately; e.g.: –Markets like the Iowa Electronic Market predict election outcomes better than polls [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002] –Futures and options markets rapidly incorporate information, providing accurate forecasts of their underlying commodities/securities [Sherrick 1996][Jackwerth 1996][Figlewski 1979][Roll 1984][Hayek 1945] –Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’03][Schmidt 2002]

Does it work? Yes... E.g. (cont’d): –Laboratory experiments confirm information aggregation [Plott 1982;1988;1997][Forsythe 1990][Chen, EC-2001] –And field tests [Plott 2002] –Theoretical underpinnings: “rational expectations” [Grossman 1981][Lucas 1972] –Procedural explanation: agents learn from prices [Hanson 1998][Mckelvey 1986][Mckelvey 1990][Nielsen 1990] –Proposals to use information markets to help science [Hanson 1995], policymakers, decision makers [Hanson 1999], government [Hanson 2002], military [DARPA FutureMAP, PAM] –Even market games work! [Servan-Schreiber 2004][Pennock 2001]

Research Example: IEM 1992 [Source: Berg, DARPA Workshop, 2002]

Research Example: IEM [Source: Berg, DARPA Workshop, 2002]

Research Example: IEM [Source: Berg, DARPA Workshop, 2002]

AAAI’04 July 2004MP1-35 Example: IEM [Source: Berg, DARPA Workshop, 2002]

AAAI’04 July 2004MP1-36 Example: IEM [Source: Berg, DARPA Workshop, 2002]

AAAI’04 July 2004MP1-37 Contract: Pays $100 if Cubs win game 6 (NLCS) Price of contract (=Probability that Cubs win) Cubs are winning 3-0 top of the 8 th 1 out. Time (in Ireland) Fan reaches over and spoils Alou’s catch. Still 1 out. The Marlins proceed to hit 8 runs in the 8 th inning [Source: Wolfers 2004] Speed: TradeSports

Research Does money matter? Play vs real, head to head Experiment 2003 NFL Season ProbabilitySports.com Online football forecasting competition Contestants assess probabilities for each game Quadratic scoring rule ~2,000 “experts”, plus: NewsFutures (play $) Tradesports (real $) Used “last trade” prices Results: Play money and real money performed similarly 6 th and 8 th respectively Markets beat most of the ~2,000 contestants Average of experts came 39 th (caveat) Electronic Markets, Emile Servan- Schreiber, Justin Wolfers, David Pennock and Brian Galebach

Research

Research Does money matter? Play vs real, head to head Statistically: TS ~ NF NF >> Avg TS > Avg

AAAI’04 July 2004MP1-41 Real markets vs. market games HSX IEM average log score arbitrage closure

AAAI’04 July 2004MP1-42 Real markets vs. market games HSXFX, F1P6 probabilistic forecasts expected value forecasts 489 movies forecast sourceavg log score F1P6 linear scoring-1.84 F1P6 F1-style scoring-1.82 betting odds-1.86 F1P6 flat scoring-2.03 F1P6 winner scoring-2.32

Research Survey Story A Wisdom of Crowds Story ProbabilitySports.com Thousands of probability judgments for sporting events Alice: Jets 67% chance to beat Patriots Bob: Jets 48% chance to beat Patriots Carol, Don, Ellen, Frank,... Reward: Quadratic scoring rule: Best probability judgments maximize expected score Research Opinion 1/7

Research Individuals Most individuals are poor predictors 2005 NFL Season Best: 3747 points Average: -944Median: ,298 out of 2,231 scored below zero (takes work!)

Research Individuals Poorly calibrated (too extreme) Teams given < 20% chance actually won 30% of the time Teams given > 80% chance actually won 60% of the time

Research The Crowd Create a crowd predictor by simply averaging everyone’s probabilities Crowd = 1/n(Alice + Bob + Carol +... ) 2005: Crowd scored 3371 points (7th out of 2231) ! Wisdom of fools: Create a predictor by averaging everyone who scored below zero 2717 points (62nd place) ! (the best “fool” finished in 934th place)

Research The Crowd: How Big? More:

Research Can We Do Better?: ML/Stats Maybe Not CS “experts algorithms” Other expert weights Calibrated experts Other averaging fn’s (geo mean, RMS, power means, mean of odds,...) Machine learning (NB, SVM, LR, DT,...) Maybe So Bayesian modeling + EM Nearest neighbor (multi-year) [Dani et al. UAI 2006]

Research Can we do better?: Markets