1. Modular Information Aggregation with Combinatorial Prediction Markets Robin Hanson Department of Economics George Mason University

2. Weather Forecasting A Canonical Forecasting Arena Many public standardized forecasts Fast feedback, so lots of stats Forecasting analysis pioneer – methods, validation, aggregation, etc. Place to try out forecasting innovations?

3. Buy Low, Sell High price buy sell Will price rise or fall? E[ price change | ?? ] Lots of ?? get tried, price includes all! Pays $1 if Bush wins (All are gambling prediction info)

4. Todays Current Prices 7-19% Bird Flu confirmed in US. By Dec. 07 11-13% Bin Laden caught by 08. 54-60% Gonzales resigns by 08 16-18% US or Israel air strike on Iran by 08. 21-25% China overt military act on Taiwan by 2011 51-52% Darling next UK Chancellor 56-59% Conservatives win next UK election TradeSports.com

5. In direct compare, beats alternatives Vs. Public Opinion –I.E.M. beat presidential election polls 451/596 (Berg et al 01) –Re NFL, beat ave., rank 7 vs. 39 of 1947 (Pennock et al 04) Vs. Public Experts –Racetrack odds beat weighed track experts (Figlewski 79) If anything, track odds weigh experts too much! –OJ futures improve weather forecast (Roll 84) –Stocks beat Challenger panel (Maloney & Mulherin 03) –Gas demand markets beat experts (Spencer 04) –Econ stat markets beat experts 2/3 (Wolfers & Zitzewitz 04) Vs. Private Experts –HP market beat official forecast 6/8 (Plott 00) –Eli Lily markets beat official 6/9 (Servan-Schreiber 05) –Microsoft project markets beat managers (Proebsting 05)

6. Hollywood Stock Exchange 2000 Oscar AssessorError Feb 18 HSX prices1.08 Feb 19 HSX prices0.854 Feb 18 Tom1.08 Feb 18 Jen1.25 Feb 18 John1.22 Feb 18 Fielding1.04 Feb 18 DPRoberts0.874 Columnist Consensus1.05 Science 291:987-988, February 9 2001

7. Track Odds Beat Handicappers Figlewski (1979) Journal of Political Economy 14 Estimated on 146 races, tested on 46

8. Economic Derivatives Market Standard Error of Forecast Non-Farm Payrolls Retail TradeISM Manuf. Pur. Index Ave of 50 Experts 99.20.551.12 Market97.30.581.20 Sample size161211 Wolfers & Zitzewitz Prediction Markets (2004) Journal of Economic Perspectives

9. NFL Markets vs Individuals Servan-Schreiber, Wolfers, Pennock & Galebach (2004) Prediction Markets: Does Money Matter? Electronic Markets, 14(3). 1,947 Forecasters Average of Forecasts

10. Item1988199219962000All # big polls 59151157229596 Poll wins 25432156145 Market wins 34108136173451 % market 58%72%87%76% P-value0.1480.000 Accuracy and Forecast Standard Error of Prediction Markets Joyce Berg, Forrest Nelson and Thomas Rietz, July 2003.

11. Iowa Electronic Markets vs. Polls Accuracy and Forecast Standard Error of Prediction Markets Joyce Berg, Forrest Nelson and Thomas Rietz, July 2003.

12. Source:

13. InputsOutputs Prediction Markets Status Quo Institution Compare!For Same

14. Not Experts vs. Amateurs Forecasting Institution Goal: –Given same participants, resources, topic –Want most accurate institution forecasts Separate question: who let participate? –Can limit who can trade in market Markets have low penalty for add fools –Hope: get more info from amateurs?

15. Advantages Numerically precise Consistent across many issues Frequently updated Hard to manipulate Need not say who how expert on what At least as accurate as alternatives

16. Old Tech Meets New To gain info, elicit probs p = {p i } i, E p [x |A] (Let verify state i later, N/Q = people/questions) Old tech (~1950+): Proper Scoring Rules N/Q 1: works well, N/Q 1: hard to combine New tech (~1990+): Info/Predict Markets N/Q 1: works well, N/Q 1: thin markets The best of both: Market Scoring Rules –modular, lab tests, compute issues, …

17. Old Tech: Proper Scoring Rules When report r, state is i, reward is s i (r) p = argmax r S i p i s i (r), S i p i s i (p) 0 Quadratic (G. Brier 1950) s i 2r i – S k r k 2 Logarithmic (I. Good 52) s i log(r i ) –Unique: reward = likelihood (R. Winkler 1969) Offers info effort incentive (R. Clemen 02) –Stronger for simul. pick: pivot mech. (T. Page 88) In principle, can elicit complex joint distributions Long used in forecasting, test scoring, lab expers

18. Old Tech Issues Problems Incentives Number shy Cognitive bias Non risk-neutral State-dependent utility Combo explosion Disagreements Solutions Proper scoring rules Prob wheel, word menu Corrections Lottery payoffs Insurance game Dependence network Dictator per Q, ??

19. Opinion Pool Impossibile Task: pool prob. T(A) from opinions p n (A) Any 2 of IPP, MP, EB dictator (T= p d ) ! IPP = if A,B indep. in all p n, are indep. in T EB = commutes: pool, update on info MP = commutes: pool, coarsen states ( - field) (MP T = n=0 w n p n, with w n indep. of A) Really want pool via belief origin theory –General solution: let best traders figure it out?

20. New Tech Issues Problems Incentives Shy, complex utility Combo explosion Who expert on what Cognitive bias, pooling Thin markets (N/Q <~1) What is independent Solutions Bet with each other Same solutions Self-select Specialist traders Market scoring rules ??

21. Thin Market, No-Trade Problems Trade requires coordination –In Time: waiting offers suffer adverse selection –In Assets: far more possible assets than trades combo match call markets help some, but matching is NP-hard for all or nothing orders –If expect traders rare, dont bother to offer Most possible markets do not exist (also illegal) No-trade among rational, info-motivated –Need fools, risk-hedgers, or outside subsidy

22. Accuracy.001.01.1110 100 Q/N = Estimates per trader Market Scoring Rules Old Tech Meet New Simple Info Markets thin market problem Scoring Rules opinion pool problem

23. q1q1 Scoring Rule = Demand Curve (L. Savage 1971) Rule is demand q D (p) –p 0 solves q=0 User sells q 1, price discr. –Let p 1 be user belief Note: = 0 if p 1 = p 0 –p 0 is like rule belief Note: can reuse rule as q D (p) - q 1 (i.e., p 0p 1 ) –So this is a market maker! Price Quantity 1 0 $1 if A p0p0 Demand p1p1 Commodity:

24. Market Scoring Rule (MSR) MSRs act as scoring rules and info market makers –are sequentially reused combinatorial scoring rules Score rule: user t faces $ rule: D s i = s i (p t ) - s i (p t-1 ) Anyone can use scoring rule if pay off last user Automated market maker: price from net sales s –Tiny sale fee: p i (s) e i (s i s i + e i ) –Big sale fee: 0 1 S i p i (s(t)) s i ´(t) dt –Log score rule gives: p i (s) = exp( s i ) / S k exp( s k ) $ s (1) -s (0) $ e i if i

25. MSR Usage Concept User browses current probabilities, expectations –Can set assumptions, browse other values given them –Can see market value of portfolio given assumptions Upon finding an odd current value E(x|A) –Can see values price/trade history –Can see how far long/short are from past trades User proposes new value to replace old –Told exact bet required to implement change –Can accept, and/or make book orders & trading agents

27. The Vision Detailed consensus forecasts always visible re many locations, times, conditions, etc. Authorized traders can change any estimates. –Possibly conditional on many other estimates. –Could write program to change many estimates. –At least until their account runs out. Win if moved estimate closer to truth. Consistent winners gain more resources. Could even privatize, and/or let public try.

28. Laboratory Tests Joint work with John Ledyard (Caltech), Takashi Ishida (Net Exchange) Caltech students, get ~$30/session 6 periods/session, 12-15 minutes each Trained in 3var session, return for 8var Metric: Kulback-Leibler i q i log(p i /q i ) distance from market prices to Bayesian beliefs given all group info

29. Mechanisms Compared Survey Mechanisms (# cases: 3var, 8var) –Individual Scoring Rule (72,144) –Log Opinion Pool (384,144) Market Mechanisms –Simple Double Auction (24,18) –Combined Value Call Market (24,18) –MSR Market Maker (36,17)

30. Environments: Goals, Training Want in Environment: –Many vars, few related, guess which –Use both theory and data –Few people, specialize in variable sets –Can compute rational group estimates –Explainable, fast, neutral Training Environment: –3 binary variables X,Y,Z, 2 3 = 8 combos –P(X=0) =.3, P(X=Y) =.2, P(Z=1)=.5 –3 people, see 10 cases of: AB, BC, AC –Random map XYZ to ABC Case A B C 1 1 - 1 2 1 - 0 3 1 - 0 4 1 - 0 5 1 - 0 6 1 - 1 7 1 - 1 8 1 - 0 9 1 - 0 10 0 - 0 Sum: 9 - 3 Same A B C A -- -- 4 B -- -- -- C -- -- -- (Actually: X Z Y )

31. MSR Info vs. Time – 7 prices 0 5 10 15 Minutes 0 1 % Info Agg. = KL(prices,group) KL(uniform,group) 1-

32. Experiment Environment 8 binary vars: STUVWXYZ 2 8 = 256 combinations 20% = P(S=0) = P(S=T) = P(T=U) = P(U=V) = … = P(X=Y) = P(Y=Z) 6 people, each see 10 cases: ABCD, EFGH, ABEF, CDGH, ACEG, BDFH random map STUVWXYZ to ABCDEFGH Case A B C D E F G H 1 0 1 0 1 - - - - 2 1 0 0 1 - - - - 3 0 0 1 1 - - - - 4 1 0 1 1 - - - - 5 0 1 1 1 - - - - 6 1 0 0 1 - - - - 7 0 1 1 1 - - - - 8 1 0 0 1 - - - - 9 1 0 0 1 - - - - 10 1 0 0 1 - - - - Sum 6 3 4 10 - - - - Same A B C D E F G H A -- 1 2 6 -- -- -- -- B -- -- 7 3 -- -- -- -- C -- -- -- 4 -- -- -- -- D -- -- -- -- -- -- -- -- … (Really: W V X S U Z Y T )

33. MSR Info vs. Time – 255 prices 0 5 10 15 Minutes 0 1 % Info Agg. = KL(prices,group) KL(uniform,group) 1-

34. Combo Market Maker Best of 5 Mechs

35. Experiment Conclusions Experiments on complex info problems –Bayesian estimates far too high a standard 7 indep. prices from 3 people in < 4 minutes –Simple DA < Indiv. Score Rule ~ Opinion Pool ~ Combined Value < Market Scoring Rule 255 indep. prices from 6 people in < 4 min. –Combined Value ~ Simple DA ~ Indiv. Score Rule < Opinion Pool ~ Market Scoring Rule

36. Log Rule is Modular Consider bet: Changes p(A|B); only log rule keeps p(B) Also keeps p(C|A&B), p(C| A &B), p(C| B), I( A, B, C ), I( B, A, C ), I( C, A, B ), I( C, B, A ) – A is var, one of whose values is A, etc. –I( A, B, C ) iff p(A|B&C) = p(A|B) for all values Log rule uniquely keeps changes modular $1 if A&B p(A|B) $1 if B

37. No Cost For Combos! Total cost: C = s i true (p final ) - s i true (p initial ) Expected cost: E p [C] S i p i (s i (1 i ) - s i ( p )) For log MSR: = S( p) = - S i p i log( p i ) –Let state i = combination of base var values v i –S( p all ) S var S( p var ), for p var = { p var value v } v –So compared to cost of log rule for each var, all var/value combos cost no more!

38. How Compute? Simple: –store 2 N probs, asset values –Can integrate book orders Feasible: overlapping var patches –With simple MSR per patch, is Markov field –Allow trade only if all vars in same patch How pick/change patch structure? Arbitrage to make patches agree? A B C F E D H G

39. Arbitrage Is Not Modular A B C F E D H G 1.Everyone agrees on prices 2.Expert on A gets new info, trades 3.Arbitrage updates all prices 4.Expert on H has no new info, but must trade to restore old info! AH

40. Bayesian/Markov Networks Local info trades not require distant corrections if updates follow Bayes rule Bayesian/Markov net tech does this –Off the shelf exact tech if net forms a tree –Many approx. techs made for non-trees Some development needed –Must update user assets as well as prices –Need robust to gaming on errors – stochastic?

41. Summary How elicit informed estimates? –Scoring rules if N/Q 1, info/predict markets if 1 Market scoring rules do both: –Key insight … reused scoring rules are market makers –Browse billions of estimates, change ones want via bets –Lab tests confirm ability; are growing # groups using Log rule has many advantages –If bet on p(A|B), keeps p(B), I( A, B, C ), I( B, A, C ) … –Noisy choices easier to interpret –Costs no more for all var/value combos –Computation simplified, but still issues to explore