1. Information Aggregation: Experiments and Industrial Applications Kay-Yut Chen HP Labs

2. Experimental Economics Program Agenda Lessons from HP Information Markets (Chen and Plott 2002) Scoring Rules and Identification of Experts (Chen, Fine and Huberman 2004) (Chen and Hogg 2004) Public Information (Chen, Fine and Huberman 2004)

3. Experimental Economics Program HP Information Markets (Chen and Plott) Summary of Events – 12 events, from 1996 to 1999 – 11 events sales related – 8 events had official forecasts Methodology & Procedures – Contingent state asset (i.e. winning ticket pays $1, others $0) – Sales amount (unit/revenue) divided into (8-10) finite intervals – Web-based real time double-auction – 15-20 min phone training for EVERY subject – Market open for one week at restricted time of the day (typically lunch and after hours) – Market size: 10-25 people

4. Experimental Economics Program

5. Results Abs % Errors of IAM Predictions Last Interval IgnoredLast Interval Mass at Lower Bound EventAbsolute % errors of HP forecasts Average last 60% trade Average last 50% trade Average last 40% trade Average last 60% trade Average last 50% trade Average last 40% trade 213.18%4.61%4.57%4.68%5.63%5.68%5.80% 359.55%57.48%55.72%54.60%59.25%57.46%56.32% 48.64%7.84%8.15%8.52%6.45%6.77%7.13% 532.08%30.93%31.57%31.83%29.74%30.33%30.48% 629.69%24.23%24.54%25.30%22.94%23.22%23.93% 74.10%7.33%7.02%6.71%5.35%4.91%4.55% 80.11%2.00%2.35%1.83%1.53%1.39%1.00% 928.31%23.85%24.85%24.39%17.55%17.32%16.54% T-test P-value0.0790.0840.0710.0340.0260.022 Random variable x=official error – market error H0: mean of x=0 Alternate: mean of x>0

6. Experimental Economics Program Business Constraints and Research Issues Not allowed to “bet” players’ own money -> stakes limited to an average of $50 per person Time horizon constraints -> 3 months to be useful Recruit the “right” people Asset design affects the results (How to set the intervals?) Thin markets (sum of price ~ $1.11 to $1.31 over the dollar) – Few players – Not enough participation

7. Experimental Economics Program Reporting with Scoring Rule Reports of Probability Distribution ABC Outcome p1p2p3 Pays C1+C2*Log(p3)

8. Experimental Economics Program Information Aggregation Function If reports are independent, Bayes Law applies …

9. Experimental Economics Program Two Complications Non-Risk Neutral Behavior Public Information

10. Experimental Economics Program Dealing with Risks Attitudes: Two-Stage Mechanism Event 1 Event 2 Event 3 Event 4 Event 5 Event 6 Event 7 Event 8 Stage 1: Information Market Call Market to Solicit Risk Attitudes Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Nonlinear Aggregated Function Time

11. Experimental Economics Program Second Stage: Aggregation Function Bayes Law with Behavioral Correction i =r(V i / i )c Holding value/Risk - measure relative risk of individuals Normalizing constant for individual risks “market” risk ~sum of prices/winning payoff

12. Experimental Economics Program Experiments: Inducing Diverse Information ABC Outcome Box of Balls A B C C C * In actual experiments, there are TEN states Random Draws Provide Info

13. Kullback-Leibler = 1.453 Comparison To All Information Probability Experiment 4, Period 17 No Information

14. Experimental Economics Program Kullback-Leibler Measure Relative entropy Always >=0 =0 if two distributions are identical Addictive for independent events

15. Kullback-Leibler = 1.337 Comparison To All Information Probability Experiment 4, Period 17 1 Player

16. Kullback-Leibler = 1.448 Comparison To All Information Probability Experiment 4, Period 17 2 Players Aggregated

17. Kullback-Leibler = 1.606 Comparison To All Information Probability Experiment 4, Period 17 3 Players Aggregated

18. Kullback-Leibler = 1.362 Comparison To All Information Probability Experiment 4, Period 17 4 Players Aggregated

19. Kullback-Leibler = 0.905 Comparison To All Information Probability Experiment 4, Period 17 5 Players Aggregated

20. Kullback-Leibler = 1.042 Comparison To All Information Probability Experiment 4, Period 17 6 Players Aggregated

21. Kullback-Leibler = 0.550 Comparison To All Information Probability Experiment 4, Period 17 7 Players Aggregated

22. Kullback-Leibler = 0.120 Comparison To All Information Probability Experiment 4, Period 17 8 Players Aggregated

23. Kullback-Leibler = 0.133 Comparison To All Information Probability Experiment 4, Period 17 9 Players Aggregated

24. Comparison To All Information Probability Experiment 4, Period 17

25. Experimental Economics Program KL Measures for Private Info Experiments 1.977 (0.312)1.222 (0.650)0.844 (0.599)0.553 (1.057) 1.501 (0.618)1.112 (0.594)1.128 (0.389)0.214 (0.195) 1.689 (0.576)1.053 (1.083)0.876 (0.646)0.414 (0.404) 1.635 (0.570)1.136 (0.193)1.074 (0.462)0.413 (0.260) 1.640 (0.598)1.371 (0.661)1.164 (0.944)0.395 (0.407) No Information Market Prediction Best Player Nonlinear Aggregation Function

26. Experimental Economics Program Group Size Performance

27. Experimental Economics Program Did the Markets Pick out Experts? GroupExp 1Exp 2Exp 3Exp 4Exp 5 Random1.360.931.181.121.15 Payoff1.451.091.241.131.39 Value0.720.910.941.131.22 Optimal0.530.720.750.830.77 KL measure of all query data Pick groups of 3

28. Experimental Economics Program Did Previous Queries Pick out Experts? GroupExp 1Exp 2Exp 3Exp 4Exp 5 Random1.150.921.181.071.21 Query0.780.890.710.920.81 Optimal0.600.590.690.72 KL measure of second half of query data Pick groups of 3

29. Experimental Economics Program Public Information Information observed by more than one Double counting problem

30. Information Aggregation with Public Information Kullback-Leibler = 2.591 Public Info Experiment 3, Period 9 11 Players Aggregated

31. Experimental Economics Program Dealing with Public Information: Add a Game to the Second Stage Event 1 Event 2 Event 3 Event 4 Event 5 Event 6 Event 7 Event 8 Stage 1: Information Market Call Market to Solicit Risk Attitudes Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Matching Game to Recover Public Information Modified Nonlinear Aggregated Function Time

32. Experimental Economics Program Assumptions Individuals know their public information Private & Public Info Independent Structure of Public Info Arbitrary

33. Experimental Economics Program Matching Game Reports of Probability Distribution ABC Outcome q 11 q 12 q 13 Player 1: q 1 q 21 q 22 q 23 Player 2: q 2 q 31 q 32 q 33 Player 3: q 3........................ Player 1’s Payoff: (match function)*(C1+C2*Log(q 33 )) Match function: f(q 1,q 2 )=(1-0.5*sum(abs(q 1i -q 2i )))^2 Choose player (3) by Max (match function)

34. Experimental Economics Program Matching Game Any match function f(q 1,q 2 ) with property – Max when q 1 =q 2 Multiple Equilibria Payoff increases as entropy decreases Hopefully, individuals report public information

35. Experimental Economics Program Aggregation Function with Public Information Correction Bayes Law with a) Behavioral Correction b) Public Info Correction i =r(V i / i )c Holding value/Risk - measure relative risk of individuals Normalizing constant for individual risks “market” risk ~sum of prices/winning payoff

36. Experimental Economics Program Public Information Experiments 5 Experiments Various Information Structures – All subject received 2 private draws & 2 public draws – All subject received 3 private draws & 1 public draws – All subject received 3 private draws & half of the subjects receive 1 public draws – All subject received 3 private draws & 1 public draws. 2 groups of independent public information. 9 to 11 participants in each experiments

37. Correcting for Public Information Public Info Experiment 3, Period 9 11 Players Aggregated Kullback-Leibler = 0.291

38. Experimental Economics Program Expt Private Info Public Info No Info Market Prediction Best Player Nonlinear Aggregation Function Public Info Correction Perfect Public Info Correction 12 draws for all 1.332 (0.595) 0.847 (0.312) 0.932 (0.566) 2.095 (1.196) 0.825 (0.549) 0.279 (0.254) 22 draws for all 1.420 (0.424) 0.979 (0.573) 0.919 (0.481) 2.911 (2.776) 0.798 (0.532) 0.258 (0.212) 33 draws for all 1 draws for all 1.668 (0.554) 1.349 (0.348) 1.033 (0.612) 2.531 (1.920) 0.718 (0.817) 0.366 (0.455) 43 draws for all 1 draws for half 1.596 (0.603) 0.851 (0.324) 1.072 (0.604) 0.951 (1.049) 0.798 (0.580) 0.704 (0.691) 5 3 draws for all Two groups of public info 1.528 (0.600) 0.798 (0.451) 1.174 (0.652) 0.886 (0.763) 1.015 (0.751) 0.472 (0.397) KL Measures for Public Info Experiments

39. Experimental Economics Program Summary IAM with public info correction did better than best person. IAM with public info correction did better than markets in 4 out of 5 cases. IAM corrected with true public info did significant better than all other methods.

40. Experimental Economics Program

42. Supplementary

43. Experimental Economics Program Previous Research Academic Studies – Information Aggregation in Markets Plott, Sunder, Camerer, Forsythe, Lundholm, Weber,… – Pari-mutuel Betting Markets Plott, Wit & Yang Real World Applications – Iowa Electronic Markets – Hollywood Stock Exchange – HP Information Markets – Newsfuture – Tradesport.com – …

44. Experimental Economics Program Risk Attitudes

45. Experimental Economics Program Dealing with Risks Attitudes: Two-Stage Mechanism Event 1 Event 2 Event 3 Event 4 Event 5 Event 6 Event 7 Event 8 Stage 1: Information Market Call Market to Solicit Risk Attitudes Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Nonlinear Aggregated Function Time

46. Experimental Economics Program Probability Reporting Reports of Probability Distribution ABC Outcome p1p2p3 Pays C1+C2*Log(p3)

47. Experimental Economics Program Second Stage: Aggregation Function Bayes Law with Behavioral Correction i =r(V i / i )c Holding value/Risk - measure relative risk of individuals Normalizing constant for individual risks “market” risk ~sum of prices/winning payoff

48. Experimental Economics Program Private Information Experiments 5 Experiments Various Information Conditions – All subject received 3 draws – Half received 5 draws, half received 1 draw – Half received 3 draws, half received random number of draws 8 to 13 participants in each experiments

49. Experimental Economics Program Next Step Field Test (Fine and Huberman) …