Information Aggregation: Experiments and Industrial Applications Kay-Yut Chen HP Labs

Experimental Economics Program Agenda Lessons from HP Information Markets (Chen and Plott 2002) Scoring Rules and Identification of Experts (Chen, Fine and Huberman 2004) (Chen and Hogg 2004) Public Information (Chen, Fine and Huberman 2004)

Experimental Economics Program HP Information Markets (Chen and Plott) Summary of Events – 12 events, from 1996 to 1999 – 11 events sales related – 8 events had official forecasts Methodology & Procedures – Contingent state asset (i.e. winning ticket pays $1, others $0) – Sales amount (unit/revenue) divided into (8-10) finite intervals – Web-based real time double-auction – min phone training for EVERY subject – Market open for one week at restricted time of the day (typically lunch and after hours) – Market size: people

Experimental Economics Program

Results Abs % Errors of IAM Predictions Last Interval IgnoredLast Interval Mass at Lower Bound EventAbsolute % errors of HP forecasts Average last 60% trade Average last 50% trade Average last 40% trade Average last 60% trade Average last 50% trade Average last 40% trade %4.61%4.57%4.68%5.63%5.68%5.80% %57.48%55.72%54.60%59.25%57.46%56.32% 48.64%7.84%8.15%8.52%6.45%6.77%7.13% %30.93%31.57%31.83%29.74%30.33%30.48% %24.23%24.54%25.30%22.94%23.22%23.93% 74.10%7.33%7.02%6.71%5.35%4.91%4.55% 80.11%2.00%2.35%1.83%1.53%1.39%1.00% %23.85%24.85%24.39%17.55%17.32%16.54% T-test P-value Random variable x=official error – market error H0: mean of x=0 Alternate: mean of x>0

Experimental Economics Program Business Constraints and Research Issues Not allowed to “bet” players’ own money -> stakes limited to an average of $50 per person Time horizon constraints -> 3 months to be useful Recruit the “right” people Asset design affects the results (How to set the intervals?) Thin markets (sum of price ~ $1.11 to $1.31 over the dollar) – Few players – Not enough participation

Experimental Economics Program Reporting with Scoring Rule Reports of Probability Distribution ABC Outcome p1p2p3 Pays C1+C2*Log(p3)

Experimental Economics Program Information Aggregation Function If reports are independent, Bayes Law applies …

Experimental Economics Program Two Complications Non-Risk Neutral Behavior Public Information

Experimental Economics Program Dealing with Risks Attitudes: Two-Stage Mechanism Event 1 Event 2 Event 3 Event 4 Event 5 Event 6 Event 7 Event 8 Stage 1: Information Market Call Market to Solicit Risk Attitudes Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Nonlinear Aggregated Function Time

Experimental Economics Program Second Stage: Aggregation Function Bayes Law with Behavioral Correction i =r(V i / i )c Holding value/Risk - measure relative risk of individuals Normalizing constant for individual risks “market” risk ~sum of prices/winning payoff

Experimental Economics Program Experiments: Inducing Diverse Information ABC Outcome Box of Balls A B C C C * In actual experiments, there are TEN states Random Draws Provide Info

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 No Information

Experimental Economics Program Kullback-Leibler Measure Relative entropy Always >=0 =0 if two distributions are identical Addictive for independent events

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 1 Player

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 2 Players Aggregated

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 3 Players Aggregated

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 4 Players Aggregated

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 5 Players Aggregated

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 6 Players Aggregated

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 7 Players Aggregated

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 8 Players Aggregated

Kullback-Leibler = Comparison To All Information Probability Experiment 4, Period 17 9 Players Aggregated

Comparison To All Information Probability Experiment 4, Period 17

Experimental Economics Program KL Measures for Private Info Experiments (0.312)1.222 (0.650)0.844 (0.599)0.553 (1.057) (0.618)1.112 (0.594)1.128 (0.389)0.214 (0.195) (0.576)1.053 (1.083)0.876 (0.646)0.414 (0.404) (0.570)1.136 (0.193)1.074 (0.462)0.413 (0.260) (0.598)1.371 (0.661)1.164 (0.944)0.395 (0.407) No Information Market Prediction Best Player Nonlinear Aggregation Function

Experimental Economics Program Group Size Performance

Experimental Economics Program Did the Markets Pick out Experts? GroupExp 1Exp 2Exp 3Exp 4Exp 5 Random Payoff Value Optimal KL measure of all query data Pick groups of 3

Experimental Economics Program Did Previous Queries Pick out Experts? GroupExp 1Exp 2Exp 3Exp 4Exp 5 Random Query Optimal KL measure of second half of query data Pick groups of 3

Experimental Economics Program Public Information Information observed by more than one Double counting problem

Information Aggregation with Public Information Kullback-Leibler = Public Info Experiment 3, Period 9 11 Players Aggregated

Experimental Economics Program Dealing with Public Information: Add a Game to the Second Stage Event 1 Event 2 Event 3 Event 4 Event 5 Event 6 Event 7 Event 8 Stage 1: Information Market Call Market to Solicit Risk Attitudes Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Matching Game to Recover Public Information Modified Nonlinear Aggregated Function Time

Experimental Economics Program Assumptions Individuals know their public information Private & Public Info Independent Structure of Public Info Arbitrary

Experimental Economics Program Matching Game Reports of Probability Distribution ABC Outcome q 11 q 12 q 13 Player 1: q 1 q 21 q 22 q 23 Player 2: q 2 q 31 q 32 q 33 Player 3: q Player 1’s Payoff: (match function)*(C1+C2*Log(q 33 )) Match function: f(q 1,q 2 )=(1-0.5*sum(abs(q 1i -q 2i )))^2 Choose player (3) by Max (match function)

Experimental Economics Program Matching Game Any match function f(q 1,q 2 ) with property – Max when q 1 =q 2 Multiple Equilibria Payoff increases as entropy decreases Hopefully, individuals report public information

Experimental Economics Program Aggregation Function with Public Information Correction Bayes Law with a) Behavioral Correction b) Public Info Correction i =r(V i / i )c Holding value/Risk - measure relative risk of individuals Normalizing constant for individual risks “market” risk ~sum of prices/winning payoff

Experimental Economics Program Public Information Experiments 5 Experiments Various Information Structures – All subject received 2 private draws & 2 public draws – All subject received 3 private draws & 1 public draws – All subject received 3 private draws & half of the subjects receive 1 public draws – All subject received 3 private draws & 1 public draws. 2 groups of independent public information. 9 to 11 participants in each experiments

Correcting for Public Information Public Info Experiment 3, Period 9 11 Players Aggregated Kullback-Leibler = 0.291

Experimental Economics Program Expt Private Info Public Info No Info Market Prediction Best Player Nonlinear Aggregation Function Public Info Correction Perfect Public Info Correction 12 draws for all (0.595) (0.312) (0.566) (1.196) (0.549) (0.254) 22 draws for all (0.424) (0.573) (0.481) (2.776) (0.532) (0.212) 33 draws for all 1 draws for all (0.554) (0.348) (0.612) (1.920) (0.817) (0.455) 43 draws for all 1 draws for half (0.603) (0.324) (0.604) (1.049) (0.580) (0.691) 5 3 draws for all Two groups of public info (0.600) (0.451) (0.652) (0.763) (0.751) (0.397) KL Measures for Public Info Experiments

Experimental Economics Program Summary IAM with public info correction did better than best person. IAM with public info correction did better than markets in 4 out of 5 cases. IAM corrected with true public info did significant better than all other methods.

Experimental Economics Program

Supplementary

Experimental Economics Program Previous Research Academic Studies – Information Aggregation in Markets Plott, Sunder, Camerer, Forsythe, Lundholm, Weber,… – Pari-mutuel Betting Markets Plott, Wit & Yang Real World Applications – Iowa Electronic Markets – Hollywood Stock Exchange – HP Information Markets – Newsfuture – Tradesport.com – …

Experimental Economics Program Risk Attitudes

Experimental Economics Program Dealing with Risks Attitudes: Two-Stage Mechanism Event 1 Event 2 Event 3 Event 4 Event 5 Event 6 Event 7 Event 8 Stage 1: Information Market Call Market to Solicit Risk Attitudes Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Nonlinear Aggregated Function Time

Experimental Economics Program Probability Reporting Reports of Probability Distribution ABC Outcome p1p2p3 Pays C1+C2*Log(p3)

Experimental Economics Program Second Stage: Aggregation Function Bayes Law with Behavioral Correction i =r(V i / i )c Holding value/Risk - measure relative risk of individuals Normalizing constant for individual risks “market” risk ~sum of prices/winning payoff

Experimental Economics Program Private Information Experiments 5 Experiments Various Information Conditions – All subject received 3 draws – Half received 5 draws, half received 1 draw – Half received 3 draws, half received random number of draws 8 to 13 participants in each experiments

Experimental Economics Program Next Step Field Test (Fine and Huberman) …