The Wisdom of Crowds in the Aggregation of Rankings Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Michael Lee, Brent Miller, Pernille Hemmer

Rank aggregation problem Goal is to combine many different rank orderings on the same set of items in order to obtain a “better” ordering Example applications Combining voters rankings: social choice theory Information retrieval and meta-search* 2 *e.g. Lebanon & Mao (2008); Klementiev, Roth et al. (2008; 2009), Dwork et al. (2001)

Ulysses S. Grant James Garfield Rutherford B. Hayes Abraham Lincoln Andrew Johnson James Garfield Ulysses S. Grant Rutherford B. Hayes Andrew Johnson Abraham Lincoln Example ranking problem in our research time What is the correct chronological order?

Aggregating ranking data 4 D A B C A B D C B A D CA C B D A D B C Aggregation Algorithm A B C D ground truth = ? group answer

Generative Approach 5 D A B C A B D C B A D CA C B D A D B C Generative Model ? ? latent truth

Wisdom of crowds phenomenon Aggregating over individuals often leads to an estimate that is among the best individual estimates (or sometimes better) 6 Galtons Ox (1907): Median of individual weight estimates came close to true answer

Approach No communication between individuals There is always a true answer (ground truth) ground truth only used in evaluation Unsupervised weighting of individuals* exploit relationship between expertise and consensus experts tend to be closer to the truth and therefore reach more similar judgments Incorporate prior knowledge about latent truth discount a priori bad rankings 7 * Klementiev, Roth et al. (2008, 2009); Dani, Madani, Pennock et al. (2006). Bayesian truth serum (Prelec et al., 2004); Cultural Consensus Theory (Batchelder and Romney, 1986)

Overview of talk General knowledge tasks reconstructing order of US presidents Thurstonian models Sports prediction forecasting NBA and NCAA outcomes Thurstonian models Episodic memory reconstructing order of personally experienced events Mallows model 8

Experiment: 26 individuals order all 44 US presidents 9 George WashingtonJohn AdamsThomas JeffersonJames Madison James MonroeJohn Quincy AdamsAndrew JacksonMartin Van Buren William Henry HarrisonJohn TylerJames Knox PolkZachary Taylor Millard FillmoreFranklin PierceJames BuchananAbraham Lincoln Andrew JohnsonUlysses S. GrantRutherford B. HayesJames Garfield Chester ArthurGrover Cleveland 1Benjamin HarrisonGrover Cleveland 2 William McKinleyTheodore RooseveltWilliam Howard TaftWoodrow Wilson Warren HardingCalvin CoolidgeHerbert HooverFranklin D. Roosevelt Harry S. TrumanDwight EisenhowerJohn F. KennedyLyndon B. Johnson Richard NixonGerald FordJames CarterRonald Reagan George H.W. BushWilliam ClintonGeorge W. BushBarack Obama

= 1 = 1+1 Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps Ordering by Individual ABECD True Order ABCDE C D E ABAB AB ECD ABCDEABCDE = 2

Empirical Results 11  (random guessing)

Classic models: Thurstone (1927) Mallows (1957); Fligner and Verducci, 1986 Diaconis (1989) Voting methods: e.g. Borda count (1770) We will focus on Thurstonian and Mallows models implemented as graphical models MCMC inference Unsupervised models for ranking data 12 Many models were developed for preference rankings and voting situations  no known ground truth

Thurstonian Model 13 A. George Washington B. James Madison C. Andrew Jackson Each item has a true coordinate on some dimension

Thurstonian Model 14 … but there is noise because of encoding errors A. George Washington B. James Madison C. Andrew Jackson

Thurstonian Model 15 A. George Washington B. James Madison C. Andrew Jackson Each persons mental encoding is based on a single sample from each distribution A B C

Thurstonian Model 16 A. George Washington B. James Madison C. Andrew Jackson A B C A < C < B The observed ordering is based on the ordering of the samples

Thurstonian Model 17 A. George Washington B. James Madison C. Andrew Jackson A B C A < B < C The observed ordering is based on the ordering of the samples

Thurstonian Model 18 A. George Washington B. James Madison C. Andrew Jackson Important assumption: across individuals, variance can vary but not the means

Graphical Model of Extended Thurstonian Model 19 j individuals Latent truth Expertise of individual Mental samples Observed ordering

Inferred Distributions for 44 US Presidents 20 George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44) error bars = median and minimum sigma

Calibration of individuals 21   inferred noise level for each individual distance to ground truth  individual

Wisdom of crowds effect 22 

Heuristic Models Many heuristic methods from voting theory E.g., Borda count method Suppose we have 10 items assign a count of 10 to first item, 9 for second item, etc add counts over individuals order items by the Borda count i.e., rank by average rank across people 23

Model Comparison 24  Borda

Other ordering tasks 25 Ten Amendments Ten Commandments

Overview of talk General knowledge tasks reconstructing order of US presidents Sports prediction forecasting NBA and NCAA outcomes Episodic memory reconstructing order of personally experienced events New directions 26

Human forecasting experiment Forecast end-of-season rankings for 15 NBA teams Eastern conference Western conference Participants were college undergraduates heterogeneous population regarding basketball expertise 172 individuals for Eastern conference 156 individuals for Western conference Experiment conducted Feb 2010 teams have played about a bit over half of games in regular season 27

Model predictions for Eastern conference 28 Borda 1.Boston 2.Cleveland 3.Orlando 4.Miami 5.Detroit 6.Chicago 7.Philadelphia 8.Atlanta 9.New York 10.New Jersey 11.Indiana 12.Washington 13.Toronto 14.Charlotte 15.Milwaukee Actual outcome 1. Cleveland 2. Orlando 3. Atlanta 4. Boston 5. Miami 6. Milwaukee 7. Charlotte 8. Chicago 9. Toronto 10. Indiana 11. New York 12. Detroit 13. Philadelphia 14. Washington 15. New Jersey Thurstonian Model Cleveland Boston Orlando Miami Atlanta Chicago Detroit Charlotte Toronto Philadelphia Washington Indiana New York Milwaukee New Jersey

29 East 73% 93% West 87% 94%  

Calibration Results 30 East West   

Heuristics: who will win more games? 31 Chicago Bulls Charlotte Bobcats Won 6 championships Team in existence for 44 years vs Won 0 championships Team in existence for 6 years Related to work on “fast and frugal heuristics” by Gigerenzer et al.

Heuristic ranking by #championships won 32 #championships 1.Boston 2.Chicago 3.Philadelphia 4.Detroit 5.Indiana 6.New York 7.New Jersey 8.Atlanta 9.Washington 10.Milwaukee 11.Miami 12.Orlando 13.Cleveland 14.Toronto 15.Charlotte Actual outcome 1. Cleveland 2. Orlando 3. Atlanta 4. Boston 5. Miami 6. Milwaukee 7. Charlotte 8. Chicago 9. Toronto 10. Indiana 11. New York 12. Detroit 13. Philadelphia 14. Washington 15. New Jersey

Informative Priors on Expertise Individuals who closely follow heuristic orderings are probably not experts Set hyperparameters of variance prior based on distance to heuristic ordering 33  prior for individual who closely follows heuristic ordering

Graphical Model 34 j individuals

35 East 96% 73% 93% West 96% 87% 94%  

Forecasting NCAA tournament (March Madness) 64 US college basketball teams are placed in a set of four seeded brackets, and play an elimination tournament. Midwest bracket:

Data Predictions from 16,718 Yahoo users Each individual predicts the winner of all games We use the predictions for the first four rounds (60 games total) Two scoring systems Number of correct predictions Points: 1 point per correct winner in 1 st round 2 points in 2 nd 4 points in 3 rd 8 points in 4 rd

Data and Results of Heuristic Strategies 38 individuals #correct predictions points Obama 47% majority rule 71% prior seeding 66% prior seeding 61% majority rule 73% Obama 83%

Thurstonian Model 39 Team A Team B Team C Each team has a mean on a single “strength” dimension Each person has single variance

Thurstonian Model 40 Team A Team B Team C A B B wins over A The probability a person will choose team A over team B is the probability their strength for team A will be sampled above team B

Thurstonian Model 41 Team A Team B Team C C B C wins over B The probability a person will choose team A over team B is the probability their strength for team A will be sampled above team B

Modeling Results 42 individuals majority rule 71% prior seeding 66% prior seeding 61% majority rule 73% Thurst model 83% Thurstonian model inform. priors 90% Thurst. model 78% Thurst. model inform. priors 81% #correct predictions points

Overview of talk General knowledge tasks reconstructing order of US presidents Sports prediction forecasting NBA and NCAA outcomes Episodic memory reconstructing order of personally experienced events 43

Recollecting Order from Episodic Memory 44 Study this sequence of images

How good is your memory? Place the images in the correct sequence (by reading order) 45 A B C D E F G H I J

Problem What if we have only a small number of individuals? How can we guard against individuals with poor memory? Idea: “smooth” the inferred group ordering with a prior 46

Approach Empirically measure the prior orderings over events Experiment: a separate group of individuals orders the images without seeing original video Use this data to construct a prior on the group ordering 47

ω yjyj θjθj Mallows Model (memory data) latent truth expertise for person j observed ranking for person j Kendall tau distance

ω yjyj θjθj θ*θ* ωoωo yojyoj θojθoj Mallows Model with an informative prior on the latent truth (prior knowledge data) (memory data) latent truth expertise for person j observed ranking for person j prior on orderings

Results when picking K worst “witnesses” 50 Number of “witnesses” (K)  uniform prior informative prior

Summary Combine ordering / ranking data going beyond numerical estimates or multiple choice questions Incorporate individual differences assume some individuals might be “experts” going beyond models that treat every vote equally Incorporate prior knowledge downweight individuals with “wrong” prior knowledge correct judgments towards natural prior orderings 51

Influence of communication Many researchers argue best aggregation is achieved by complete independence between individuals But does sharing of information always lead to worse aggregates? 52

Iterated Learning Experiment: each individual refines the previous ordering 53 Abraham Lincoln Andrew Johnson James Garfield Ulysses S. Grant R. B. Hayes Andrew Johnson Abraham Lincoln individual 1 Related to work by Griffiths and colleagues on iterated learning Abraham Lincoln James Garfield Ulysses S. Grant R. B. Hayes Andrew Johnson individual 2 Andrew Johnson James Garfield R. B. Hayes Andrew Johnson Abraham Lincoln individual 3

Influence of information sharing Comparing independent judgments and an iterated learning task 54 independent iterated Number of individuals 

55 Do the experiments yourself:

Predicting problem difficulty 56  std  dispersion of expertise distance of inferred truth to actual truth  ordering states geographically city size rankings

Effect of Group Size 57 

Notes Bradley Terry model is another model for paired comparisons 58

To do look hyperparameter parametrization for Matlab and other languages What are natural priors for standard deviation? inverse gamma? Look up Babington model in Marden (1997) Look up un. of new mexico lady Look up recent research by Pennock and Klementiev Look up Hal Stern 59

Average results across 6 problems 60 Mean 

Find the shortest route between cities 61 B30-21 Individual 5Individual 83 Individual 60 Optimal

Dataset Vickers, Bovet, Lee, & Hughes (2003) 83 participants 7 problems of 30 cities

TSP Aggregation Problem Data consists of city order only No access to city locations 63

Heuristic Approach Idea: find tours with edges for which many individuals agree Calculate agreement matrix A A = n × n matrix, where n is the number of cities a ij indicates the number of participants that connect cities i and j. use a non-linear transform function f() to emphasize high agreement edges Find tour that maximizes 64 (this itself is a non-Euclidian TSP problem)

Line thickness = agreement 65

Blue = Aggregate Tour 66

Results averaged across 7 problems aggregate

Calibration of individuals 68 inferred noise level distance to ground truth   individual (pizza sequence; perturbation model)

Average results over 17 Problems 69 Individuals Mean  Strong wisdom of crowds effect across problems

Results when randomly selecting individuals 70 Group size  uniform prior informative prior

Experiment 2 78 participants 17 problems each with 10 items Chronological Events Physical Measures Purely ordinal problems, e.g. Ten Amendments Ten commandments 71

Ordering states west-east 72 Oregon (1) Utah (2) Nebraska (3) Iowa (4) Alabama (6) Ohio (5) Virginia (7) Delaware (8) Connecticut (9) Maine (10)