1. The Wisdom of Crowds in the Aggregation of Rankings Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Michael Lee, Brent Miller, Pernille Hemmer

2. Rank aggregation problem Goal is to combine many different rank orderings on the same set of items in order to obtain a “better” ordering Example applications Combining voters rankings: social choice theory Information retrieval and meta-search* Difficult problem: with N items, there are N! orderings efficient models to explore the space of all permutations 2 *e.g. Lebanon & Mao (2008); Klementiev, Roth et al. (2008; 2009), Dwork et al. (2001)

3. Ulysses S. Grant James Garfield Rutherford B. Hayes Abraham Lincoln Andrew Johnson James Garfield Ulysses S. Grant Rutherford B. Hayes Andrew Johnson Abraham Lincoln Example ranking problem in our research time What is the correct chronological order?

4. Aggregating ranking data 4 D A B C A B D C B A D CA C B D A D B C Aggregation Algorithm A B C D ground truth = ? group answer

5. Generative Approach 5 D A B C A B D C B A D CA C B D A D B C Generative Model ? ? latent truth

6. Wisdom of crowds phenomenon Aggregating over individuals often leads to an estimate that is among the best individual estimates (or sometimes better) 6 Galtons Ox (1907): Median of individual weight estimates came close to true answer

7. Approach No communication between individuals There is always a true answer (ground truth) ground truth only used in evaluation Unsupervised weighting of individuals* exploit relationship between expertise and consensus experts tend to be closer to the truth and therefore reach more similar judgments 7 * Klementiev, Roth et al. (2008, 2009); Dani, Madani, Pennock et al. (2006). Bayesian truth serum (Prelec et al., 2004); Cultural Consensus Theory (Batchelder and Romney, 1986)

8. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes Traveling salesman problems Effect of communication between individuals 8

9. Experiment: 26 individuals order all 44 US presidents 9 George WashingtonJohn AdamsThomas JeffersonJames Madison James MonroeJohn Quincy AdamsAndrew JacksonMartin Van Buren William Henry HarrisonJohn TylerJames Knox PolkZachary Taylor Millard FillmoreFranklin PierceJames BuchananAbraham Lincoln Andrew JohnsonUlysses S. GrantRutherford B. HayesJames Garfield Chester ArthurGrover Cleveland 1Benjamin HarrisonGrover Cleveland 2 William McKinleyTheodore RooseveltWilliam Howard TaftWoodrow Wilson Warren HardingCalvin CoolidgeHerbert HooverFranklin D. Roosevelt Harry S. TrumanDwight EisenhowerJohn F. KennedyLyndon B. Johnson Richard NixonGerald FordJames CarterRonald Reagan George H.W. BushWilliam ClintonGeorge W. BushBarack Obama

10. = 1 = 1+1 Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps Ordering by Individual ABECD True Order ABCDE C D E ABAB AB ECD ABCDEABCDE = 2

11. Empirical Results 11  (random guessing)

12. Classic models: Thurstone (1927) Mallows (1957); Fligner and Verducci, 1986 Diaconis (1989) Voting methods: e.g. Borda count (1770) We will focus on Thurstonian models implemented as graphical models MCMC inference Unsupervised models for ranking data 12 Many models were developed for preference rankings and voting situations  no known ground truth

13. Thurstonian Model 13 A. George Washington B. James Madison C. Andrew Jackson Each item has a true coordinate on some dimension

14. Thurstonian Model 14 … but there is noise because of encoding errors or partial knowledge A. George Washington B. James Madison C. Andrew Jackson

15. Thurstonian Model 15 A. George Washington B. James Madison C. Andrew Jackson Each persons mental encoding is based on a single sample from each distribution A B C

16. Thurstonian Model 16 A. George Washington B. James Madison C. Andrew Jackson A B C A < C < B The observed ordering is based on the ordering of the samples

17. Thurstonian Model 17 A. George Washington B. James Madison C. Andrew Jackson A B C A < B < C The observed ordering is based on the ordering of the samples

18. Thurstonian Model 18 A. George Washington B. James Madison C. Andrew Jackson Key extension is to allow for individual-level differences in accuracy of knowledge The means are fixed, and the variance is fixed for the same person, but differs between people

19. Graphical Model of Extended Thurstonian Model 19 j individuals Latent truth Individual Expertise Mental samples Observed order

20. Inferred Distributions for 44 US Presidents 20 George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44) error bars = median and minimum sigma

21. Wisdom of crowds effect 21 

22. Calibration of individuals 22   inferred noise level for each individual distance to ground truth  individual People who agree with each other are regarded as experts, and are “up-weighted” in forming the aggregate order

23. Heuristic Models Many heuristic methods from voting theory E.g., Borda count method Suppose we have 10 items assign a count of 10 to first item, 9 for second item, etc add counts over individuals order items by the Borda count i.e., rank by average rank across people 23

24. Model Comparison 24  Borda

25. Other ordering tasks 25 Ten Amendments Ten Commandments

26. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes Traveling salesman problems Effect of communication between individuals 26

27. Recollecting Order from Episodic Memory 27 Study this sequence of images

28. How good is your memory? Place the images in the correct sequence (by reading order) 28 A B C D E F G H I J

29. Average results across 6 problems 29 Mean 

30. Calibration of individuals 30 inferred noise level distance to ground truth   individual (pizza sequence; perturbation model)

31. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes Traveling salesman problems Effect of communication between individuals 31

32. Human forecasting experiment Forecast end-of-season rankings for 15 NBA teams Eastern conference Western conference Participants were college undergraduates Varied basketball knowledge 172 individuals for Eastern conference 156 individuals for Western conference Experiment conducted Feb 2010 teams have played about a bit over half of games in regular season 32

33. Model predictions for Eastern conference 33 Borda 1.Boston 2.Cleveland 3.Orlando 4.Miami 5.Detroit 6.Chicago 7.Philadelphia 8.Atlanta 9.New York 10.New Jersey 11.Indiana 12.Washington 13.Toronto 14.Charlotte 15.Milwaukee Actual outcome 1. Cleveland 2. Orlando 3. Atlanta 4. Boston 5. Miami 6. Milwaukee 7. Charlotte 8. Chicago 9. Toronto 10. Indiana 11. New York 12. Detroit 13. Philadelphia 14. Washington 15. New Jersey Thurstonian Model Cleveland Boston Orlando Miami Atlanta Chicago Detroit Charlotte Toronto Philadelphia Washington Indiana New York Milwaukee New Jersey

34. 34 East 73% 93% West 87% 94%  

35. Calibration Results 35 East West   

36. Heuristics: who will win more games? 36 Chicago Bulls Charlotte Bobcats Won 6 championships Team in existence for 44 years vs Won 0 championships Team in existence for 6 years Related to work on “fast and frugal heuristics” by Gigerenzer et al.

37. Heuristic ranking by #championships won 37 #championships 1.Boston 2.Chicago 3.Philadelphia 4.Detroit 5.Indiana 6.New York 7.New Jersey 8.Atlanta 9.Washington 10.Milwaukee 11.Miami 12.Orlando 13.Cleveland 14.Toronto 15.Charlotte Actual outcome 1. Cleveland 2. Orlando 3. Atlanta 4. Boston 5. Miami 6. Milwaukee 7. Charlotte 8. Chicago 9. Toronto 10. Indiana 11. New York 12. Detroit 13. Philadelphia 14. Washington 15. New Jersey

38. Approach Discount people who seem to use overly simplistic heuristics Use an informative prior on expertise A priori, assume low expertise for individuals who produce orderings that can be explained by simple heuristics 38

39. 39 East 96% 73% 93% West 96% 87% 94%  

40. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes Traveling salesman problems Effect of communication between individuals 40

41. Find the shortest route between cities 41 B30-21 Individual 5Individual 83 Individual 60 Optimal

42. Idea: analyze the agreement on edges that are part of the tour 42 Line thickness = agreement

43. Blue = Tour that maximizes agreement 43 Line thickness = agreement

44. Results averaged across 7 problems aggregate

45. Minimum Spanning Trees (MST) Goal: create a network that minimizes the total length of connections 45

46. Wisdom of the Crowd Within Vul and Pashler (2008): repeated responses from the same individual may also result in a wisdom of the crowd effect We performed a “wisdom within” experiment on MST Subjects solved the same MST problem eight times Each repetition involved a rotation/reflection of the original problem

47. Example repetitions of same MST problem (random reflections and rotations applied to the original problem)

48. Solutions from one individual (Subjects were not aware they were solving the same problem)

49. Wisdom of Crowd effect: between vs. within

50. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes Traveling salesman problems Effect of communication between individuals 50

51. Influence of communication Many researchers argue best aggregation is achieved by complete independence between individuals But does sharing of information always lead to worse aggregates? 51

52. Iterated Learning Experiment: each individual refines the previous ordering 52 Abraham Lincoln Andrew Johnson James Garfield Ulysses S. Grant R. B. Hayes Andrew Johnson Abraham Lincoln individual 1 Related to work by Griffiths and colleagues on iterated learning Abraham Lincoln James Garfield Ulysses S. Grant R. B. Hayes Andrew Johnson individual 2 Andrew Johnson James Garfield R. B. Hayes Andrew Johnson Abraham Lincoln individual 3

53. Influence of information sharing Comparing independent judgments and an iterated learning task 53 independent iterated Number of individuals 

54. Conclusions Four psychological ideas individual differences the wisdom within effect of communication prior knowledge Goal is to build a better theory of the wisdom of crowds phenomenon as fundamentally a cognitive modeling problem

55. 55 Do the experiments yourself: http://psiexp.ss.uci.edu/

56. Predicting problem difficulty 56  std  dispersion of expertise distance of inferred truth to actual truth  ordering states geographically city size rankings

57. Summary Combine ordering / ranking data going beyond numerical estimates or multiple choice questions Incorporate individual differences assume some individuals might be “experts” going beyond models that treat every vote equally Incorporate prior knowledge downweight individuals with “wrong” prior knowledge correct judgments towards natural prior orderings 57

58. Effect of Group Size 58 

59. Heuristic Approach Idea: find tours with edges for which many individuals agree Calculate agreement matrix A A = n × n matrix, where n is the number of cities a ij indicates the number of participants that connect cities i and j. use a non-linear transform function f() to emphasize high agreement edges Find tour that maximizes 59 (this itself is a non-Euclidian TSP problem)

60. Forecasting NCAA tournament (March Madness) 64 US college basketball teams are placed in a set of four seeded brackets, and play an elimination tournament. Midwest bracket:

61. Data Predictions from 16,718 Yahoo users Each individual predicts the winner of all games We use the predictions for the first four rounds (60 games total) Two scoring systems Number of correct predictions Points: 1 point per correct winner in 1 st round 2 points in 2 nd 4 points in 3 rd 8 points in 4 rd

62. Data and Results of Heuristic Strategies 62 individuals #correct predictions points Obama 47% majority rule 71% prior seeding 66% prior seeding 61% majority rule 73% Obama 83%

63. Thurstonian Model 63 Team A Team B Team C Each team has a mean on a single “strength” dimension Each person has single variance

64. Thurstonian Model 64 Team A Team B Team C A B B wins over A The probability a person will choose team A over team B is the probability their strength for team A will be sampled above team B

65. Thurstonian Model 65 Team A Team B Team C C B C wins over B The probability a person will choose team A over team B is the probability their strength for team A will be sampled above team B

66. Modeling Results 66 individuals majority rule 71% prior seeding 66% prior seeding 61% majority rule 73% Thurst model 83% Thurstonian model inform. priors 90% Thurst. model 78% Thurst. model inform. priors 81% #correct predictions points

67. Modeling Results 67 individuals majority rule 71% prior seeding 66% prior seeding 61% majority rule 73% Thurstonian model 90% Thurst. model 81% #correct predictions points

68. Rank aggregation applications in this research Combining rankings related to general knowledge, e.g. Orderings by time (e.g. US Presidents) Orderings by length / size (e.g. famous rivers, countries) Combining forecasted team rankings in sports Combining multiple eyewitness testimonies memory for order of events 68