1. The Wisdom of Crowds in the Aggregation of Rankings Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Michael Lee, Brent Miller, Pernille Hemmer

2. Wisdom of crowds phenomenon Aggregating over individuals often leads to an estimate that is among the best individual estimates (or sometimes better) Vul & Pashler (2008) averaging over repeated estimates from the same individual also results in a wisdom of the crowd effect 2 Galtons Ox (1907): Median of individual weight estimates came close to true answer

3. Rank aggregation problem Goal is to combine many different rank orderings on the same set of items in order to obtain a “better” ordering Example applications Combining voters rankings: social choice theory Aggregating search engine rankings* Difficult problem: with N items, there are N! orderings efficient models to explore the space of all permutations 3 *e.g. Lebanon & Mao (2008); Klementiev, Roth et al. (2008; 2009), Dwork et al. (2001)

4. Ulysses S. Grant James Garfield Rutherford B. Hayes Abraham Lincoln Andrew Johnson James Garfield Ulysses S. Grant Rutherford B. Hayes Andrew Johnson Abraham Lincoln Example ranking problem in our research time What is the correct chronological order? Alice Healy and Roddy Roediger performed similar experiments to study serial recall with preexisting knowledge

5. Aggregating ranking data 5 D A B C A B D C B A D CA C B D A D B C Aggregation Algorithm A B C D ground truth = ? group answer

6. Generative Approach 6 D A B C A B D C B A D CA C B D A D B C Generative Model ? ? latent truth

7. General Approach No communication between individuals There is always a true answer (ground truth) ground truth only used in evaluation Unsupervised weighting of individuals* exploit relationship between expertise and consensus experts tend to be closer to the truth and therefore produce more similar judgments 7 * Klementiev, Roth et al. (2008, 2009); Dani, Madani, Pennock et al. (2006). Bayesian truth serum (Prelec et al., 2004); Cultural Consensus Theory (Batchelder and Romney, 1986)

8. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes “Wisdom Within” Effect of communication between individuals 8

9. Experiment: 26 individuals order all 44 US presidents 9 George WashingtonJohn AdamsThomas JeffersonJames Madison James MonroeJohn Quincy AdamsAndrew JacksonMartin Van Buren William Henry HarrisonJohn TylerJames Knox PolkZachary Taylor Millard FillmoreFranklin PierceJames BuchananAbraham Lincoln Andrew JohnsonUlysses S. GrantRutherford B. HayesJames Garfield Chester ArthurGrover Cleveland 1Benjamin HarrisonGrover Cleveland 2 William McKinleyTheodore RooseveltWilliam Howard TaftWoodrow Wilson Warren HardingCalvin CoolidgeHerbert HooverFranklin D. Roosevelt Harry S. TrumanDwight EisenhowerJohn F. KennedyLyndon B. Johnson Richard NixonGerald FordJames CarterRonald Reagan George H.W. BushWilliam ClintonGeorge W. BushBarack Obama

10. = 1 = 1+1 Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps Ordering by Individual ABECD True Order ABCDE C D E ABAB AB ECD ABCDEABCDE = 2

11. Empirical Results 11  (random guessing)

12. Statistical models: Mallows (1957); Fligner and Verducci (1986) Voting theoretic methods e.g. Borda count (1770) Psychological models Thurstone (1927) Perturbation model (Estes) We will focus on Thurstonian models implemented as graphical models MCMC inference Models for ranking data 12

13. Thurstonian Model 13 A. George Washington B. James Madison C. Andrew Jackson Each item has a true coordinate on some dimension

14. Thurstonian Model 14 … but there is noise because of encoding errors or partial knowledge A. George Washington B. James Madison C. Andrew Jackson

15. Thurstonian Model 15 A. George Washington B. James Madison C. Andrew Jackson Each persons mental encoding is based on a single sample from each distribution A B C

16. Thurstonian Model 16 A. George Washington B. James Madison C. Andrew Jackson A B C A < C < B The observed ordering is based on the ordering of the samples

17. Thurstonian Model 17 A. George Washington B. James Madison C. Andrew Jackson A B C A < B < C The observed ordering is based on the ordering of the samples

18. Thurstonian Model 18 A. George Washington B. James Madison C. Andrew Jackson Key extension is to allow for individual-level differences in accuracy of knowledge The means are fixed, and the variance is fixed for the same person, but differs between people

19. Graphical Model of Extended Thurstonian Model 19 j individuals Latent truth Individual Expertise Mental samples Observed order

20. Inferred Distributions for 44 US Presidents 20 George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44) error bars = median and minimum sigma

21. Wisdom of crowds effect 21 

22.  Calibration of individuals 22  inferred noise level for each individual distance to ground truth  individual People who agree with each other are regarded as experts, and are “up-weighted” in forming the aggregate order

23. Heuristic Models Many heuristic methods from voting theory E.g., Borda count method Suppose we have 10 items assign a count of 10 to first item, 9 for second item, etc add counts over individuals order items by the Borda count i.e., rank by average rank across people 23

24. Model Comparison 24  Borda

25. Other ordering tasks 25 Ten Amendments Ten Commandments

26. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes “Wisdom Within” Effect of communication between individuals 26

27. Recollecting Order from Episodic Memory 27 Study this sequence of images

28. How good is your memory? Place the images in the correct sequence (by reading order) 28 A B C D E F G H I J

29. Average results across 6 problems 29 Mean 

30. Calibration of individuals 30 inferred noise level distance to ground truth   individual (pizza sequence; perturbation model)

31. Problem What if we have only a small number of individuals? How can we guard against individuals with poor memory? Idea: use prior knowledge incorporate prior knowledge in aggregation process 31

32. Measuring Prior Knowledge for Event Sequences 32 A B C D E F G H I J “Place the images in a sequence that looks natural to you”

33. Modeling Approach 33 Prior Studied Event Sequence (latent) Memory Reconstructions from K individuals Prior Orderings from N individuals ?

34. Results when picking K worst “witnesses” 34 Number of “witnesses” (K)  without prior knowledge with prior knowledge

35. Results when randomly selecting individuals 35 Group size  without prior knowledge with prior knowledge

36. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes “Wisdom Within” Effect of communication between individuals 36

37. Human forecasting experiment Forecast end-of-season rankings for 15 NBA teams Eastern conference Western conference Participants were college undergraduates Varied basketball knowledge 172 individuals for Eastern conference 156 individuals for Western conference Experiment conducted Feb 2010 teams have played about a bit over half of games in regular season 37

38. 38 Borda 1.Boston 2.Cleveland 3.Orlando 4.Miami 5.Detroit 6.Chicago 7.Philadelphia 8.Atlanta 9.New York 10.New Jersey 11.Indiana 12.Washington 13.Toronto 14.Charlotte 15.Milwaukee Actual outcome 1. Cleveland 2. Orlando 3. Atlanta 4. Boston 5. Miami 6. Milwaukee 7. Charlotte 8. Chicago 9. Toronto 10. Indiana 11. New York 12. Detroit 13. Philadelphia 14. Washington 15. New Jersey Thurstonian Model Cleveland Boston Orlando Miami Atlanta Chicago Detroit Charlotte Toronto Philadelphia Washington Indiana New York Milwaukee New Jersey

39. 39 East 73% 93% West 87% 94%  

40. Calibration Results 40 East West   

41. Problem Current model identifies expertise by consensus Non-experts might use simple heuristics that also lead to agreement 41

42. Heuristics: who will win more games? 42 Chicago Bulls Charlotte Bobcats Won 6 championships Team in existence for 44 years vs Won 0 championships Team in existence for 6 years Related to work on “fast and frugal heuristics” by Gigerenzer et al.

43. Heuristic ranking by #championships won 43 #championships 1.Boston 2.Chicago 3.Philadelphia 4.Detroit 5.Indiana 6.New York 7.New Jersey 8.Atlanta 9.Washington 10.Milwaukee 11.Miami 12.Orlando 13.Cleveland 14.Toronto 15.Charlotte Actual outcome 1. Cleveland 2. Orlando 3. Atlanta 4. Boston 5. Miami 6. Milwaukee 7. Charlotte 8. Chicago 9. Toronto 10. Indiana 11. New York 12. Detroit 13. Philadelphia 14. Washington 15. New Jersey

44. Approach We need to find consensus solutions where people agre with each other but do not seem to use (simplistic) heuristics 44

45. Mixture Model Incorporating Heuristics 45 D A B C A B D C B A D CA C B D A D B C Mixture Model heuristic 1 heuristic 2 belief 1 e.g. Number of years team exists e.g. Number of championships won belief 2 …

46. 46 East 97% 73% 93% West 97% 87% 94%  

47. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes “Wisdom Within” Effect of communication between individuals 47

48. Minimum Spanning Trees (MST) Goal: create a network that minimizes the total length of connections 48

49. Wisdom of the Crowd Within Vul and Pashler (2008): repeated responses from the same individual may also result in a wisdom of the crowd effect We performed a “wisdom within” experiment on MST Subjects solved the same MST problem eight times Each repetition involved a rotation/reflection of the original problem

50. Example repetitions of same MST problem (random reflections and rotations applied to the original problem)

51. Solutions from one individual (Subjects were not aware they were solving the same problem)

52. Comparing Between vs. Within Individual Aggregation

53. Overview of talk General knowledge tasks reconstructing order of US presidents Episodic memory reconstructing order of personally experienced events Forecasting NBA outcomes Traveling salesman problems Effect of communication between individuals 53

54. Influence of communication Many researchers argue best aggregation is achieved by complete independence between individuals But does sharing of information always lead to worse aggregates? 54

55. Iterated Learning Experiment: each individual refines the previous ordering 55 Abraham Lincoln Andrew Johnson James Garfield Ulysses S. Grant R. B. Hayes Andrew Johnson Abraham Lincoln individual 1 Related to work by Griffiths and colleagues on iterated learning Abraham Lincoln James Garfield Ulysses S. Grant R. B. Hayes Andrew Johnson individual 2 Andrew Johnson James Garfield R. B. Hayes Andrew Johnson Abraham Lincoln individual 3

56. Influence of information sharing Comparing independent judgments and an iterated learning task 56 independent iterated Number of individuals 

57. Conclusions Four psychological ideas individual differences incorporating heuristics incorporating prior knowledge effect of communication Goal is to build a better theory of the wisdom of crowds phenomenon fundamentally a cognitive modeling problem

58. 58 Do the experiments yourself: http://psiexp.ss.uci.edu/

59. Predicting problem difficulty 59  std  dispersion of expertise distance of inferred truth to actual truth  ordering states geographically city size rankings

60. Summary Combine ordering / ranking data going beyond numerical estimates or multiple choice questions Incorporate individual differences assume some individuals might be “experts” going beyond models that treat every vote equally Incorporate prior knowledge downweight individuals with “wrong” prior knowledge correct judgments towards natural prior orderings 60

61. Effect of Group Size 61 

62. Heuristic Approach Idea: find tours with edges for which many individuals agree Calculate agreement matrix A A = n × n matrix, where n is the number of cities a ij indicates the number of participants that connect cities i and j. use a non-linear transform function f() to emphasize high agreement edges Find tour that maximizes 62 (this itself is a non-Euclidian TSP problem)

63. Forecasting NCAA tournament (March Madness) 64 US college basketball teams are placed in a set of four seeded brackets, and play an elimination tournament. Midwest bracket:

64. Data Predictions from 16,718 Yahoo users Each individual predicts the winner of all games We use the predictions for the first four rounds (60 games total) Two scoring systems Number of correct predictions Points: 1 point per correct winner in 1 st round 2 points in 2 nd 4 points in 3 rd 8 points in 4 rd

65. Data and Results of Heuristic Strategies 65 individuals #correct predictions points Obama 47% majority rule 71% prior seeding 66% prior seeding 61% majority rule 73% Obama 83%

66. Thurstonian Model 66 Team A Team B Team C Each team has a mean on a single “strength” dimension Each person has single variance

67. Thurstonian Model 67 Team A Team B Team C A B B wins over A The probability a person will choose team A over team B is the probability their strength for team A will be sampled above team B

68. Thurstonian Model 68 Team A Team B Team C C B C wins over B The probability a person will choose team A over team B is the probability their strength for team A will be sampled above team B

69. Modeling Results 69 individuals majority rule 71% prior seeding 66% prior seeding 61% majority rule 73% Thurst model 83% Thurstonian model inform. priors 90% Thurst. model 78% Thurst. model inform. priors 81% #correct predictions points

70. Modeling Results 70 individuals majority rule 71% prior seeding 66% prior seeding 61% majority rule 73% Thurstonian model 90% Thurst. model 81% #correct predictions points

71. Rank aggregation applications in this research Combining rankings related to general knowledge, e.g. Orderings by time (e.g. US Presidents) Orderings by length / size (e.g. famous rivers, countries) Combining forecasted team rankings in sports Combining multiple eyewitness testimonies memory for order of events 71

72. Find the shortest route between cities 72 B30-21 Individual 5Individual 83 Individual 60 Optimal

73. Idea: analyze the agreement on edges that are part of the tour 73 Line thickness = agreement

74. Blue = Tour that maximizes agreement 74 Line thickness = agreement

75. Results averaged across 7 problems aggregate