1. Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Brent Miller, Pernille Hemmer, Mike Yi Michael Lee, Bill Batchelder, Paolo Napoletano

2. Ulysses S. Grant James Garfield Rutherford B. Hayes Abraham Lincoln Andrew Johnson James Garfield Ulysses S. Grant Rutherford B. Hayes Andrew Johnson Abraham Lincoln What is the correct chronological order? time

3. Research goal: aggregating responses 3 D A B C A B D C B A D CA C B D A D B C Aggregation Algorithm A B C D ground truth = ? group answer

4. Task constraints No communication between individuals There is always a true answer (ground truth) Aggregation algorithm is unsupervised ground truth only used for evaluation 4

5. Wisdom of crowds phenomenon Group estimate often performs as well as or better than best individual in the group 5

6. Examples of wisdom of crowds phenomenon 6 Who wants to be a millionaire? Galton’s Ox (1907): Median of individual estimates comes close to true answer

7. Relation to Cultural Consensus Theory (CCT) Developed by Batchelder and Romney CCT can recover the answer key of a multiple choice test by analyzing responses across individuals Key assumption: questions vary in difficulty and individuals vary in ability Our models will be similar to the ideas of CCT, but the emphasis is different Each problem studied has a ground truth We focus on “wisdom of crowds” phenomenon 7

8. Overview of talk Ordering problems – general knowledge what is the order of US presidents? Ordering problems – episodic memory what is the order of events you have experienced? Matching problems memory for pairs: what object was paired with what person? Recognition memory problems what words were studied? 8

9. Experiment: 26 individuals order all 44 US presidents 9 George WashingtonJohn AdamsThomas JeffersonJames Madison James MonroeJohn Quincy AdamsAndrew JacksonMartin Van Buren William Henry HarrisonJohn TylerJames Knox PolkZachary Taylor Millard FillmoreFranklin PierceJames BuchananAbraham Lincoln Andrew JohnsonUlysses S. GrantRutherford B. HayesJames Garfield Chester ArthurGrover Cleveland 1Benjamin HarrisonGrover Cleveland 2 William McKinleyTheodore RooseveltWilliam Howard TaftWoodrow Wilson Warren HardingCalvin CoolidgeHerbert HooverFranklin D. Roosevelt Harry S. TrumanDwight EisenhowerJohn F. KennedyLyndon B. Johnson Richard NixonGerald FordJames CarterRonald Reagan George H.W. BushWilliam ClintonGeorge W. BushBarack Obama

10. = 1 = 1+1 Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps Ordering by Individual ABECD True Order ABCDE C D E ABAB AB ECD ABCDEABCDE = 2

11. Empirical Results 11  (random guessing)

12. A Bayesian (generative) approach 12 D A B C A B D C B A D CA C B D ? ? latent “input” Model …

13. Bayesian models We extend two models: Thurstone’s (1927) model Estes (1972) perturbation model 13

14. Bayesian Thurstonian Approach 14 Each item has a true coordinate on some dimension A B C

15. Bayesian Thurstonian Approach 15 A B C … but there is noise because of encoding and/or retrieval error Person 1

16. Bayesian Thurstonian Approach 16 Each person’s mental representation is based on (latent) samples of these distributions B C A B C Person 1 A

17. Bayesian Thurstonian Approach 17 B C A B C The observed ordering is based on the ordering of the samples A < B < C Observed Ordering: Person 1 A

18. Bayesian Thurstonian Approach 18 People draw from distributions with common means but different variances Person 1 B C A B C A < B < C Observed Ordering: Person 2 A B C B C Observed Ordering: A < C < B A A

19. Graphical Model Notation 19 j=1..3 shaded = observed not shaded = latent

20. Graphical Model of Bayesian Thurstonian Model 20 j individuals Latent group means Individual noise level Mental representation Observed ordering

21. Inference Need the posterior distribution Markov Chain Monte Carlo Gibbs sampling on Metropolis-hastings on and 21

22. Inferred Distributions for 44 US Presidents 22 George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44) median and minimum sigma

23. Model can predict individual performance 23   inferred noise level for each individual distance to ground truth  individual

24. (Weak) Wisdom of Crowds Effect 24  model’s ordering is as good as best individual (but not better)

25. Extension of Estes (1972) Perturbation Model Main idea: item order is perturbed locally Our extension: perturbation noise varies between individuals and items 25 A True order BCDE Recalled order DB C E A

26. Modified Perturbation Model 26

27. Inferred Perturbation Matrix and Item Accuracy 27 Abraham Lincoln Richard Nixon James Carter

28. Strong wisdom of crowds effect 28  Perturbation model’s ordering is better than best individual Perturbation

29. Alternative Heuristic Models Many heuristic methods from voting theory E.g., Borda count method Suppose we have 10 items assign a count of 10 to first item, 9 for second item, etc add counts over individuals order items by the Borda count i.e., rank by average rank across people 29

30. Model Comparison 30  Borda

31. Ordering Ten Amendments 31 Freedom of speech & religion (1) Right to bear arms (2) No quartering of soldiers (4) No unreasonable searches (3) Due process (5) Trial by Jury (6) Civil Trial by Jury (7) No cruel punishment (8) Right to non-specified rights (10) Power for the States & People (9)

32. Ordering Ten Commandments 32

33. Overview of talk Ordering problems – general knowledge what is the order of US presidents? Ordering problems – episodic memory what is the order of events you have experienced? Matching problems memory for pairs: what object was paired with what person? Recognition memory problems what words were studied? 33

34. Recollecting order from episodic memory 34 http://www.youtube.com/watch?v=a6tSyDHXViM&feature=related

35. Place scenes in correct order (serial recall) 35 time A B C D

36. Recollecting Order from Episodic Memory 36 Study this sequence of images

37. Place the images in correct sequence (serial recall) 37 A B C D E F G H I J

38. Average results across 6 problems 38 Mean 

39. Example calibration result for individuals 39 inferred noise level distance to ground truth   individual (pizza sequence; perturbation model)

40. Overview of talk Ordering problems – general knowledge what is the order of US presidents? Ordering problems – episodic memory what is the order of events you have experienced? Matching problems memory for pairs: what object was paired with what person? Recognition memory problems what words were studied? 40

41. Study these combinations 41

42. 23451 BCDE A Find all matching pairs 42

43. Bayesian Matching Model Proposed process: match “known” items guess between remaining ones Individual differences some items easier to know some participants know more 43

44. Graphical Model 44 i items Latent answer key Observed matching Knowledge State Prob. of knowing j individuals person ability item easiness

45. Results across 8 problems 45

46. General Knowledge Matching Problems 46 Dutch Danish Yiddish Thai Vietnamese Chinese Georgian Russian Japanese A B C D E F G H I godt nytår gelukkig nieuwjaar a gut yohr С Новым Годом สวัสดีปีใหม่ Chúc Mừng Nǎm Mới გილოცავთ ახალ წელს

47. Modeling Results – General Knowledge Tasks 47

48. Overview of talk Ordering problems – general knowledge what is the order of US presidents? Ordering problems – episodic memory what is the order of events you have experienced? Matching problems memory for pairs: what object was paired with what person? Recognition memory problems what words were studied? 48

49. Systematic Errors and Biases Some memory errors are systematic When averaging over biased individuals, the group estimate will also be systematically biased … unless the aggregation model can explain the bias 49

50. Listen to these words… 50

51. Associative structure influences false memories 51 cow calf bull herd pasture cattle milk graze

52. Experiment Study list 10 lists of 15 spoken words Recognition memory test Targets (15 items) Lure (1 item) Related distractors (15 items) Unrelated distractors (15 items) Confidence ratings 5-point confidence ratings 1=definitely not on list; 2 = probably not on list; 3 = not sure; 4 = probably on list; 5 = sure it was on the list 52

53. Mean Confidence ratings for 12 individuals 53 Confidence

54. ROC plots for individuals 54

55. Signal Detection Aggregation Model 55 new (z=0) old (z=1) Important: model needs to infer z, whether an item is old or new 3 2145

56. Incorporating Associative Structure 56 cow calf bull herd pasture cattle milk graze

57. Incorporating Associative “Boost” 57 new (z=0) old (z=1) Associative “boost” depends on 1)set of items that are considered “old” 2)vulnerability of individuals to associative influences 3 2145

58. Inferred target status over mcmc iterations 58

59. ROC Curves for SDT Aggregation Models 59

60. Performance of Individuals and Aggregate 60

61. Summary Aggregation of combinatorially complex data going beyond numerical estimates or multiple choice questions Incorporate individual differences going beyond models that treat every vote equally assume some individuals might be “experts” Take cognitive processes into account going beyond mere statistical aggregation allows us to correct for systematic errors and biases 61

62. That’s all 62 Do the experiments yourself: http://psiexp.ss.uci.edu/

63. Predictive Rankings: fantasy football 63 South Australian Football League (32 people rank 9 teams) Australian Football League (29 people rank 16 teams)

64. Experiment 78 participants 17 ordering problems each with 10 items Chronological Events Physical Measures Purely ordinal problems, e.g. Ten Amendments Ten commandments 64

65. Ordering states west-east 65 Oregon (1) Utah (2) Nebraska (3) Iowa (4) Alabama (6) Ohio (5) Virginia (7) Delaware (8) Connecticut (9) Maine (10)

66. Question How many individuals do we need to average over? 66

67. Effect of Group Size: random groups 67 

68. How effective are small groups of experts? Want to find experts endogenously – without feedback Approach: select individuals with the smallest estimated noise levels based on previous tasks We are identifying general expertise (“Pearson’s g”) 68

69. Group Composition based on prior performance 69  T = 0 # previous tasks T = 2 T = 8 Group size (best individuals first)

70. 70 Endogenous no feedback required Exogenous selecting people based on actual performance  

71. Online Experiments Experiment 1 (Prior knowledge) http://madlab.ss.uci.edu/dem2/examples/ Experiment 2a (Serial Recall) study sequence of still images http://madlab.ss.uci.edu/memslides/ Experiment 2b (Serial Recall) study video http://madlab.ss.uci.edu/dem/ 71

72. MDS solution of pairwise tau distances 72 distance to truth

73. MDS solution of pairwise tau distances 73

74. Thurstonian Model – stereotyped event sequences 74

75. Thurstonian Model – “random” videos 75

76. Heuristic Aggregation Approach Combinatorial optimization problem maximizes agreement in assigning N items to N responses Hungarian algorithm construct a count matrix M M ij = number of people that paired item i with response j find row and column permutations to maximize diagonal sum O( n 3 ) 76

77. Hungarian Algorithm Example 77 = correct= incorrect

78. What are methods for finding experts? 1) Self-reported expertise: unreliable  has led to claims of “myth of expertise” 2) Based on explicit scores by comparing to ground truth but ground truth might not be immediately available 3) Endogenously discover experts Use the crowd to discover experts Small groups of experts can be effective 78

79. Predicting problem difficulty 79  std  dispersion of noise levels across individual distance of group answer to ground truth  ordering states geographically city size rankings

80. Mean p( “yes” ) 80 note: confidence ratings were converted to yes/no judgments. Yes = rating >= 3; No = rating < 3

81. Average results over 17 Problems 81 Individuals Mean  Strong wisdom of crowds effect across problems

82. Recollection of 9/11 Event Sequence (Altmann, 2003) 82 AAAAAAAAAAAACCAAAAAAAACEEE BBBBBCCDDBCBAABBBBCCDEDACC CCDCDBBBBDBEBBDDEFDEFBABAA DEFDCDEFFCDCDEEFDDBBBCBCBD FDCEFFDCEEEDFDFEFCFDCDFDDB EFEFEEFECFFFEFCCCEEFEFEFFF Correct Most frequent response (i.e, mode) A = One plane hits the WTC B = A second plane hits the WTC C = One plane crashes into the Pentagon D = One tower at the WTC collapses E = One plane crashes in Pennsylvania F = A second tower at the WTC collapses

83. Example tasks studied in our research Ordering problems what is the order of US presidents? Matching problems memory for pairs: what object was paired with what person? Recognition memory problems what set of words were studied? 83  problems involving combinatorially complex inference problems