Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Brent Miller, Pernille Hemmer, Mike Yi Michael Lee, Bill Batchelder, Paolo Napoletano

Wisdom of crowds phenomenon Group estimate often performs as well as or better than best individual in the group 2

Examples of wisdom of crowds phenomenon 3 Who wants to be a millionaire? Galton’s Ox (1907): Median of individual estimates comes close to true answer

Tasks studied in our research Ordering/ranking problems declarative memory: order of US presidents, ranking cities by size episodic memory: order of events (i.e., serial recall) predictive rankings: fantasy football Matching problems assign N items to N responses e.g., match paintings to artists, or flags to countries Traveling Salesman problems find shortest route between cities  problems involving permutations 4

Ulysses S. Grant James Garfield Rutherford B. Hayes Abraham Lincoln Andrew Johnson James Garfield Ulysses S. Grant Rutherford B. Hayes Andrew Johnson Abraham Lincoln Recollecting order from Declarative Memory time Place these presidents in the correct order

Recollecting order from episodic memory 6

Place scenes in correct order (serial recall) 7 time A B C D

Goal: aggregating responses 8 D A B C A B D C B A D CA C B D A D B C Aggregation Algorithm A B C D ground truth = ? group answer

Bayesian Approach 9 D A B C A B D C B A D CA C B D A D B C Generative Model A B C D group answer = latent random variable

Task constraints No communication between individuals There is always a true answer (ground truth) Aggregation algorithm never has access to ground truth unsupervised methods ground truth only used for evaluation 10

Research Goals Aggregation of permutation data going beyond numerical estimates or multiple choice questions combinatorially complex Incorporate individual differences going beyond models that treat every vote equally assume some individuals might be “experts” Take cognitive processes into account going beyond mere statistical aggregation  Hierarchical Bayesian models 11

Part I Ordering Problems 12

Experiment 1 Task: order all 44 US presidents Methods 26 participants (college undergraduates) Names of presidents written on cards Cards could be shuffled on large table 13

= 1 = 1+1 Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps Ordering by Individual ABECD True Order ABCDE C D E ABAB AB ECD ABCDEABCDE = 2

Empirical Results 15  (random guessing)

Probabilistic models Thurstone (1927), Mallows (1957), Plackett-Luce (1975) Lebanon-Mao (2008) Spectral methods Diaconis (1989) Heuristic methods from voting theory Borda count … however, many of these approaches were developed for preference rankings Many methods for analyzing rank data… 16

Bayesian models constrained by human cognition Extension of Thurstone’s (1927) model Extension of Estes (1972) perturbation model 17

Bayesian Thurstonian Approach 18 Each item has a true coordinate on some dimension A B C

Bayesian Thurstonian Approach 19 A B C … but there is noise because of encoding and/or retrieval error Person 1

Bayesian Thurstonian Approach 20 Each person’s mental representation is based on (latent) samples of these distributions B C A B C Person 1 A

Bayesian Thurstonian Approach 21 B C A B C The observed ordering is based on the ordering of the samples A < B < C Observed Ordering: Person 1 A

Bayesian Thurstonian Approach 22 People draw from distributions with common means but different variances Person 1 B C A B C A < B < C Observed Ordering: Person 2 A B C B C Observed Ordering: A < C < B A A

Graphical Model Notation 23 j=1..3 shaded = observed not shaded = latent

Graphical Model of Bayesian Thurstonian Model 24 j individuals Latent ground truth Individual noise level Mental representation Observed ordering

Inference Need the posterior distribution Markov Chain Monte Carlo Gibbs sampling on Metropolis-hastings on and Draw 400 samples group ordering based on average of across samples 25

(weak) wisdom of crowds effect 26  model’s ordering is as good as best individual (but not better)

Inferred Distributions for 44 US Presidents 27 George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44) median and minimum sigma

Model can predict individual performance 28   inferred noise level for each individual distance to ground truth  individual

Extension of Estes (1972) Perturbation Model Main idea: item order is perturbed locally Our extension: perturbation noise varies between individuals and items 29 A True order BCDE Recalled order DB C E A

Modified Perturbation Model 30

Strong wisdom of crowds effect 31  Perturbation model’s ordering is better than best individual Perturbation

Inferred Perturbation Matrix and Item Accuracy 32 Abraham Lincoln Richard Nixon James Carter

Alternative Heuristic Models Many heuristic methods from voting theory E.g., Borda count method Suppose we have 10 items assign a count of 10 to first item, 9 for second item, etc add counts over individuals order items by the Borda count i.e., rank by average rank across people 33

Model Comparison 34  Borda

Experiment 2 78 participants 17 problems each with 10 items Chronological Events Physical Measures Purely ordinal problems, e.g. Ten Amendments Ten commandments 35

Example results Oregon (1) 2. Utah (2) 3. Nebraska (3) 4. Iowa (4) 5. Alabama (6) 6. Ohio (5) 7. Virginia (7) 8. Delaware (8) 9. Connecticut (9) 10. Maine (10) 1. Freedom of speech & relig... (1) 2. Right to bear arms (2) 3. No quartering of soldiers... (3) 4. No unreasonable searches (4) 5. Due process (5) 6. Trial by Jury (6) 7. Civil Trial by Jury (7) 8. No cruel punishment (8) 9. Right to non-specified ri... (10) 10. Power for the States & Pe... (9) Perturbation ModelThurstonian Model

Average results over 17 Problems 37 Individuals Mean  Strong wisdom of crowds effect across problems

Predicting problem difficulty 38  std  dispersion of noise levels across individual distance of group answer to ground truth  ordering states geographically city size rankings

Effect of Group Composition How many individuals do we need to average over? 39

Effect of Group Size: random groups 40 

Experts vs. Crowds Can we find experts in the crowd? Can we form small groups of experts? Approach Form a group for some particular task Select individuals with the smallest sigma (“experts”) based on previous tasks Vary the number of previous tasks 41

Group Composition based on prior performance 42  T = 0 # previous tasks T = 2 T = 8 Group size (best individuals first)

Methods for Selecting Experts 43 Endogenous: no feedback required Exogenous: selecting people based on actual performance  

Aggregating Episodic Memories 44 Study this sequence of images

Place the images in correct sequence (serial recall) 45 A B C D E F G H I J

Average results across 6 problems 46 Mean 

Example calibration result for individuals 47 inferred noise level distance to ground truth   individual (pizza sequence; perturbation model)

Predictive Rankings: fantasy football 48 South Australian Football League (32 people rank 9 teams) Australian Football League (29 people rank 16 teams)

Part II Matching Problems 49

Study these combinations 50

23451 BCDE A Find all matching pairs 51

Experiment 15 subjects 8 problems 4 problems with 5 items 4 problems with 10 items 52

Mean accuracy across 8 problems 53

Bayesian Matching Model Proposed process: match “known” items guess between remaining ones Individual differences some items easier to know some participants know more 54

Graphical Model 55 i items Latent ground truth Observed matching Knowledge State Prob. of knowing j individuals person ability item easiness

Modeling results across 8 problems 56

Calibration at level of items and people 57 ITEMS INDIVIDUALS (for weapons and faces 10 items problem)

Varying number of individuals 58

How predictive are subject provided confidence ratings? 59 # guesses estimated by individual Accuracy # guesses estimated by model (based on variable A) r=-.50 r=-.81

Another matching problem 60 Dutch Danish Yiddish Thai Vietnamese Chinese Georgian Russian Japanese A B C D E F G H I godt nytår gelukkig nieuwjaar a gut yohr С Новым Годом สวัสดีปีใหม่ Chúc Mừng Nǎm Mới გილოცავთ ახალ წელს

Experiment 17 Participants 8 matching problems, e.g. car logo’s and brand names first and last names philosophers flags and countries greek symbols and letter names Number of items varied between 10 and 24 with 24 items, we have 24! possibilities 61

Modeling Results – Declarative Tasks 62

Calibration at level of items and people (for paintings problem) 63 ITEMS INDIVIDUALS

How predictive are subject provided confidence ratings? 64 # guesses estimated by individual Accuracy # guesses estimated by model (based on variable A) r=-.42 r=-.77

Part III Traveling Salesman Problems 65

Find the shortest route between cities 66 B30-21 Individual 5Individual 83 Individual 60 Optimal

Dataset Vickers, Bovet, Lee, & Hughes (2003) 83 participants 7 problems of 30 cities

TSP Aggregation Problem Propose a good solution based on all individual solutions Task constraints Data consists of city order only No access to city locations 68

Approach Find tours with edges for which many individuals agree Calculate agreement matrix A A = n × n matrix, where n is the number of cities a ij indicates the number of participants that connect cities i and j. Find tour that maximizes 69 (this itself is a non-Euclidian TSP problem)

Line thickness = agreement 70

Blue = Aggregate Tour 71

Results averaged across 7 problems aggregate

Results Weight: c = 2.0 path length # subj better # subj same # subj worse % % % % % % % %0083 Problemsubj Min.subj Mean A %+3.246% A %+4.791% A %+5.936% B %+5.502% B %+4.992% B %+5.325% B %+5.497% All %+5.041% Individuals Model best individual performance across 7 problems model performance across 7 problems outperforms best individual

Part IV Summary & Conclusions 74

When do we get wisdom of crowds effect? Independent errors different people knowing different things Some minimal number of individuals individuals often sufficient 75

What are methods for finding experts? 1) Self-reported expertise: unreliable  has led to claims of “myth of expertise” 2) Based on explicit scores by comparing to ground truth but ground truth might not be immediately available 3) Endogenously discover experts Use the crowd to discover experts Small groups of experts can be effective 76

What to do about systematic biases? In some tasks, individuals systematically distort the ground truth spatial and temporal distortions memory distortions (e.g. false memory) decision-making distortions Does this diminish the wisdom of crowds effect? maybe… but a model that predicts these systematic distortions might be able to “undo” them 77

Conclusion Effective aggregation of human judgments requires cognitive models Psychology and cognitive science can inform aggregation models 78

That’s all 79 Do the experiments yourself:

Online Experiments Experiment 1 (Prior knowledge) Experiment 2a (Serial Recall) study sequence of still images Experiment 2b (Serial Recall) study video 80

Graphical Model 81 i items Latent ground truth Observed matching Knowledge State Prob. of knowing j individuals item and person parameters

MDS solution of pairwise tau distances 82 distance to truth

MDS solution of pairwise tau distances 83

Hierarchical Bayesian Models Generative models ordering information cognitively plausible individual differences Group response = probability distribution over all permutations of N items With N=44 items, we have 44! > combinations Approximate inference methods: MCMC 84

Model incorporating overall person ability 85 j individuals Overall ability Task specific ability m tasks j individuals

Average results over 17 Problems 86 Mean  new model

Thurstonian Model – stereotyped event sequences 87

Thurstonian Model – “random” videos 88

Heuristic Aggregation Approach Combinatorial optimization problem maximizes agreement in assigning N items to N responses Hungarian algorithm construct a count matrix M M ij = number of people that paired item i with response j find row and column permutations to maximize diagonal sum O( n 3 ) 89

Hungarian Algorithm Example 90 = correct= incorrect