Matching Experiment Class Results

Experiment Analyzed 114 subjects after removal of subjects who completed fewer than 4 problems 8 problems 2

Problem 1: Greek symbols 3

Problem 2: Philosophers 4

Problem 3: Philosophers 5

Problem 4: Presidents 6

Problem 5: Car Logo’s 7

Problem 6: Languages 8

Problem 7: Languages 9

Mean accuracy by problem: Greek symbols0.48 Philosophers0.47 Flags0.53 US presidents 0.50 Car logos0.86 Languages0.54 Sport balls0.79 accuracy is the mean number of correct matches averaged over subjects; note: removed the artists problem because of data-coding problem 10

Mean accuracy by subject: 11 Individuals are ordered from best (left) to worst (right)

Goal: aggregating responses 12 D A B C A B D C B A D CA C B D A D B C Aggregation Algorithm A B C D ground truth = ? group answer

Heuristic Aggregation Approach Combinatorial optimization problem maximizes agreement in assigning N items to N responses Hungarian algorithm construct a count matrix M M ij = number of people that paired item i with response j find row and column permutations to maximize diagonal sum O( n 3 ) 13

Hungarian Algorithm Example (based on a small dataset) 14 = correct= incorrect

Bayesian Matching Model Proposed process: match “known” items guess between remaining ones Individual differences some items easier to know some participants know more 15

Graphical Model 16 i items Latent ground truth Observed matching Knowledge State Prob. of knowing j individuals person ability item easiness

Wisdom of Crowds effect 17 aggregation models outperform all individuals

Effect of Varying Number of Subjects for Hungarian Algorithm 18