1. Matching Experiment Class Results

2. Experiment Analyzed 114 subjects after removal of subjects who completed fewer than 4 problems 8 problems 2

3. Problem 1: Greek symbols 3

4. Problem 2: Philosophers 4

5. Problem 3: Philosophers 5

6. Problem 4: Presidents 6

7. Problem 5: Car Logo’s 7

8. Problem 6: Languages 8

9. Problem 7: Languages 9

10. Mean accuracy by problem: Greek symbols0.48 Philosophers0.47 Flags0.53 US presidents 0.50 Car logos0.86 Languages0.54 Sport balls0.79 accuracy is the mean number of correct matches averaged over subjects; note: removed the artists problem because of data-coding problem 10

11. Mean accuracy by subject: 11 Individuals are ordered from best (left) to worst (right)

12. Goal: aggregating responses 12 D A B C A B D C B A D CA C B D A D B C Aggregation Algorithm A B C D ground truth = ? group answer

13. Heuristic Aggregation Approach Combinatorial optimization problem maximizes agreement in assigning N items to N responses Hungarian algorithm construct a count matrix M M ij = number of people that paired item i with response j find row and column permutations to maximize diagonal sum O( n 3 ) 13

14. Hungarian Algorithm Example (based on a small dataset) 14 = correct= incorrect

15. Bayesian Matching Model Proposed process: match “known” items guess between remaining ones Individual differences some items easier to know some participants know more 15

16. Graphical Model 16 i items Latent ground truth Observed matching Knowledge State Prob. of knowing j individuals person ability item easiness

17. Wisdom of Crowds effect 17 aggregation models outperform all individuals

18. Effect of Varying Number of Subjects for Hungarian Algorithm 18