1. Matching Experiment
Class Results

2. Experiment Analyzed 114 subjects 8 problems
after removal of subjects who completed fewer than 4 problems
8 problems

3. Problem 1: Greek symbols

4. Problem 2: Philosophers

5. Problem 3: Philosophers

6. Problem 4: Presidents

7. Problem 5: Car Logo’s

8. Problem 6: Languages

9. Problem 7: Languages

10. Mean accuracy by problem:
Greek symbols 0.48
Philosophers 0.47
Flags 0.53
US presidents 0.50
Car logos 0.86
Languages 0.54
Sport balls 0.79
accuracy is the mean number of correct matches averaged over subjects;
note: removed the artists problem because of data-coding problem

11. Mean accuracy by subject:
Individuals are ordered from best (left) to worst (right)

12. Goal: aggregating responses
group answer
ground truth ?
A B C D =
A B C D
Aggregation Algorithm
D A B C
A B D C
B A D C
A C B D
A D B C

13. Heuristic Aggregation Approach
Combinatorial optimization problem
maximizes agreement in assigning N items to N responses
Hungarian algorithm
construct a count matrix M
Mij = number of people that paired item i with response j
find row and column permutations to maximize diagonal sum
O( n3 )

14. Hungarian Algorithm Example (based on a small dataset)
= correct
= incorrect

15. Bayesian Matching Model
Proposed process:
match “known” items
guess between remaining ones
Individual differences
some items easier to know
some participants know more

16. Graphical Model item easiness person ability Prob. of knowing
i items
Prob. of knowing
Latent ground truth
Knowledge State
Observed matching
j individuals

17. Wisdom of Crowds effect
aggregation models outperform all
individuals

18. Effect of Varying Number of Subjects for Hungarian Algorithm