1. 1 Differences Between Observed and Latent Confidence in Rank Ordering Brent J. Miller Mark Steyvers University of California, Irvine

2. 2 Getting Expert Information

3. 3 Delphi Method Iterative Collaboration

4. 4 George Washington John Adams James Monroe James Madison Thomas Jefferson John Adams James Madison James Monroe Independent Rank-Ordering Tasks Order these U.S. Presidents, by term of office: George Washington 1 st Participant2 nd Participant3 rd Participant John Adams George Washington Thomas Jefferson James Madison James Monroe Thomas Jefferson

5. 5 James Madison Thomas Jefferson John Adams James Madison James Monroe Iterative Rank-Ordering Tasks Order these U.S. Presidents, by term of office: George Washington 1 st Participant George Washington John Adams George Washington James Madison Thomas Jefferson James Monroe 2 nd Participant3 rd Participant John Adams

6. 6 100 Subjects 17 Ranking Questions – Chronological Events(Holidays, U.S. Presidents) – Physical Measures(Landmass, City Pop., State Locations) – Ordinal Information(Ten Amendments, Ten Commandments) 2 Conditions Rank-Ordering Experiment

7. 7 Rank Ordering – Independent & Iteratives 17 Questions via Mechanical Turk 17 Questions from University Students x 4

8. 8 Participant 1 Y 1 : A < B < C X 11 X 12 X 13 ABC Thurstonian Model for Rank Ordering σ1σ1 8

9. 9 Participant 1 Y 1 : A < B < C X 11 X 12 X 13 Y 2 : A < C < B Participant 2 X 22 X 23 X 21 ABC σ2σ2 Thurstonian Model for Rank Ordering 9

10. 10 Iterative Collaboration 33 34 35 36 37 38 39 Participants Position

11. 11 Participant 1 Y 1 : A < B < C X 11 X 12 X 13 ABC Thurstonian Model for Item Representation σ 11 σ 12 σ 13

12. 12 50 Subjects 8 Ranking Questions – Chronological Events(Holidays, U.S. Presidents) – Physical Measures(Landmass, City Pop., State Locations) – Ordinal Information(Ten Amendments, Ten Commandments) Rank-Ordering Experiment 7 5 6 4 1 8 3 2 Independent (8)Iterative (8)

13. 13 Independent vs. Iterative Aggregate Performance Independent Iterative

14. 14 Independent Subject Generative Model μiμi x ij σ ij yjyj i Items j Subjects µ i ≡ Ground truth σ 2 ij ~ InvGamma(.2,1) x i ~ Norm( µ i,σ 2 ij ) Y j ~ Rank(x j )

15. 15 Iterative Subject Generative Model μ ij x i,j-1 σ i,j-1 y j-1 i Items j Subjects x ij σ ij cut yjyj µ i ≡ Ground truth σ 2 ij ~ InvGamma(.2,1) X ij-1 ~ Norm( µ i,σ 2 ij-1 ) Y j-1 ~ Rank(x j-1 )

16. 16 Simulated Aggregate Performance

17. 17 Reduction in Iterative Error by Item Confidence Confidence for Items with Error > 4 Tau No Change

18. 18 Conclusions Item-level knowledge may allow partial expertise between participants, allowing for improvement in group aggregation. Subject item-level confidence appears to support the validity of our proposed model. We should be able to use subjects’ confidence as an estimate of underlying item-level expertise.

19. 19 Danke! Differences Between Observed and Latent Confidence in Rank Ordering