1. When Do Noisy Votes Reveal the Truth? Ioannis Caragiannis 1 Ariel D. Procaccia 2 Nisarg Shah 2 ( speaker ) 1 University of Patras & CTI 2 Carnegie Mellon University

2. What? Why? What?  Alternatives to be compared  True order (unknown ground truth)  Noisy estimates (votes) drawn from some distribution around it  Q: How many votes are needed to accurately find the true order? Why?  Practical motivation  Theoretical motivation 2 a > b > c > d b > a > c > d a > c > b > d a > b > d > c Alternatives a, b, c, d

3. Practical Motivation 1. Human Computation  EteRNA, Foldit, Crowdsourcing …  How many users/workers are required? 2. Judgement Aggregation  Jury system, experts ranking restaurants, …  How many experts are required?

4. Theoretical Motivation Maximum Likelihood Estimator (MLE) View: Is a given voting rule the MLE for any noise model? Problems  Only 1 MLE/noise model  Strange noise models  Noise model is usually unknown Our Contribution  MLE is too stringent!  Just want low sample complexity  Family of reasonable noise models 4 Voting Rules Noise Models

5. Boring Stuff! 5

6. Sample Complexity for Mallows’ Model 6

7. PM-c and PD-c Rules 7 Pairwise Majority Consistent Rules (PM-c)  Must match the pairwise majority graph whenever it is acyclic  Condorcet consistency for social welfare functions a b c d

8. PM-c and PD-c Rules 8 PD-c rules  similar, but focus on positions of alternatives PM-c PD-c KM SL CP RP SC BL PSR

9. The Big Picture 9  Kemeny rule + uniform tie breaking  Optimal sample complexity PM-c  PM-c  O(log m) (m = #alternatives)  Any voting rule  Ω(log m) Logarithmic Polynomial Exponential Many scoring rules  Plurality, veto  Strictly exponential

10. Take-Away - I 10  Given any fixed noise model, sample complexity is a clear and useful criterion for selecting voting rules Hey, what happened to the noise model being unknown?

11. Generalization 11 Stronger need  Unknown noise model Working well on a family of reasonable noise models Problems 1.What is reasonable? 2.HUGE sample complexity for near-extreme parameter values! Relaxation  Accuracy in the Limit Ground truth with probability 1 given infinitely many samples Novel axiomatic property

12. Accuracy in the Limit 12 Voting RulesNoise models for which they are accurate in the limit PM-c + PD-cMallows’ model (probability decreases exponentially in the KT distance) PM-c + PD-cAll KT-monotonic noise models (probability decreases monotonically in the KT distance) PM-cAll d-monotonic iff d = Majority Concentric (MC) PD-cAll d-monotonic iff d = Position Concentric (PC) PM-c + PD-cAll d-monotonic iff d = both MC and PC Monotonicity is reasonable, but why Kendall-Tau distance?

13. Take-Away - II 13  Robustness  accuracy in the limit over a family of reasonable noise models  d-monotonic noise models  reasonable  If you believe in PM-c and PD-c rules  look for distances that are both MC and PC  Kendall-Tau, footrule, maximum displacement  Cayley distance and Hamming distance are neither MC nor PC  Even the most popular rule – plurality – is not accurate in the limit for any monotonic noise model over either distance !  Lose just too much information for the true ranking to be recovered

14. Distances over Rankings 14 σ*σ* σ*σ*

15. Discussion 15