Collective Revelation: A Mechanism for Self-Verified, Weighted, and Truthful Predictions Sharad Goel, Daniel M. Reeves, David M. Pennock Presented by: Nir Shabbat

Outline Introduction – Background and Related Work The Settings A Mechanism For Collective Revelation – The Basic Mechanism – A General Technique for Balancing Budgets Summary

Introduction In many cases, a decision maker may seek to elicit and aggregate the opinions of multiple experts. Ideally, would like a mechanism that is: 1.Incentive Compatible - Rewards participants to be truthful 2.Information Weighted - Adjusts for the fact that some experts are better informed than others 3.Self-Verifying - Works without the need for objective, “ground truth” observations 4.Budget Balanced - Makes no net transfers to agent

Background and Related Work Proper Scoring Rules – Rewards the agent by assessing his forecast against an actual observed outcome. Agents are incentivized to be truthful. For example, using the Brier scoring rule: We get:

Background and Related Work Peer Prediction Methods – Can be Incentive Compatible and Self-Verifying. For example, the Bayesian truth serum asks agents to report both their own prediction and their prediction of other agents’ predictions. Relies on the general property of information- scores, that a truthful answer constitutes the best guess about the most “surprisingly common” answer.

Background and Related Work

Predictions Markets – encourage truthful behavior and automatically aggregate predictions from agents with diverse information. For example, a prediction market for U.S. presidential elections. Agents buy and sell assets tied to an eventual Democratic or Republican win. Each share pay $1 if the corresponding event occurs.

Background and Related Work Delphi Method – generates consensus predictions from experts generally through a process of structured discussion. In each round, each expert anonymously provides his forecast and reasons, at the end of the round, market maker summarize the forecasts and reasons. Process is repeated until forecasts converge.

Background and Related Work Competitive Forecasting – elicits confidence intervals on predictions, thereby facilitating information weighting. Like prediction markets, competitive forecasting rewards accuracy, though is not rigorously incentive compatible and relies on benchmarking against objective measurements.

Background and Related Work Incentive Compatible Information Weighted Self- Verifying Proper Scoring Rules ● Prediction Markets ○● Peer Prediction ○● Delphi Method ● Competitive Forecasting ○● Polls ● Collective Revelation ○●●

The Settings

A Mechanism For Collective Revelation

The Basic Mechanism

Simply put, scoring against a single Bernoulli observation can only reveal the agent’s expectation and not the uncertainty of it’s prediction (as quantified by the variance). So in order for a mechanism to be Information-Weighted (for Bernoulli outcome) it must score agents against multiple observations.

The Basic Mechanism

THEOREM 1 (Cont’d) Then: (*) is a strict Nash Equilibrium.

The Basic Mechanism

CORROLARY Suppose all agents play the equilibrium strategy. Define:

The Basic Mechanism

A General Technique for Balancing Budgets

PROOF (Cont’d) Summing over all players we get:

A General Technique for Balancing Budgets

Summary We’ve seen a mechanism, i.e. Collective Revelation, for aggregating experts opinions which is: – Incentive Compatible – Information Weighed – Self-Verifying We’ve seen a general technique for constructing budget balanced mechanisms that applies both to collective revelation and to past peer- predictions method.

THE END