Crowdsourcing Control: Moving Beyond Multiple Choice (Full paper in UAI 2012) Christopher H. Lin, Mausam, Daniel S. Weld University of Washington HCOMP-2012, Toronto, ON, Canada TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A AA 1

2 Is this a face? Yes No Yes No

3 Is this a face? Yes No Yes No

4 Is this a face? Yes No Yes No

5 Is this a face? Yes No Yes No

Update Posteriors 6

7 How many people are there in this military base? 30,000 Submit Source:Wikipedia

Update Posteriors 8

9

= ?

= ? EASY d = 0

12 ? =

13 ? = HARD d = 1

Source:Wikipedia 14

Worker Ability Difficulty Accuracy 15

[Aldous, 1985] 16

A C B ?? 17

A C B ?? 18

A C B ?? + 19

A C B ?? 20

A C B ?? + 21

Correct Answer Difficulty 22 Agent State

23

24

25

LazySusanMV Average Accuracy (%) Average Utility

27

LazySusan (EM) MV Average Accuracy (%)

Contributions A probabilistic generative model for worker responses on tasks with an infinite number of possible answers A decision-theoretic agent that dynamically infers the right answer Real experiments that show our agent works 29

Repeated Labeling Romney et al, 1986; Sheng et al, 2008; Raykar et al, 2010; Whitehill et al, 2009; Dai et al, 2010, 2011; Kamar et al, 2012; Parameswaran et al, 2010; Karger et al, 2011; Snow et al, 2008; Complicated Workflows Little et al, 2009; Bernstein et al, 2010; Liem et al, 2011; Noronha et al, 2011; Lasecki et al, 2011; Kulkarni et al, 2012; AI Control of Workflows Dai et al, 2010,2011; Shahaf et al, 2010; Weld et al, 2011; Kamar et al, 2012; 30

Christopher H. Lin, Mausam, and Daniel S. Weld. “Dynamically Switching between Synergistic Workflows.” Wednesday, AI and the Web 2 2:25 PM 31