1. Crowdsourcing Control: Moving Beyond Multiple Choice (Full paper in UAI 2012) Christopher H. Lin, Mausam, Daniel S. Weld University of Washington HCOMP-2012, Toronto, ON, Canada TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A AA 1

2. 2 www.crowdsourcing.com Is this a face? Yes No Yes No

3. 3 www.crowdsourcing.com Is this a face? Yes No Yes No

4. 4 www.crowdsourcing.com Is this a face? Yes No Yes No

5. 5 www.crowdsourcing.com Is this a face? Yes No Yes No

6. Update Posteriors 6

7. 7 How many people are there in this military base? www.crowdsourcing.com 30,000 Submit Source:Wikipedia

8. Update Posteriors 8

9. 9

10. 10 2 + 2 = ? www.crowdsourcing.com

11. 11 2 + 2 = ? www.crowdsourcing.com EASY d = 0

12. 12 www.crowdsourcing.com ? =

13. 13 www.crowdsourcing.com ? = HARD d = 1

14. Source:Wikipedia 14

15. Worker Ability Difficulty Accuracy 15

16. [Aldous, 1985] 16

17. A C B ?? 17

18. A C B ?? 18

19. A C B ?? + 19

20. A C B ?? 20

21. A C B ?? + 21

22. Correct Answer Difficulty 22 Agent State

23. 23

24. 24

25. 25

26. LazySusanMV Average Accuracy (%) 99.2595.52 Average Utility -5.92-9.94 26

27. 27

28. LazySusan (EM) MV Average Accuracy (%) 65.554.1 28

29. Contributions A probabilistic generative model for worker responses on tasks with an infinite number of possible answers A decision-theoretic agent that dynamically infers the right answer Real experiments that show our agent works 29

30. Repeated Labeling Romney et al, 1986; Sheng et al, 2008; Raykar et al, 2010; Whitehill et al, 2009; Dai et al, 2010, 2011; Kamar et al, 2012; Parameswaran et al, 2010; Karger et al, 2011; Snow et al, 2008; Complicated Workflows Little et al, 2009; Bernstein et al, 2010; Liem et al, 2011; Noronha et al, 2011; Lasecki et al, 2011; Kulkarni et al, 2012; AI Control of Workflows Dai et al, 2010,2011; Shahaf et al, 2010; Weld et al, 2011; Kamar et al, 2012; 30

31. Christopher H. Lin, Mausam, and Daniel S. Weld. “Dynamically Switching between Synergistic Workflows.” Wednesday, AI and the Web 2 2:25 PM 31