1. Labor Marketplace Applications: Human Computer Interaction KAIST KSE Uichin Lee 9/26/2011

2. Soylent: A Word Processor with a Crowd Inside Michael Bernstein, Greg Little, Rob Miller, David Karger, David Crowell, Katrina Panovich MIT CSAIL Björn Hartmann, Mark Ackerman UC Berkeley University of Michigan UIST 2010 Part of slides are from http://projects.csail.mit.edu/soylent/

3. Soylent: a word processing interface Uses crowd contributions to aid complex writing tasks.

4. Challenges for Programming Crowds The authors have interacted with ~9000 Turkers on ~2000 different tasks Key Problem: crowd workers often produce poor output on open-ended tasks – High variance of effort: Lazy Turker vs. Eager Beaver – Errors made by Turkers; e.g., “Of Mice and Men”  “Of Mice” or calling it a movie.. 30% Rule: ~30% of the results from open-ended tasks will be unsatisfactory Solution: find-fix-verify

5. (at least 20% of the Turkers agree upon) (e.g., given to 10 Tuckers) (e.g., given to 5 Tuckers)

6. Find-Fix-Verify Discussion Why split Find and Fix? – Force Lazy Turkers (tend to choose easiest work) to work on a problem of our choice – Allows us to merge work completed in parallel Why add Verify? – Quality rises when we place Turkers in productive tension – Allows us to trade off lag time with quality Timeout at each stage for responsive operations (e.g., 10-20 minutes)

7. Evaluation Implementation – Microsoft World plug-in using MS Visual Studio Tools for Office (VSTO) and Windows Presentation Foundation (WPF) – TurKit Mechanical Turk toolkit Evaluation: – Shortn, Crowdproof, Human Macro Metrics: quality, delay, cost

8. Shortn Evaluation Setting: – Find: 6-10 workers ($0.08 per Find) – Fix/Verify: 3-5 workers ($0.05 per Fix, $0.04 per Verify) – Delay: wait time, work time Results: – 78%-90% reduction.. – Wait time of all stages: median 18.5 minutes – Work time: median 118 seconds – Average costs: $1.41 ($0.55 + $0.48 + $0.38) – Caveats: some modifications are grammatically appropriate, but stylistically incorrect

9. Crowdproof Evaluation Tested 5 input texts – Manually labeled all spelling, grammatical and style errors in each of the five inputs (total 49 errors) Ran Crowdproof w/ a 20-minute stage timeout; and measure how many corrections Crowdproof makes – Task setup: Find: $0.06, Fix: $0.08, Verify: $0.04 Soylent: 67% vs. MS Word (grammar check): 30% – Combined: 82%

10. Human Macro Evaluation Give 5 prompts (Input and Output) to users (CS students, Admin, Author) Ask to generate descriptions (Request) used for Human Macro tasks

11. Discussion Delay in interface outsourcing: minutes to hours.. Privacy? Confidential documents? Legal ownership? Lack of domain knowledge or shared context to usefully contribute

12. VizWiz: Nearly Real-time Answers to Visual Questions Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. UIST 2010. Part of slides are from: http://husk.eecs.berkeley.edu/courses/cs298-52-sp11/images/8/8d/Vizwiz_soylent.pdf

13. VizWiz “automatic and human-powered services to answer general visual questions for people with visual impairments.” Lets blind people use mobile phone to: 1. Take a photo 2. Speak a question 3. Receive multiple spoken answers

14. Motivation Current technology uses automatic approaches to help blind people access visual information – Optical Character Recognition (OCR) – Ex) Kurzweil knfbReader: ~$1,000 Problems: error-prone, limited in scope, expensive – Ex: OCR cannot read graphic labels, handwritten menu, street sign

15. Motivation Solution: ask real people Can phrase questions naturally – “What is the price of the cheapest salad?” vs. OCR reading the entire menu Feedback – Real people can guide blind people to take better photos Focus on blind people’s needs, not current technology

16. Human-Powered Services Problem: Response latency Solution: quikTurkit (and some tricks) – “First attempt to get work done by web based workers in nearly real-time” – Maintain a pool of workers to answer questions

18. quikTurkit “requesters create their own web site on which Mechanical Turk workers answer questions.” “answers are posted directly to the requester’s web site, which allows [them]... to be returned before an entire HIT is complete.” “workers are required to answer multiple previously-asked questions to keep workers around long enough to possibly answer new questions”

19. Answering Site

20. Deployment Setting: 11 blind iPhone users quikTurkit: – Reward range: $0.01 (1/2 of jobs), $0.02/3 (1/2) Results: 86.6% of first answers “correct” – Average of 133.3s latency for first answer Problems: Photos too dark or too blurry and thus unanswerable. VizWiz 2.0 detects and alerts users if photo is too dark or blurry

21. VizWiz: LocateIt Combine VizWiz with computer vision to help blind people locate objects LocateIt mobile: Sensor (zoom and filter) and sonification modules Web interface (angle) (distance)

22. Future Work Expand worker pools beyond Mechanical Turk (e.g. social network) Reduce cost by using game, volunteers, friends Improve interface to make photo-taking easier for blind people Combine automatic approaches to improve delay

23. Discussion Resource usage vs. delay (wasting resources for better responses or near real-time services?) – Any better approach than quikTurkit? Quality control? How do we make sure that the workers correctly identified the photos? – How do systems accept/reject the submissions? (if it’s becoming a large scale service?) Other application scenarios with quikTurkit? Adding inertial sensors to LocateIt?