1. Task and Workflow Design I KSE 801 Uichin Lee

2. TurKit: Human Computation Algorithms on Mechanical Turk Greg Little, Lydia B. Chilton, Rob Miller, and Max Goldman (MIT CSAIL) UIST 2010

3. Workflow in M-Turk HIT Data Collected in CSV File Requester posts HIT Groups to Mechanical Turk Data Exported for Use

4. Workflow: Pros & Cons Easy to run simple, parallelized tasks. Not so easy to run tasks in which turkers improve on or validate each others’ work. TurKit to the rescue!

5. The TurKit Toolkit Arrows indicate the flow of information. Programmer writes 2 sets of source code: – HTML files for web servers – JavaScript executed by TurKit Output is retrieved via a JavaScript database. Turkers Mechanical Turk Web ServerTurKit *.html*.js Programmer JavaScript Database

6. Crash-and-rerun programming model Observation: local computation is cheap, but the external class cost money Managing states over a long running program is challenging – Examples: Computer restarts? Errors? Solution: store states in the database (in case) If an error happens, just crash the program and re-run by following the history in DB – Throw a “crash” exception; the script is automatically re-run. New keyword “once”: – Remove non-determinism – Don’t need to re-execute an expensive operation (when re-run) But why should we re-run???

7. Example: quicksort

8. Parallelism First time the script runs, HITs A and C will be created For a given forked branch, if a task fails (e.g., HIT A), TurKit crashes the forked branch (and re-run) Synchronization w/ join()

9. MTurk Functions Prompt(message, # of people) – mturk.prompt("What is your favorite color?", 100) Voting(message, options) Sort(message, items) VOTE()SORT()

10. TurKit: Implementation TurKit: Java using Rhino to interpret JavaScript code, and E4X2 to handle XML results from MTurk IDE: Google App Engine3 (GAE) Online IDE

11. Exploring Iterative and Parallel Human Computation Processes Greg Little, Lydia B. Chilton Max Goldman, Robert C. Miller HCOMP 2010

12. HC Task Model Dimension: – Dependent (iterative) or independent (parallel) tasks – Creation and decision tasks Task model examples Creation tasks (creating new content): e.g., writing ideas, imagery solutions, etc. Decision tasks (voting/rating): e.g., rating the quality of a description of an image

13. HC Task Model Combining tasks: iterative and parallel tasks Iterative pattern: a sequence of creation tasks where the result of each task feeds into the next one, followed by a comparison task Parallel pattern: a set of creation tasks executed in parallel, followed by a task of choosing the best

14. Experiment: Writing Image Description Iterative vs. parallel; each 6 creation tasks ($0.02), followed by rating tasks (1-10 scale, $0.01)

15. Experiment: Writing Image Description Turkers in iterative condition gave better description while parallel condition always shows an empty text area.

16. Experiment: Writing Image Description Average rating after n iterations – After six iterations: 7.9 vs. 7.4, t-test T 29 =2.1, p=0.04 iterative parallel

17. Experiment: Writing Image Description Length vs. rating: positive correlation The two outliers (circled) represent instances of text copied from the Internet (with superficial description) Length (characters) Rating

18. Experiment: Writing Image Description Work Quality: – 31% mainly append content at the end, and make only minor modifications (if any) to existing content; – 27% modify/expand existing content, but it is evident that they use the provided description as a basis; – 17% seem to ignore the provided description entirely and start over; – 13% mostly trim or remove content; – 11% make very small changes (adding a word, fixing a misspelling, etc); – 1% copy-paste superficially related content found on the internet. Creating vs. improving (takes about the same time, avg. 211 seconds)

19. Experiment: Brainstorming

20. Iterative work: higher average rating – Biased thinking: e.g., tech -> xxtech -> yytech Parallel work: diversity, higher deviation (rating) – No iteration for brainstorming IterationRating Avg. Rating iterative parallel

21. Example: Blurry Text Recognition

22. Iterative performs better than parallel Iteration Accuracy

23. Summary TurKit: a flexible programming tool for m-turk Various work-flow can be designed; e.g., iterative, parallel, and hybrid Iterative performs better than parallel in several cases (e.g., image description, brainstorming, text recognition)