Download presentation
Presentation is loading. Please wait.
Published byDwight Campbell Modified over 9 years ago
1
Task and Workflow Design I KSE 801 Uichin Lee
2
TurKit: Human Computation Algorithms on Mechanical Turk Greg Little, Lydia B. Chilton, Rob Miller, and Max Goldman (MIT CSAIL) UIST 2010
3
Workflow in M-Turk HIT Data Collected in CSV File Requester posts HIT Groups to Mechanical Turk Data Exported for Use
4
Workflow: Pros & Cons Easy to run simple, parallelized tasks. Not so easy to run tasks in which turkers improve on or validate each others’ work. TurKit to the rescue!
5
The TurKit Toolkit Arrows indicate the flow of information. Programmer writes 2 sets of source code: – HTML files for web servers – JavaScript executed by TurKit Output is retrieved via a JavaScript database. Turkers Mechanical Turk Web ServerTurKit *.html*.js Programmer JavaScript Database
6
Crash-and-rerun programming model Observation: local computation is cheap, but the external class cost money Managing states over a long running program is challenging – Examples: Computer restarts? Errors? Solution: store states in the database (in case) If an error happens, just crash the program and re-run by following the history in DB – Throw a “crash” exception; the script is automatically re-run. New keyword “once”: – Remove non-determinism – Don’t need to re-execute an expensive operation (when re-run) But why should we re-run???
7
Example: quicksort
8
Parallelism First time the script runs, HITs A and C will be created For a given forked branch, if a task fails (e.g., HIT A), TurKit crashes the forked branch (and re-run) Synchronization w/ join()
9
MTurk Functions Prompt(message, # of people) – mturk.prompt("What is your favorite color?", 100) Voting(message, options) Sort(message, items) VOTE()SORT()
10
TurKit: Implementation TurKit: Java using Rhino to interpret JavaScript code, and E4X2 to handle XML results from MTurk IDE: Google App Engine3 (GAE) Online IDE
11
Exploring Iterative and Parallel Human Computation Processes Greg Little, Lydia B. Chilton Max Goldman, Robert C. Miller HCOMP 2010
12
HC Task Model Dimension: – Dependent (iterative) or independent (parallel) tasks – Creation and decision tasks Task model examples Creation tasks (creating new content): e.g., writing ideas, imagery solutions, etc. Decision tasks (voting/rating): e.g., rating the quality of a description of an image
13
HC Task Model Combining tasks: iterative and parallel tasks Iterative pattern: a sequence of creation tasks where the result of each task feeds into the next one, followed by a comparison task Parallel pattern: a set of creation tasks executed in parallel, followed by a task of choosing the best
14
Experiment: Writing Image Description Iterative vs. parallel; each 6 creation tasks ($0.02), followed by rating tasks (1-10 scale, $0.01)
15
Experiment: Writing Image Description Turkers in iterative condition gave better description while parallel condition always shows an empty text area.
16
Experiment: Writing Image Description Average rating after n iterations – After six iterations: 7.9 vs. 7.4, t-test T 29 =2.1, p=0.04 iterative parallel
17
Experiment: Writing Image Description Length vs. rating: positive correlation The two outliers (circled) represent instances of text copied from the Internet (with superficial description) Length (characters) Rating
18
Experiment: Writing Image Description Work Quality: – 31% mainly append content at the end, and make only minor modifications (if any) to existing content; – 27% modify/expand existing content, but it is evident that they use the provided description as a basis; – 17% seem to ignore the provided description entirely and start over; – 13% mostly trim or remove content; – 11% make very small changes (adding a word, fixing a misspelling, etc); – 1% copy-paste superficially related content found on the internet. Creating vs. improving (takes about the same time, avg. 211 seconds)
19
Experiment: Brainstorming
20
Iterative work: higher average rating – Biased thinking: e.g., tech -> xxtech -> yytech Parallel work: diversity, higher deviation (rating) – No iteration for brainstorming IterationRating Avg. Rating iterative parallel
21
Example: Blurry Text Recognition
22
Iterative performs better than parallel Iteration Accuracy
23
Summary TurKit: a flexible programming tool for m-turk Various work-flow can be designed; e.g., iterative, parallel, and hybrid Iterative performs better than parallel in several cases (e.g., image description, brainstorming, text recognition)
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.