CrowdForge: Crowdsourcing Complex Work Aniket Kittur, Boris Smus, Robert E. Kraut February 1, 2011 Presenter: Karalis Alexandros 1

Micro-task markets example: Amazon’s Mechanical Turk (MTurk) Simple, independent tasks Large population of workers Less cost for employers 2

Why not complex tasks? “Turkers” do it for: an additional income to kill some of their free time as a hobby NOT as a their primary job Result: They will not engage in long or complex tasks (most of them at least) 3

Example of a problem in real world: “Newspaper article” article structure collect information editing taking pictures decide the scope other … 4

Complex tasks need coordination 5

Approach Support the coordination dependencies involved in complex work through micro-task markets A pre-specified partition that breaks up the audio into smaller subtasks A flow that controls the sequencing of the tasks and transfer of information between them A quality control phase that involves verification of one task by another worker Automatic aggregation of the results 6

Example: Audio transcription task from Castingwords.com 7

The proposed framework aims to support: Multi-level partitions in which a task can be broken up by more than one partition Dynamic partitioning so that workers themselves can decide how to partition a task, with their results generating new subtasks during the flow (rather than the task designer needing fully specify partitioning beforehand) Complex flows involving many tasks and many workers A variety of quality control methods including voting, verification, or merging items Intelligent aggregation of results both automatically and by workers. A simple method for specifying and managing tasks and flows between tasks 8

Similarities with MapReduce framework Partition Tasks: Larger task → Discrete subtasks Map Tasks: Specified task → Process by on or more workers Reduce Tasks: Result of multiple workers’ tasks → Merge into one 9

A representative complex task example: An article Step #1: Partition Article outline for specific section (e.g. History) Step #2 : Map Not a whole paragraph but a single fact Step #3: Reduce Concatenation or integration 10

Conclusion of article example 11 Mean values of evaluation: Group: 4.01 Individually: 3.75 Wikipedia: 3.95 Total costs: Group: $3.26 Individual: $3.05 Criteria: Usage of facts Spelling & grammar Article structure Personal preference

Quality control possible problem: a bad outline → a bad task Approach (Reduce type task): all the outlines ↓ Given as sets to new workers to merge ↓ Error correction + Better combination of elements = Better result! 12

Results of quality control 13 Merged outlines were preferred more!

It looks very good and effective! But… What about its generality? Is it for those kinds of work only? 14

Should we try it differently? Maybe in an everyday problem of decision: A suburban family with two children want to buy a new car: Car usage: ● Taking children to school and back. ● Occasionally enjoys road trips. (Researching purchase decision problem) 15

Approach Partition: One worker’s task: Submit criteria for a suitable automotive purchase Other worker’s task: Submit a list of potential suitable cars → Ended up to an empty matrix with characteristics like safety and reliability 16

Approach Map: Filing cells in the matrix with facts Reduce: Based on the facts from mapping workers wrote a sentence 17 Total cost: $3.70 HITs: 54 Note: Not individual task

Prototype Written in Python Communication with Mturk via Boto Notification-based control mechanism: Result retrieve HIT expired HIT completion Stage completion 18

Conclusion Complex tasks do not work if they are not been defined exactly from the beginning! The CroudForge framework helps this problem by dynamically predefine subtasks by the market itself. 19

Future work GUI improvement for more complex tasks Framework progression for other kinds of task markets 20

The end Thank you for your time. 21