Utility data annotation via Amazon Mechanical Turk Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign X = $5000

Motivation Unlabeled data is free Labels are useful We need large volumes of labeled data Different labeling needs: Is there X in the image? Outline X. Where is part Y of X. Of these 500 images, which belong to category X? ……………. and many more ……………….

Task Amazon Mechanical Turk Is this a dog? o Yes o No Workers Answer: Yes Task: Dog? Pay: $0.01 Broker $0.01

Motivation X = $5000 Custom annotations Large scaleLow price

Annotation protocols Type keywords Select relevant images Click on landmarks Outline something Detect features ……….. anything else ………

Type keywords $0.01

Select examples Joint work with Tamara and Alex Berg

Select examples requester mtlabel $0.02

Click on landmarks $0.01

Outline something $ Data from Ramanan NIPS06

Detect features Measuring molecules. Joint work with Rebecca Schulman (Caltech) ?? $0.1

Motivation X = $5000 Custom annotations Large scaleLow price

Issues Quality? –How good is it? –How to be sure? Price? –How to price it? How does MTurk compare with others? How do I sign up?

Annotation quality Agree within 5-10 pixels on 500x500 screen There are bad ones. ACEG

Grading tasks Take 10 submitted results Create new task to verify the result Verification is easy –Pay the same or slightly higher price Total overhead - 10% (work in progress)

Price $0.01 per image (16 clicks) ~ $1500 / images >1000 images per day <4 months Workers suggested $ $0.05/img –$ $5500 / images

Is the price right? $0.01/ 40 clicks 15 hours 900 labels $0.01 / 14 clicks 1.6 hours 900 labels $0.01 / 16 clicks 4 hours 900 labels

Annotation Method Comparison ApproachCostScaleSetup effort CentralizedQualityElastic to $ MTurk$+++*no+/ LabelME++Yes++ ImageParsing.com$$++**Yes Games with purpose (ESP++) ++++***Yes++ In house$$$+*no+++++

How do I sign up? Go to our web page: Send us an Register at Amazon Mechanical Turk

Acknowledgments Special thanks to: David Forsyth Tamara Berg Rebecca Schulman David Martin Kobus Barnard Mert Dikmen All workers at Amazon Mechanical Turk This work was funded in part by ONR

Thank you X = $5000