1. Mechanical Turk Lessons: CardLasso
John Barnes, CTO
August 18th, 2009

2. Model Metrics Overview
Founded in 2003
Over 80 employees worldwide
Headquartered in Chicago with offices in Minneapolis, Detroit, San Francisco, New York, Dallas and Southern California
Laser focus on Cloud Computing
More than 1,000 successful Cloud Computing implementations
Amazon Web Service Charter Partner
Salesforce Premier Implementation Partner 2

3. CardLasso – What Is It?
Cloud Based product that transcribes Business Cards
Available for the iPhone and works with a Web Cam
Mechanical Turk is used for the transcription of the card data

4. Original HIT Process

5. Problems with First Process
Issue
People were submitting pictures that either were not business cards or were not readable
Results
Customers were getting no results or bad results from Transcription
MTurk Workers were frustrated and were not set up for success
Solution
Add another HIT to the process

6. New HIT Process

7. Results with the Second Process
MTurk Workers
HIT’s are now optimized for workers to do either Validation or Transcription
Workers are more efficient and work on the tasks they prefer
CardLasso Customers
Now get immediate feedback if they need to resubmit an unreadable card
Increased transcription accuracy
Model Metrics
MTurk Workers are policing each other
Very scalable, no internal intervention required

8. Mechanical Turk: Lessons Learned
Start with small manually created HIT’s to confirm the approach and instructions
Pay close attention to feedback from the Workers
Iterate with small batches to confirm the instructions are clear and the process works
Only move to API/External Questions once the process is refined
Introduce yourself on TurkerNation
Strongly consider a multiple-step HIT process up-front (we didn’t and had to add one)
Track Worker results and use Qualifications