Mechanical Turk Lessons: CardLasso

1
Mechanical Turk LessonsCardLasso
John Barnes, CTO August 18th, 2009
2
Model Metrics Overview

• Founded in 2003
• Over 80 employees worldwide
• Headquartered in Chicago with offices in
  Minneapolis, Detroit, San Francisco, New York,
  Dallas and Southern California
• Laser focus on Cloud Computing
• More than 1,000 successful Cloud Computing
  implementations
• Amazon Web Service Charter Partner
• Salesforce Premier Implementation Partner

3
CardLasso What Is It?

• Cloud Based product that transcribes Business
  Cards
• Available for the iPhone and works with a Web Cam
• Mechanical Turk is used for the transcription of
  the card data

4
Original HIT Process
5
Problems with First Process

• Issue
• People were submitting pictures that either were
  not business cards or were not readable
• Results
• Customers were getting no results or bad results
  from Transcription
• MTurk Workers were frustrated and were not set up
  for success
• Solution
• Add another HIT to the process

6
New HIT Process
7
Results with the Second Process

• MTurk Workers
• HITs are now optimized for workers to do either
  Validation or Transcription
• Workers are more efficient and work on the tasks
  they prefer
• CardLasso Customers
• Now get immediate feedback if they need to
  resubmit an unreadable card
• Increased transcription accuracy
• Model Metrics
• MTurk Workers are policing each other
• Very scalable, no internal intervention required

8
Mechanical Turk Lessons Learned

• Start with small manually created HITs to
  confirm the approach and instructions
• Pay close attention to feedback from the Workers
• Iterate with small batches to confirm the
  instructions are clear and the process works
• Only move to API/External Questions once the
  process is refined
• Introduce yourself on TurkerNation
• Strongly consider a multiple-step HIT process
  up-front (we didnt and had to add one)
• Track Worker results and use Qualifications