Selecting, recruiting and informing users

1
Selecting, recruiting and informing users

• Alfred Kobsa
• University of California, Irvine

2
Recruiting users

• How many?
• Using 5-8 users for a set of tasks will already
  reveal most major UI problems related to those
  tasks
• Use 3-5 users for each subgroup (? potentially
  exponential growth of subject size)
• Considerably more if statistically significant
  differences are at issue
• Inhouse or external?
• Inhouse confidentiality preserved, good
  motivation, inexpensive(?), fewer logistics, can
  be rushed
• External users often more representative,
  neutral
• Recruitment channels
• announcements (posters, flyers, online bulletin
  boards, online classified ads, Internet forums,
  email, ), word-of-mouth, crowdsourcing,
  recruiting agencies, sales / distribution
  channels (?)

3
Selecting and scheduling users

• Screener set of criteria to help filter
  participants
• Positive criteria specific demographics, skills
• Go for facts, avoid self-assessment
• Negative criteria contacts to competitors, odd
  people, professional test users
• Screen potential participants on first contact
  (preferably via phone or in person, 10-15 mins)
• Send out invitation in writing, and an email
  reminder on the day before (with directions,
  phone number for last-minute changes)
• Over-recruit (?), or better have backup
• Create an inhouse database of potential subjects
  (also for tax purposes)

4
Participant incentives

• Prime motivation to participate should be
• being interested in your product (but not too
  much)
• being curious about user studies
• feeling flattered
• wanting to help develop a better product
• wanting to help you and others
• Incentives can also be a motivator, but generally
  should not be the only one
• Freebies, gift coupons, cash, charitable
  donation,
• Thoughtful selection can be very important
• Incentive should not be too high
• Give everyone the same amount, or make sure that
  participants are not aware of discrimination

5
Legal requirements

• Company policies often require subjects to sign
  liability waivers, video release forms,
  non-disclosure agreements,
• Federal agencies and colleges receiving federal
  fundingFederal regulations for the protection
  of human subjects may require informed consent of
  subjects if scientific research is performed
  (whose purpose is to enhance human knowledge, and
  not only to improve a product).
• Codes of conducts of professional societies (such
  as the American Psychological Association) may
  regulate how their members have to carry out
  human subjects studies.

6
Informed consent

• Purpose (in general terms deception permitted,
  specifically if disclosure at the end of
  experiment)
• Procedures
• Risks
• Benefits, Incentive/Compensation
• Confidentiality
• Right to withdraw
• Consent to being video/audiotaped
• Retention and secondary use of data
• Point of contact