Running a User Study

1
Running a User Study

• Alfred Kobsa
• University of California, Irvine

2
Pilot Test

• A pilot test is extremely valuable for detecting
  problems in the planned user test.
• These problems may be harmless, but they may also
  render test results unusable.
• 2-5 pilot subjects are sufficient.
• Some of them should be people who are very much
  willing to criticize the experiment (e.g., good
  friends, colleagues).
• At least one of them should be a real test
  subject.
• Pilot test should be run two days before the
  actual study at the latest (and there are many
  good reasons to do them far earlier).
• Pilot test should include all procedures.

3
Problems that are frequentlyuncovered in pilot
tests

• Subjects take considerably longer/shorter than
  expected.
• Task instructions are not well understood by
  subjects
• Subjects cannot carry out a task since some
  necessary functionality has not been explained to
  them
• Subjects use procedures for solving a task that
  are different from the one that the experimenters
  wanted to test
• Subjects overwrite each others data files
• Experimenters overwrite data of previous
  experiments
• Software crashes
• Experimenters forget to do certain things
• Some materials are missing
• Users have troubles finding parking, the building
  entrance, the usability lab, etc.

4
Welcome, Briefing,Instruction and Training

• Welcome
• Make participants feel comfortable / reduce
  anxiety
• Bridge time until everyone has arrived
• Let them show their ID (for security/tax
  purposes), and announce this beforehand
• Ask them to switch their cell phones off.
• Briefing
• Inform about purpose of the experiment (as far as
  this is possible)
• Emphasize that it will help develop a better
  product
• Encourage criticism and praise
• Emphasize that the product is being tested and
  not the user
• Emphasize that the people they will see are not
  the developers
• Show them the lab (or movies or pictures of it)
• Have them sign legal forms
• Let them fill in a pre-questionnaire (e.g.,
  demographics, pre-test)
• Instruction and Training
• Instruction through video, or through instructor
  who follows a written script
• Subsequent training tasks allow users to practice
  what they learned and to understand it better.
• Guided training tasks task description contains
  (partial) instructions on how to carry out a
  task, which subjects are asked to follow

5
Conducting the Test

• Setup
• (Bring subjects to test computers)
• Verify initial settings and materials, start
  recording devices.
• Ask them to begin (and to let you know when they
  are done)
• Enter users in logbook/timesheet (e.g., who used
  which computer)
• During the test
• Inform users about timeouts
• Answer questions generically avoid biasing
  participants
• If applicable, remind them to think aloud if they
  become silent
• Watch test users and take notes, or monitor users
  from a distance
• Record unusual occurrences in logbook/timesheet
• If applicable remind users occasionally to think
  aloud
• If applicable make sure users dont influence
  each other
• After the test
• Post-questionnaire (e.g., opinions on software,
  test procedures post-test)
• Payment, escort
• Prepare test station for new test subjects

6
At any time
Be organized! Follow checklists!
7
Unexpected events

• A participant does not arrive in time
• A participant forgot(?) his or her ID
• A participant refuses to sign the informed
  consent form or the non-disclosure agreement
• A participant refuses to be videotaped
• A participant does not want to switch off his/her
  cell phone
• A participant is called away in the middle of the
  test
• A participants cell phone rings continuously
• A participant does not have the required
  qualifications
• A participant becomes confrontational with other
  participants or with the experimenters
• A participant takes over the group
• Software freezes, computer breaks down, etc.
• Outside interference (construction noise,
  vacuums, )