Running a User Study

1
Running a User Study

• Alfred Kobsa
• University of California, Irvine

2
Pilot Test

• A pilot test is extremely valuable for detecting
  problems in the planned user test.
• These problems may be harmless, but they may also
  obstruct the user test or render its results
  unusable.
• 2-5 pilot subjects are sufficient.
• Some of them should be people who are very much
  willing to criticize the experiment (e.g., good
  friends, colleagues).
• At least one of them should be a real test
  subject.
• Pilot test should be run two days before the
  actual study at the latest (and there are many
  good reasons to do them far earlier).
• Pilot test should include all procedures.

3
Problems that are frequentlyuncovered in pilot
tests

• Subjects take considerably longer/shorter than
  expected.
• Task instructions are not well understood by
  subjects
• Subjects cannot carry out a task since some
  necessary functionality has not been explained to
  them
• Subjects use procedures for solving a task that
  are different from the one that the experimenters
  wanted to test
• Subjects overwrite each othersdata files
• Experimenters overwrite data of previous
  experiments
• Software crashes
• Experimenters forget to do certain things
• Some materials are missing
• Users have troubles finding parking, the building
  entrance, the usability lab, etc.

4
Welcome, Briefing,Instruction and Training

• Welcome
• Make participants feel comfortable / reduce
  anxiety
• Bridge time until everyone has arrived
• Let them show their ID (for security/tax
  purposes), and announce this beforehand
• Ask them to switch off their cell phones.
• Briefing
• Inform about purpose of the experiment (as far as
  this is possible)
• Emphasize that it will help develop a better
  product
• Encourage criticism and praise
• Emphasize that the product is being tested and
  not they themselves
• Emphasize that the people they will see are not
  the developers
• Show them the lab (or movies or pictures of it)
• Have them sign legal forms
• Let them fill in a pre-questionnaire (e.g.,
  demographics, pre-test)
• Instruction and Training
• Instruction through video, Powerpoint, or
  instructor who follows a written script
• Subsequent training tasks allow users to practice
  what they learned and to understand it better.
• Guided training tasks task description contains
  (partial) instructions on how to carry out a
  task, which subjects are asked to follow

5
Conducting the Test

• Setup
• (Bring subjects to test computers)
• Verify initial settings and materials, start
  recording devices.
• (Instruct them to think aloud this makes task
  completion times unreliable)
• Ask them to begin (and to let you know when they
  are done)
• Enter users in logbook/timesheet (e.g., who used
  which computer)
• During the test
• Watch test users and take notes, or monitor users
  from a distance
• Record unusual occurrences in logbook/timesheet
• Answer questions generically avoid biasing
  participants
• Inform users about timeouts
• If applicable, remind them to think aloud if they
  become silent
• If applicable make sure that users dont
  influence each other
• After the test
• Post-questionnaire (e.g., opinions on software
  and on test procedures post-test)
• Payment, call escort
• Prepare test station for new test subjects

6
At any time
Be organized! Follow checklists!
7
Unexpected events

• A participant
• does not arrive on time
• says he forgot his ID
• refuses to sign the informed consent or
  non-disclosure form
• refuses to be videotaped
• does not want to switch off his/her cell phone
• is called away in the middle of the test
• has a cell phone on that rings continuously
• does not have the required qualifications
• exceeds the time limit for a task without
  finishing it
• struggles mightily with a task
• becomes confrontational w/ other subjects or the
  experimenters
• takes over the group
• Software freezes, computer breaks down, etc.
• Outside interference (construction noise,
  vacuums, )