Usability Testing

1
Usability Testing
2

• Reliability
• Repeatable results
• Vs.
• Validity
• Do results reflect usability issues one wants to
  test
• Wrong users, wrong tasks

3
Evaluation

• Informal
• go play around with this thing and tell us what
  to fix
• Formative
• as developed
• Summative
• after product finished

4
Formative Evaluation

• All thru project
• Use think aloud
• 3 major cycles followed by iterative redesign
• Get most data from the first major cycle
• Fewer newer discoveries later

5
Summative Evaluation

• Good for comparing 2 products
• Performed once
• If you wait, may be too late
• Better for comparison of two systems

6
Usability Testing

• Plan and prepare
• Conduct the test
• Collect data
• Analyze data
• Draw conclusions
• Document results
• Lather, rinse, repeat

7
1. Plan and Prepare

• Decide on ease of learning/ease of use focus for
  the test
• Decide on user and task focus for the test
• Design test tasks
• identify usability specification attributes and
  levels
• develop benchmark and representative tasks
• what the user should do
• should be quantifiable (measurable)

8

• Design the test and develop test materials
• Protocols and procedures
• Scripts, instruments, props
• Design and assemble the test environment
• Recruit/schedule pilot test users (1-2)

9

• Run pilot test
• How long does it take?
• Revise test procedures and materials
• Recruit/schedule test users (3-5)

10
2. Conduct the tests

• Run the test
• Welcome user
• Have user sign consent form
• Get pretest info (interview)
• Read intro script to user
• Give user written list of tasks
• Decide how to handle anomalies
• What happens if people vary
• Experimenter refrains from interfacing with user
  do NOT make comments
• Refrain from helping user be cautious

11
Data
12
3. Collect the data

• Quantitative data
• Benchmark tests
• Time to complete tasks
• Number of tasks in set time
• Number of errors
• User preference scales, rankings, ratings

13

• Qualitative
• Concurrent verbal protocol - think aloud
• Constructive interaction 2 users, discovery
• Retrospective protocol testing and debriefing
• Critical incident
• positive - light bulb
• negative - error block
• Coaching method good for novice

14

• Structural interview
• Debrief user
• Questionnaire
• Preplanned questions
• what did you like best?
• what would you change?
• Collection techniques
• note taking
• collection sheets
• video, audio
• built-in instruments

15

• Worry about
• reactivity with subjects
• halo/pitch fork effect

16
4. Analyze data

• Summarize data
• Interpret data

17

• 5. Draw conclusions and formulate recommended
  design changes
• 6. Document/present results
• 7. Repeat from step 1