Predictive Evaluation

1
Predictive Evaluation

• Overview
• Heuristic Evaluation
• Discount Usability Testing

This material has been developed by Georgia Tech
HCI faculty, and continues to evolve.
Contributors include Gregory Abowd, Jim Foley,
Diane Gromala, Elizabeth Mynatt, Jeff Pierce,
Colin Potts, Chris Shaw, John Stasko, and Bruce
Walker. Comments directed to foley_at_cc.gatech.edu
are encouraged. Permission is granted to use with
acknowledgement for non-profit purposes. Last
revision May 2004.
2
Evaluation

• Evaluation in general
• Gathering data about usability of a design by a
  specified group of users for a particular
  activity within a specified environment
• Goals
• 1. Assess extent of systems functionality
• 2. Assess effect of interface on user
• 3. Identify specific problems with system

3
Types

• Formative
• Help form the result. All through the lifecycle.
  Early, continuous. Iterative.
• Summative
• After a system has been finished. Make judgments
  about final result. Sum things up.

4
Predictive Evaluation

• Why do it
• Observing users can be time-consuming and
  expensive
• Try to predict usability rather than observing it
  directly
• Conserve resources (quick low cost)

5
Approach

• Expert reviewers often used
• HCI experts interact with system and try to find
  potential problems and give prescriptive feedback
• Best if
• Havent used earlier prototype
• Familiar with domain or task
• Understand user perspectives

6
Predictive Evaluation Methods

• Heuristic Evaluation
• Discount usability testing
• Cognitive Walkthrough
• Separate presentation
• User Modeling
• Separate presentation
• Hicks, Fitts KSLM, GOMS

7
1. Heuristic Evaluation

• Developed by Jakob Nielsen
• Several expert usability evaluators assess system
  based on simple and general heuristics
  (principles or rules of thumb)

(Web site www.useit.com)
8
HE Procedure

• Gather inputs
• Evaluate system against heuristics
• Organize and debrief
• Severity rating

9
HE 1 Gather Inputs

• Who are evaluators?
• Several usability experts
• Need to learn about domain, its practices
• Get the prototype to be studied
• May vary from mock-ups and storyboards to a
  working system

10
HE 2 Evaluate System against Heuristic Design
Guidelines

• Reviewers evaluate system based on high-level
  heuristics

Use simple and natural dialog Provide
clearly marked exits Speak users language
Provide shortcuts Minimize memory load
Provide good error messages Be consistent
Prevent errors Provide feedback
11
HE 2 Evaluate System

• Emphasis on

Recognition rather than recall Flexibility
and efficiency of use Recognition, diagnosis
and recovery from errors Help and
documentation Match between system and real
world
Visibility of system status Aesthetic and
minimalist design User control and
freedom Consistency and standards Error
prevention
12
HE 2 Evaluate

• Perform two or more passes through system
  inspecting
• Flow from screen to screen
• Each screen
• Evaluate against heuristics
• Find problems
• Subjective (if you think it is, it is)
• Dont dwell on whether it is or isnt
• Each reviewer works independently

13
HE 3 Organize and Debrief

• Organize all problems found by different
  reviewers
• At this point, decide what are and arent
  problems
• Structure the problems by categories
• Document and record

14
HE 4 Severity Rating

• 0-4 rating scale
• Based on
• Frequency
• User Ompact
• Persistence
• Market impact

15
HE Advantage

• Cheap, good for small companies who cant afford
  more
• Getting someone practiced in method is valuable

16
HE Application

• Nielsen fiveevaluators found75 of the
  problems
• Above that you find more, but at decreasing
  efficiency

17
Cost/Benefit

• 1994 case study showed savings of 48 times cost
  of H.E.
• Your results may vary

18
HE Somewhat Controversial

• Very subjective assessment of problems
• Depends of expertise of reviewers
• Why are these the right heuristics?
• Others have been suggested
• How to determine what is a true usability problem
• Some recent papers suggest that many identified
  problems really arent
• Bottom line - useful!

19
2. Discount Usability Testing

• Hybrid of empirical usability testing and
  heuristic evaluation
• Have 2 or 3 think-aloud user sessions with paper
  or prototype-produced mock-ups

20
Discount Usability in Action

• Mockups are not supposed to be perfect!
• A variety of approaches for mockups
• Must be quick to create economical in use of
  resources
• Sketches most common
• Paper has its limitations tends to focus on the
  visual elements
• Sometimes awkward to use in usability testing

21
Remember

• Formative evaluation
• Predictive evaluation
• Heuristic Evaluation
• Discount usability testing
• Cognitive Walkthrough
• User Modeling
• Summative evaluation

22
The End