Evaluation Methods

1
Evaluation Methods

• Human-Computer Interaction
• Jenny Preece et al.

2
The HCI design model

•  HCI design should
• be user-centred and involve users as much as
  possible
• integrate knowledge and expertise from different
  disciplines
• be highly iterative so that testing can be done
  to check that the design does indeed meet user
  requirements

3
The star life cycle (Hix Hartson)
4
The Hawthorne effect

• Observation can change behaviour
• Western Electric at Hawthorne, Illinois (1939)
• Also
• The record is always incomplete

5
Preeces classification of evaluation methods

• Usage data
• Experiments and benchmarking
• Interpretive evaluation
• Predictive evaluation

6
Usage data

• Observing users
• Direct and indirect observation
• Verbal protocols
• Thinking aloud method
• Software logging
• Time-stamped keypresses interaction logging
• Interviews
• Questionnaires

7
Questionnaire on User Interaction Satisfaction
(QUIS)
8
Experiments

• Traditional experiments
• Usability engineering

9
User-based evaluation process
10
Usability lab
11
Usability Engineering

• The process of usability engineering
• Defining usability goals through metrics
• Setting planned levels of usability that need to
  be achieved.
• Analysing the impact of possible design
  solutions.
• Incorporating user-derived feedback in product
  design.
• Iterating through the design-evaluate-design
  loop until planned levels are achieved.

12
Interpretive Evaluation

• Contextual inquiry
• Cooperative and participative evaluation
• Ethnography

13
Contextual Inquiry

• Users and researchers participate to identify and
  understand usability problems within the normal
  working environment of the user.
•  Makes use of the contextual interview.
•  Recommendations to evaluator
• Get as close to work as possible
• Uncover work practice hidden in words
• Create interpretations with customers
• Let customers expand the scope of the discussion

14
Ethnography

• a holistic interpretation of a groups culture
• Blomberg et al. (1993) highlight four main
  principles that guide much ethnographic work
• Ethnography is grounded in fieldwork - people are
  studied in their natural settings.
• To understand the influence of context on
  peoples activities one must take a holistic
  perspective.
• Ethnographers build up a descriptive account of
  how people behave, not how they ought to behave.
• Importance is given to understanding things from
  the point-of-view of those studied.

15
Predictive Evaluation

• Inspection methods
• Usage simulations
• Structured expert reviewing
• Modelling e.g. KLM
• Do not directly involve users

16
Structured expert reviewing

• Example Heuristic evaluation
• Expert reviewers examine the system guided by a
  set of high-level heuristics.
• A1 Match structure with user tasks.
• Does the structure of the website enable the
  users to easily achieve their tasks?
• 0 No problem 4 Usability catastrophe

17
Keystroke Level Model (KLM)

• Keystrokes (K)
• Pointing (P)
• Homing (H)
• Drawing (D)
• Thinking (M)
• Response (R)
• Example Enter the command put from the keyboard
  and start its execution by striking the Enter
  key.
• MKpKuKtKEnter, or in shorthand
  M4KputEnter

18
Differences between methods

• Factors
• Purpose of the evaluation
• Stage of system development
• Involvement of users
• Type of data