Evaluation Overview

1
Evaluation Overview
2
Agenda

• Questions
• Intro to evaluation
• Forms of evaluation
• Questionnaire design

3
Why evaluation is done?

• Summative
• assess an existing system
• judge if it meets some criteria
• Formative
• assess a system being designed
• gather input to inform design

• Summative or formative?
• Depends on
• maturity of system
• how evaluation results will be used
• Same technique can be used for either

4
Other distinctions

• Form of results of obtained
• Quantitative
• Qualitative
• Who is experimenting with the design
• End users
• HCI experts

• Approach
• Experimental
• Naturalistic
• Predictive

5
Experimental Approach

• Lab studies, quantitative results
• Typically in a closed, lab setting
• Manipulate independent variables to see effect on
  dependent variables
• Replicable
• Expensive, requires real users and lab

6
Naturalistic Approach

• Field studies (qualitative results)
• Observation occurs in real life setting
• Watch process over time
• Ecologically valid
• Cheap, quick, less training required
• Not reproducible user-specific results
• Not quantitative (how much better is it?)

7
Predictive Evaluation

• Try to predict usage before real users are
  involved
• Conserve resources (quick low cost)
• Expert reviewers often used
• HCI experts (not real users) interact with
  system, find potential problems, and give
  prescriptive feedback
• Best if they
• Havent used earlier prototype
• Familiar with domain or task
• Understand user perspectives

8
Making an evaluation plan

• What criteria are important?
• What resources available?
• evaluators, prototype, subjects
• Required authenticity of system

9
Evaluation techniques

• Predictive modeling
• Questionnaire
• Empirical user studies
• Heuristic evaluation
• Cognitive walkthrough
• Think aloud (protocol analysis)
• Interviews
• Experience Sampling
• Focus Groups

10
Questionnaire design

• Summative or formative
• Quantitative or qualitative
• Usually inexpensive way to get lots of information

11
Questionnaires

• Preparation is expensive, but administration is
  cheap
• Oral vs. written
• Oral advs Can ask follow-up questions
• Oral disadvs Costly, time-consuming
• Forms can provide better quantitative data

12
Questionnaire design

• Take your own survey first
• Know what answers you are trying to elicit
• Too long, and youll be sorry

13
Questionnaires

• Issues
• Only as good as questions you ask
• Establish purpose of questionnaire
• Dont ask things that you will not use
• Who is your audience?
• How do you deliver and collect questionnaire?

14
Questionnaire Topic

• Can gather demographic data and data about the
  interface being studied
• Demographic data
• Age, gender
• Task expertise
• Motivation
• Frequency of use
• Education/literacy

15
Interface Data

• Can gather data about
• screen
• graphic design
• terminology
• capabilities
• learning
• overall impression
• ...

16
Question Format

• Closed format
• Answer restricted to a set of choices
• Typically very quantifiable
• Variety of styles

17
Closed Format

• Likert Scale
• Typical scale uses 5, 7 or 9 choices
• Above that is hard to discern
• Doing an odd number gives the neutral choice in
  the middle

Characters on screen hard to read
easy to read 1
2 3 4 5 6
7
18
Other Styles
Rank from 1 - Very helpful 2 - Ambivalent 3 - Not
helpful 0 - Unused
Which word processingsystems do you use?
LaTeX
Word
___ Tutorial ___ On-line help ___ Documentation
FrameMaker
WordPerfect
19
Closed Format

• Advantages
• Clarify alternatives
• Easily quantifiable
• Eliminate useless answer

• Disadvantages
• Must cover whole range
• All should be equally likely
• Dont get interesting, different reactions

20
Open Format

• Asks for unprompted opinions
• Good for general, subjective information, but
  difficult to analyze rigorously
• May help with design ideas
• Can you suggest improvements to this interface?

21
Questionnaire Issues

• Question specificity
• Do you have a computer?
• Language
• Beware terminology, jargon
• Clarity
• Leading questions
• Can be phrased either positive or negative

22
Questionnaire Issues

• Prestige bias
• People answer a certain way because they want you
  to think that way about them
• Embarrassing questions
• Hypothetical questions

23
Deployment

• Steps
• Discuss questions among team
• Administer verbally/written to a few people
  (pilot). Verbally query about thoughts on
  questions
• Administer final test

24
Examples

• See Web