Chapter 12/13: Evaluation/Decide Framework

1
Chapter 12/13 Evaluation/Decide Framework
2
Why Evaluate?

• Why to check that users can use the product and
  that they like it.
• Designers need to check that they understand
  users requirements.
• Avoid wasting money

3
What to Evaluate

• What a conceptual model, early prototypes of a
  new system and later, more complete prototypes.

4
Where to Evaluate

• Where in natural and laboratory settings.

5
When to Evaluate

• When throughout design finished products can be
  evaluated to collect information to inform new
  products.

6
The language of evaluation

• Analytical evaluation
• Controlled experiment
• Formative evaluation
• Heuristic evaluation
• Predictive evaluation

• Summative evaluation
• Usability laboratory
• User testing

7
Evaluation approaches

• Usability testing
• Field studies
• Analytical evaluation
• Combining approaches
• Opportunistic evaluations

8
Characteristics of approaches
Usability testing Field studies Analytical
Users do task natural not involved
Location controlled natural anywhere
When prototype early prototype
Data quantitative qualitative problems
Feed back measures errors descriptions problems
9
Evaluation approaches and methods
Method Usability testing Field studies Analytical
Observing x x
Asking users x x
Asking experts x x
Testing x
Modeling x
10
Six evaluation case studies

• Evaluating early design ideas for a mobile device
  for rural nurses in India.
• Evaluating cell phones for different markets.
• Evaluating affective issues challenge and
  engagement in a collaborative immersive game.
• Improving a design the Hutch World patient
  support system.
• Multiple methods help ensure good usability the
  Olympic messaging system (OMS).
• Evaluating a new kind of interaction an ambient
  system.

11
Team Extra Credit

• Book Exercise in Chapter 12(Interaction Design
  Book). Due November 1st in class.

12
DECIDE a framework to guide evaluation

• Determine the goals.
• Explore the questions.
• Choose the evaluation approach and methods.
• Identify the practical issues.
• Decide how to deal with the ethical issues.
• Evaluate, analyze, interpret and present the data.

13
Determine the goals

• What are the high-level goals of the evaluation?
• Who wants it and why?
• Some examples of goals
• Check to ensure that the final interface is
  consistent.
• Investigate how technology affects working
  practices.
• Improve the usability of an existing product .

14
Explore the questions

• All evaluations need goals questions to guide
  them.
• What questions might you ask about the design of
  a cell phone?

15
Choose the evaluation approach methods

• The evaluation approach influences the methods
  used, and in turn, how data is collected,
  analyzed and presented.
• E.g. field studies typically
• Involve observation and interviews.
• Do not involve controlled tests in a laboratory.
• Produce qualitative data.

16
Identify practical issues

• For example, how to
• Select users
• Stay on budget
• Stay on schedule
• Find evaluators
• Select equipment

17
Decide about ethical issues

• Develop an informed consent form
• Participants have a right to- Know the goals of
  the study- Know what will happen to the
  findings- Privacy of personal information-
  Leave when they wish - Be treated politely.

18
Evaluate, interpret present data

• The approach and methods used influence how data
  is evaluated, interpreted and presented.
• The following need to be considered-
  Reliability can the study be replicated?-
  Validity is it measuring what you expected?-
  Biases is the process creating biases?- Scope
  can the findings be generalized?- Ecological
  validity is the environment influencing the
  findings? - i.e. Hawthorn effect.