An%20evaluation%20framework

1
An evaluation framework
2
The aims

• Explain key evaluation concepts terms.
• Describe the evaluation paradigms techniques
  used in interaction design.
• Discuss the conceptual, practical and ethical
  issues that must be considered when planning
  evaluations.
• Introduce the DECIDE framework.

3
Evaluation paradigm

• Any kind of evaluation is guided explicitly or
  implicitly by a set of beliefs, which are often
  under-pined by theory. These beliefs and the
  methods associated with them are known as an
  evaluation paradigm

4
User studies

• User studies involve looking at how people behave
  in their natural environments, or in the
  laboratory, both with old technologies and with
  new ones.

5
Four evaluation paradigms

• quick and dirty
• usability testing
• field studies
• predictive evaluation

6
Quick and dirty

• quick dirty evaluation describes the common
  practice in which designers informally get
  feedback from users or consultants to confirm
  that their ideas are in-line with users needs
  and are liked.
• Quick dirty evaluations are done any time.
• The emphasis is on fast input to the design
  process rather than carefully documented
  findings.

7
Usability testing

• Usability testing involves recording typical
  users performance on typical tasks in controlled
  settings. Field observations may also be used.
• As the users perform these tasks they are watched
  recorded on video their key presses are
  logged.
• This data is used to calculate performance times,
  identify errors help explain why the users did
  what they did.
• User satisfaction questionnaires interviews are
  used to elicit users opinions.

8
Field studies

• Field studies are done in natural settings
• The aim is to understand what users do naturally
  and how technology impacts them.
• In product design field studies can be used to-
  identify opportunities for new technology-
  determine design requirements - decide how best
  to introduce new technology- evaluate technology
  in use.

9
Predictive evaluation

• Experts apply their knowledge of typical users,
  often guided by heuristics, to predict usability
  problems.
• Another approach involves theoretically based
  models.
• A key feature of predictive evaluation is that
  users need not be present
• Relatively quick inexpensive

10
Overview of techniques

• observing users,
• asking users their opinions,
• asking experts their opinions,
• testing users performance
• modeling users task performance

11
DECIDE A framework to guide evaluation

• Determine the goals the evaluation addresses.
• Explore the specific questions to be answered.
• Choose the evaluation paradigm and techniques to
  answer the questions.
• Identify the practical issues.
• Decide how to deal with the ethical issues.
• Evaluate, interpret and present the data.

12
Determine the goals

• What are the high-level goals of the evaluation?
  
• Who wants it and why?
• The goals influence the paradigm for the study
• Some examples of goals
• Identify the best metaphor on which to base the
  design.
• Check to ensure that the final interface is
  consistent.
• Investigate how technology affects working
  practices.
• Improve the usability of an existing product .

13
Explore the questions

• All evaluations need goals questions to guide
  them so time is not wasted on ill-defined
  studies.
• For example, the goal of finding out why many
  customers prefer to purchase paper airline
  tickets rather than e-tickets can be broken down
  into sub-questions- What are customers
  attitudes to these new tickets? - Are they
  concerned about security?- Is the interface for
  obtaining them poor?
• What questions might you ask about the design of
  a cell phone?

14
Choose the evaluation paradigm techniques

• The evaluation paradigm strongly influences the
  techniques used, how data is analyzed and
  presented.
• E.g. field studies do not involve testing or
  modeling

15
Identify practical issues

• For example, how to
• select users
• stay on budget
• staying on schedule
• find evaluators
• select equipment

16
Decide on ethical issues

• Develop an informed consent form
• Participants have a right to- know the goals of
  the study- what will happen to the findings-
  privacy of personal information- not to be
  quoted without their agreement - leave when they
  wish - be treated politely

17
Evaluate, interpret present data

• How data is analyzed presented depends on the
  paradigm and techniques used.
• The following also need to be considered-
  Reliability can the study be replicated?-
  Validity is it measuring what you thought?-
  Biases is the process creating biases?- Scope
  can the findings be generalized?- Ecological
  validity is the environment of the study
  influencing it - e.g. Hawthorn effect

18
Pilot studies

• A small trial run of the main study.
• The aim is to make sure your plan is viable.
• Pilot studies check- that you can conduct the
  procedure- that interview scripts,
  questionnaires, experiments, etc. work
  appropriately
• Its worth doing several to iron out problems
  before doing the main study.
• Ask colleagues if you cant spare real users.

19
Key points

• An evaluation paradigm is an approach that is
  influenced by particular theories and
  philosophies.
• Five categories of techniques were identified
  observing users, asking users, asking experts,
  user testing, modeling users.
• The DECIDE framework has six parts - Determine
  the overall goals - Explore the questions that
  satisfy the goals - Choose the paradigm and
  techniques - Identify the practical issues -
  Decide on the ethical issues - Evaluate ways to
  analyze present data
• Do a pilot study

20
Observing users
21
The aims

• Discuss the benefits challenges of different
  types of observation.
• Describe how to observe as an on-looker, a
  participant, an ethnographer.
• Discuss how to collect, analyze present
  observational data.
• Examine think-aloud, diary studies logging.
• Provide you with experience in doing observation
  and critiquing observation studies.

22
What and when to observe

• Goals questions determine the paradigms and
  techniques used.
• Observation is valuable any time during design.
• Quick dirty observations early in design
• Observation can be done in the field (i.e., field
  studies) and in controlled environments (i.e.,
  usability studies)
• Observers can be- outsiders looking on-
  participants, i.e., participant observers-
  ethnographers

23
Frameworks to guide observation

• - The person. Who? - The place. Where?- The
  thing. What?
• The Goetz and LeCompte (1984) framework- Who
  is present? - What is their role? - What is
  happening? - When does the activity occur?-
  Where is it happening? - Why is it happening? -
  How is the activity organized?

24
The Robinson (1993) framework

• Space. What is the physical space like?
• Actors. Who is involved?
• Activities. What are they doing?
• Objects. What objects are present?
• Acts. What are individuals doing?
• Events. What kind of event is it?
• Goals. What do they to accomplish?
• Feelings. What is the mood of the group and of
  individuals?

25
You need to consider

• Goals questions
• Which framework techniques
• How to collect data
• Which equipment to use
• How to gain acceptance
• How to handle sensitive issues
• Whether and how to involve informants
• How to analyze the data
• Whether to triangulate

26
Observing as an outsider

• As in usability testing
• More objective than participant observation
• In usability lab equipment is in place
• Recording is continuous
• Analysis observation almost simultaneous
• Care needed to avoid drowning in data
• Analysis can be coarse or fine grained
• Video clips can be powerful for telling story

27
Participant observation ethnography

• Debate about differences
• Participant observation is key component of
  ethnography
• Must get co-operation of people observed
• Informants are useful
• Data analysis is continuous
• Interpretivist technique
• Questions get refined as understanding grows
• Reports usually contain examples

28
Data collection techniques

• Notes still camera
• Audio still camera
• Video
• Tracking users- diaries- interaction logging

29
Data analysis

• Qualitative data - interpreted used to tell the
  story about what was observed.
• Qualitative data - categorized using techniques
  such as content analysis.
• Quantitative data - collected from interaction
  video logs. Presented as values, tables, charts,
  graphs and treated statistically.

30
Interpretive data analysis

• Look for key events that drive the groups
  activity
• Look for patterns of behavior
• Test data sources against each other -
  triangulate
• Report findings in a convincing and honest way
• Produce rich or thick descriptions
• Include quotes, pictures, and anecdotes
• Software tools can be useful e.g., NUDIST,
  Ethnograph (see URL resource list for examples)

31
Looking for patterns

• Critical incident analysis
• Content analysis
• Discourse analysis
• Quantitative analysis - i.e., statistics

32
Key points

• Observe from outside or as a participant
• Analyzing video and data logs can be
  time-consuming.
• In participant observation collections of
  comments, incidents, and artifacts are made.
  Ethnography is a philosophy with a set of
  techniques that include participant observation
  and interviews.
• Ethnographers immerse themselves in the culture
  that they study.