User testing: Hard performance measures in usability evaluation

1
User testingHard performance measures in
usability evaluation
Christof van Nimwegen
2
Week 19

• Maandag (College)
• Algemeen - Wat is een experiment?
• - Wat kunnen we meten
• - Methoden
• - Issues
• Dinsdag (College )
• 1. Als introductie op assignment 1 behandeling
  paradox of the active user guidance can be
  counterproductive
• 2. Instructies/uitleg assignment

3
Week 20

• Maandag (werkgroep in lab)
• 13-15 Werkgroep in Usability lab (CGN, C201)
• Groep 1 1300
• Groep 2 1330
• Groep 3 1400
• Groep 4 1430
• Dinsdag (practicum in lab t/m Donderdag max)
• - op een tijd die jullie past, in groepjes van 2
  experiment uitvoeren met zelf-geregelde
  proefpersonen

4
Dus.nu mee beginnen!

• Groep 2 6 proefpersonen
• Groep 3 4 proefpersonen
• Groep 4 6 proefpersonen
• Totaal 14 pp, medestudenten, of wie dan ook
  aanwezig op de
• uithof, niet elkaar uiteraard ?. Schedulen met
  mij,
• christof_at_cs.uu.nl of 06-41472701
• Lab-ruimte Di 1530
• Wo 9-18?
• Do 9-18?
• Vr indien nodig.

5
Hard performance measures the experiment
Empirical research is research that bases its
findings on direct or indirect observation as its
test of reality It generally involves the
collection of a large amount of data on which to
base a theory or derive a conclusion in science
6
What is an experiment

• a series of actions carried out to test a theory,
  demonstrate a fact or find out what happens
• the act of conducting a controlled test or
  investigation
• something one does deliberately to see what
  happens
• direct observation under controlled conditions.
  Most experiments involve carefully changing one
  variable and observing the effect on another
  variable (for example, changing temperature of a
  water sample and recording the change volume that
  results).

7
No experiment
8
Scientific experiment in a nutshell

• Demand from academic/industrial field (theory)
• Hypotheses/questions
• Define design, method procedure
• Do experiment
• Process results
• Analyze data
• Conclusions
• What to do now?

9
User testing (experiment) in UE

• is a central aspect of interaction design
• it often attempts to quantify "ease of use" and
  "user friendliness"
• not only on computers/web
• usually involves a controlled experiment to
  determine how well people can use the product
• can be done in different phases of product design
• is in practice often a combination of various
  techniques (quantitative qualitative data)
• Performance measurements
• Think aloud techniques
• Questionnaires
• .

10
User Testing

• Bestaat al heel lang..
• Niet alleen pure usability in de zin van
  gebruiksvriendelijkheid
• Voorbeelden
• - Bell
• - Xerox
• - Defensie

11
Usability evaluation by means of user testing

• Applied form of experiment
• Has a clearly defined purpose as opposed to
  fundamental research
• Should include statistical handling
• Subjects (amounts are important) as opposed to
  discount usability studies
• If you really want to know!(dont just ask, do
  it!)

12
Bekend voorbeeld UE with user testing
13
En deze.?
14
User Testing

• Global idea
• Users interact with a product/material to
  discover usability problems
• Many variations possible
• Number of users at the same time
• Interact with mock-up or final system
• Combinations with other methods
• Aim controlled studies (hypothesis testing)
• Example
• Comparison of two different sets of icons to be
  used at airports

15
Planning a usability test

• Define usability goals
• Define user profile, and participants
• Selecting and presentation of tasks
• How to measure usability
• Preparing test materials
• Conducting a pilot test
• ..

16
Making choices

• You can hardly test everything!
• More data does not mean better results
• Define/select/prepare material
• Select the tasks/function set
• Select the users
• Select the usability focus
• Arrange paerticipants
• Define amount of training and preparation for
  participants

17
A Usability lab
18
User Testing in a Lab Issues (just a few)

• Control
• Cost
• Naturalness (environment, setting)
• Registration (logging)
• Test-behavior (tendencies)
• Fear (they will look in my head)
• Participants (selection)
• Ethical issues
• The role of the experimenter
• everything!!

19
(No Transcript)
20
Japan
21
Material

• Paper prototypes
• Mockups
• Beta releases
• End product
• Wizard of Oz
• ..

22
Wizard of Oz?

• Enables unimplemented technology to be evaluated
  by using a human to simulate
• the response of a system. It can be used to test
  device concepts and techniques and
• suggested functionality before implementation
• Method
• The "wizard" observes the user's actions, and
  simulates the system's responses in
• real-time. Often users are unaware that the
  system was not real.

23
What do we have..

• variable -- (something that is likely to vary
  something that is subject to variation "the
  weather is one variable to be considered")
• Variables can be..
• Dependent Daar waarop gescoord wordt
  (hoeveel peren kan men eten?)
• Independent Wat we zelf aangebracht
  hebben (mannen en vrouwen)
• Between subjects (appels vs peren)
• Within subjects (het effect van elk uur een peer)

24
What can we measure?

• Attitudes
• Opinions
• Physiological measurements
• Performance
• Time
• Correctness
• Speed
• Paths
• Recognition/Memory
• Efficiency
• ..

25
Keystroke level model (KLM)

• Low level tasks
• KLM does not focus on highly cognitive tasks
• Provides rapid estimates of performance without
  any theoretical knowledge
• K keystroking, actually hitting keys on the
  keyboard
• B pressing a mouse button
• P pointing, moving the mouse or other device to a
  target
• H homing, switching the hand from the mouse to
  the keyboard or vice versa
• D drawing lines using the mouse
• M mentally preparing for a physical action
• R system response, this may be ignored if the
  user does not have to wait for it

26
Measures Registration

• Data-Logging by software
• Watch videos score (Observer)
• Live scoring by experimenter (field)
• Physiological measures
• ..

27
Logging by software
28
Watch videos score (Observer)
29
Live scoring
30
Physiological measures
31
On our wish list ? Eye tracking
32
Activity chart

• You will use this in the lab experiment
• Group 2-3-4 will use the log-data
• Group 1 explores the activity charts

33
Data processing

• Checkcheckcheckdouble check!
• Filling up the holes!
• Remove outlierstricky business ?
• Normally divided (or approximation)
• Transformations necessary?
• Choose methodology (whats legal)
• ..
• And of course, be honest and accurate

34
Van ruwe data bv. TXT.
35
Naar iets minder ruw bv. Excel
36
Naar uiteindelijke data bv. SPSS
37
What do my results mean?
Interaction Effect

• Effect

Stuks fruit gegeten
KM per uur
Vraag Wie kan meer eten? Mannen kunnen meer
appels eten, vrouwen kunnen meer peren eten
Vraag wie rent harder? Paarden kunnen harder
rennen dan koeien
38
Rapportage

• Hoofdeffect
• Interactie effect
• Tendens
• Hierbij
• Vermeld a
• Vermeld significantie waarde
• Vermeld gemiddelden
• Vermeld standaardafwijkingen