From Design to Evaluation

1
From Design to Evaluation

• Loren Terveen
• CS 5115, Fall 2008
• October 15

2
Agenda for today

• Introduction to interface evaluation
• Goals of evaluation
• Evaluation without users
• GOMS
• Cognitive walkthroughs

3
Hall of Fame/Shame

• Dryer User Interface
• Dillon Hodapp
• Matt Nohelty

4
Hall of Shame
Feedback Knob has orange mark indicating
position. This is impossible to see in a dark
basement. Labeling The text is small. Text is
arranged horizontally, but corresponds to radial
ticks, hard to tell which tick mark text
corresponds to. The word Automatic is used to
describe 2 of the settings, but not the other 2.
What is automatic about these 2 that arent
automatic when using the other settings. Gulf of
Evaluation to set the dryer User knows to turn
the knob to some position, but there are 4
different categories to choose from. There are
also 2 different timed dries, 2 very dry and less
dry settings and there are 4 off positions.
Also, when the dryer ends the cycle the user
needs to turn the knob all the way around to
start the same cycle again. Gulf of Execution to
turn on dryer Knob affords turning, but does not
afford pushing or pulling at first glance. It is
necessary to push the knob to start the dryer.
5
Hall of Fame
Feedback There is backlit text for each option
and when a button is pressed the lights change
indicating the success of the button press.
Labeling The settings are clearly aligned above
each of the buttons indicating which buttons are
used to change each setting. Cultural Symbols
The power button is labeled with the universal
1/0 symbol and the Start/Pause button has the
right pointing arrow and two vertical
lines. Affordances The dryer is completely
controlled by buttons. This is consistent
throughout the interface. The buttons only afford
pushing so there is no way to mistake their
function. No Gulf of Evaluation to set the dryer
It is clear which settings are selected and the
user can clearly and easily determine what the
dryer is going to do. There are also preset
options for your usual cycle settings. Note
Because this dryer provides additional
functionality, its interface is more complex.
For instance, it requires more interaction with
the interface to start the machine. However, the
interface is laid out in a much more intuitive
way overall.
6
Ketchup Bottles

• The True Measure of Humanity

Vs.
Jerod Pekuri Adam Ranfelt
7
Glass Ketchup BottleHall of Shame

• Standard
• Restaurants
• Affordance
• Bad Constraints
• Mappings
• Gulf of Execution
• Shake? Hit? Knife? Got a knife? Roll?
• Feedback Where's the ketchup?

8

• Small gulf of execution
• Soft plastic is immediately perceived as
  squeezable (affordance)?
• Physical constraint ketchup only comes out of
  one end
• Low gulf of evaluation ketchup comes out in a
  steady stream when squeezed
• Instant feedback

9
Towards evaluation...
10
Basic idea of UI Development

• Prototype and Iterate
• keep iterating until it is good enough
• evaluate along the way to assess
• What is good? What is good enough?
• set usability goals
• should relate to tasks

11
Informal Iteration

• Find major usability problems
• missing features
• user confusion
• inefficient interaction

12
Informal Iteration

• Remember the goal
• dont defend the interface - learn
• dont bias the tests towards the interface
• If possible, allow user exploration
• may even lead to capturing new tasks
• Consider alternative ways to fix a problem

13
Limits of Informal Iteration

• Does not indicate when to stop
• Financial trade-offs
• Justification of delay

14
Usability Goals and Measures

• Concrete, quantitative measures of usability
• learning time
• performance time for specific tasks and users
• error rates
• measures of user satisfaction
• Comparative usability goals
• compare with prior versions or competitors

15
Things to Watch

• Goals should be realistic
• 100 is never realistic
• Many goals go beyond the application UI
• training, manuals
• Goals should help improve the UI
• Detail not just good/bad

16
ExerciseSetting Usability Goals

• In project groups, come up with 3 usability goals
  for your project
• Discuss the feasibility of testing these goals
• what is needed for the test?
• when in the process can they be tested?
• how much effort, user preparation/training, etc.?
• what would you learn from the test?

17
Evaluation methods
18
Evaluation may be done with or without users

• Users arent perfect
• User time is valuable find/solve obvious
  problems so you dont waste users time
• Some users provide great information, others
  little
• But users are necessary
• cannot be simulated perfectly
• So test both ways

19
Evaluation Without Users

• Quantitative Methods
• GOMS/keystroke analysis
• back-of-the-envelope action analysis
• Qualitative Methods
• expert evaluation
• cognitive walkthrough
• heuristic evaluation

20
GOMS

• Goals Operators Methods Selection
• Based on information processing model of human
  cognition

21
GOMS

• Formal action analysis
• accurately predicts task completion time for
  skilled users
• Break task into tiny steps
• keystroke, mouse movement, refocus gaze
• retrieve item from long-term memory
• Look up average step times
• tables from large experiments

22
Examples

• http//www.cs.umd.edu/class/fall2002/cmsc838s/tich
  i/printer/goms.html

23
GOMS

• Primary utility repetitive tasks
• Success story telephone operators
• Gray, W. D., John, B. E., Atwood, M. E. (1993).
  Project Ernestine Validating GOMS for predicting
  and explaining real-world task performance. Human
  Computer Interaction, 8(3), 237-309.
• Benefit can be very accurate (within 20)?
• May identify bottlenecks
• Difficulties
• challenging to decompose accurately
• long/laborious process
• not useful with non-experts