Chapter 7 Usability and Evaluation

1
Chapter 7 Usability and Evaluation

• Dov Teeni
• Jane M. Carey

2
Chapter 7 Usability and Evaluation

• Usability and evaluation are closely related
  concepts
• Usability means that the HCI design of the system
  supports the users cognitive and ergonomic
  limitations and is easy to use and learn
• Evaluation means to assess the system for
  functionality as well as usability
• We explore how to compare, assess, and improve
  interfaces

3
Chapter 7 Usability and Evaluation

• Mini-Case 1984 Olympic message system
  demonstrates
• You will not get it right the first time
• You will get different answers when you use
  different methods of observation
• You will need to come up with integrative
  solutions to problems

4
Chapter 7 Usability and Evaluation

• Designers of the OMS (Olympic Message System)
  followed 3 principles
• Early focus on user and tasks
• Empirical measure on users behavior
• Iterative design

5
Chapter 7 Usability and Evaluation
Figure 7.2 Attitudes, use, performance, and
satisfaction relationship
6
Chapter 7 Usability and Evaluation

• ISO (International Standards Organization)
  defines usability as a concept comprising the
  effectiveness, efficiency, and satisfaction with
  which specified users can achieve specified goals
  in a particular environment

7
Chapter 7 Usability and Evaluation
Table 7-1 Usability indicators based on
performance 1. Goal achievement indicators
(success rate, failure rate, accuracy,
effectiveness).2. Work rate indicators (speed,
completion rate, efficiency, productivity,
productivity gain).3. Operability indicators of
the user's ability to make use of the systems
features (error rate, problem rate, function
usage)4. Knowledge acquisition indicators of
the user's ability and effort in learning to use
the system (learnability and learning).
8
Chapter 7 Usability and Evaluation
Usability measures based on performance and
HCI 1. Time to complete a task2. Number of user
commands to complete task3. Fraction of task
completed4. Fraction of task completed in a
given time5. Number of errors6. Time spent on
errors7. Frequency of online help used8. Number
of available commands not used9. When task is
repeated, ratio of successes to failures10.
Fraction of positive comments made by user11.
Fraction of good to bad features recalled by
user12. Number of expressions of frustration and
satisfaction13. Number of times user loses
control over system14. Number of times the user
needs to devise a way of working around the
problem/system.
9
Chapter 7 Usability and Evaluation
Table 7-3 The usability engineering life
cycle Know the user Analyze competing
products Set usability goals Consider
alternative designs Engage in participatory
design Coordinate the total interface Check
against heuristic guidelines Prototype
Evaluate interface Design in iterations
Follow up with studies of installed systems
10
Chapter 7 Usability and Evaluation

• Methods of usability assessments include
• Thinking aloud (protocol analysis)
• Observation
• Interviews
• Focus groups
• Automatic logs
• Questionnaires

11
Table 7-4 Attitude questionnaire 1. Using a
microcomputer could provide me with information
that would lead to a better decisions.2. I
wouldn't use a microcomputer because programming
it would take too much time.3. I'd like to use a
microcomputer because it is oriented to user
needs.4. I wouldn't use a microcomputer because
it is too time consuming.5. Using a
microcomputer would take too much time away from
my normal duties.6. Using a microcomputer would
involve too much time doing mechanical operations
(e.g., programming, inputting data) to allow
sufficient time for managerial analysis.7. A
microcomputer would be of no use to me because of
its limited computing power.8. I'd like to learn
about ways that microcomputers can be used as
aids in managerial tasks.9. Using a
microcomputer would result in a tendency to over
design simple tasks.10. I wouldn't want to have
a microcomputer at work because it would distract
me from my normal job duties.11. A microcomputer
would give me more opportunities to obtain the
information that I need12. I wouldn't favor
using a microcomputer because there would be a
tendency to use it even when it was more time
consuming than manual methods.13. I'd like to
have a microcomputer because it is so easy to
use.14. I'd hesitate to acquire a microcomputer
for my use at work because of the difficulty of
integrating it with existing information
systems.15. I'd discourage my company from
acquiring microcomputers because most application
packages would need to be modified before they
could be useful in our specific situation.16. It
is easy to access and store data in a
microcomputer.17. A microcomputer would be of no
use to me because of the limited availability of
application program packages.18. A microcomputer
would be of no use to me because of its small
storage capacity.19. It is easy to retrieve or
store information from/to a microcomputer.20.
Using a microcomputer would give me much greater
control over important information.
12
Chapter 7 Usability and Evaluation
Step 7 in the engineering life cycle is check
against heuristic guidelines
Heuristic guidelines 1. Create simple and
natural dialog2. Speak the user's language3.
Minimize the user's memory load4. Be
consistent5. Provide feedback6. Provide clearly
marked exits7. Provide shortcuts8. Provide
specific, corrective and positive error
messages9. Minimize propensity for error
13
Chapter 7 Usability and Evaluation

• Evaluation is a broader term than usability it
  includes several goals
• 1) assess the system's functionality against
  the intended specifications, 2) assess the
  system's effect on the user's behavior and
  attitude, 3) assess the system's impact on
  measures of performance that are related to the
  user or the objective of the system, 4) discover
  unintended problems and perhaps opportunities.

14
Chapter 7 Usability and Evaluation

• Evaluation techniques include
• 1) Exploratory vs. model based.2) Design or
  implementation.3) Field study vs. laboratory
  testing.4) Design vs. use.5) Level of
  performance measures.6) Degree of designed
  manipulation and intrusion.

15
Chapter 7 Usability and Evaluation
Cognitive Walkthrough (Polson et al, 1992)
Task description from the first-time user's
viewpoint. Include any special assumptions about
the state of the system assumed when the user
begins work. Action sequence Make a numbered
list of the atomic actions that the user should
perform to accomplish the task. Anticipated
users Briefly describe the class of users who
will use this system. Note what experience they
are expected to have with similar or previous
versions. User's initial goals List the goals
the user is likely to form when starting the
task. If there are other likely goals list them,
and estimate for each what percentage of user are
likely to have them.
16
Chapter 7 Usability and Evaluation
Table 7-6 Measures for comparing displays (Scott
Findlay, 1991) Typing mistakes made before
correction Mistakes remaining Pauses of 3
seconds or more immediately before mode change
Other pauses of 3 seconds or more Length of
pause immediately before mode change Attempting
to type-over whilst in insert mode Attempting
to insert whilst in type-over mode
17
Chapter 7 Usability and Evaluation

• Ben Shneidermans Questionnaire for user
  satisfaction

18
Chapter 7 Usability and Evaluation

• Other methods of evaluation
• Metrics (Rengger, 1991)

19
Chapter 7 Usability and Evaluation

• Conclusion Flow diagram from Olson and Moran
  (1996)

20
Chapter 7 Usability and Evaluation

• Table 7.8 ANSI/HFS 200 outline and status (1997)

Section Status1 Introduction draft2
accessibility draft3 presentation of information
ISO document4 user guidance ISO document5
direct manipulation ISO document6 color draft7
forms fill-in ISO document8 command languages
ISO document9 voice i/o Voice recognition,
non-speech auditory output, interactive voice
response drafts10 visually displayed menus
re-drafted ISO doc