Metrics from Chapters 6

1
Metricsfrom Chapters 6 21
2

• The desired qualitative or quantitative usability
  goals for a system
• Qualitative desired goals for usability
• Quantitative usability metrics

3
Early Views of Usability

• Learnability time and effort required to reach
  a specified level of use performance
• Throughput tasks accomplished by experienced
  users, speed, and errors
• Flexibility extent to which system can
  accommodate changes
• Attitude positive attitude of users

4
Possible measurement Criteria

• 1. Time to complete task.
• 2. Percentage of task completed.
• 3. Percentage of task completed per unit time
  (speed metric).
• 4. Ratio of successes to failures.
• 5. Time spent on errors.
• 6. Percentage or number of errors.
• 7. Percentage or number of competitors that do
  this better than current product.
• 8. Number of commands used.

5
Possible measurement Criteria (part 2)

• 9. Frequency of help or documentation use.
• 10. Time spent using help or documentation.
• 11. Percentage of favorable to unfavorable user
  commands.
• 12. Number of repetitions of failed commands.
• 13. Number of runs of successes and of failures.
• 14. Number of times the interface misleads the
  user.
• 15. Number of good and bad features recalled by
  users.
• 16. Number of available commands not invoked.

6
Possible measurement Criteria (part 3)

• 17. Number of regressive behaviors.
• 18. Number of users preferring your system.
• 19. Number of times users need to work around a
  problem.
• 20. Number of times the user is disrupted from a
  work task.
• 21. Number of times the user loses control of the
  system.
• 22. Number of times the user expresses
  frustration or satisfaction.

7
A Sample Row
8
Metrics Exercise

• Think of some usability metrics that may be
  appropriate for laser bar code scanning systems.
  These systems could be for grocery store use,
  perhaps, or for use by clothing retailers,
  DIY/home improvement stores, or electrical goods
  outlets.

9
Possible Metric Answers

• The system should be learnable by cashiers with
  not more than one hour of training. (This amount
  of training may seem excessive, but cashiers will
  also need to know what to do in cases where codes
  will not scan or where codes show the wrong
  price.)
• The system should be relearnable after a period
  of up to one year in less than 10 minutes.
• The system should provide a response to a scanned
  item in not more than two seconds.

10
Plan - Deciding on what you need to evaluate
11
Creating an Evaluation Strategy

• What Is the Purpose of This Evaluation?
• Does the Web Site meet usability
  requirements/concerns
• Qualitative Usability Requirements
• Desired features
• The users on an e-shopping site should be able
  to order an item easily and without assistance.
• Railway clerks work in extremely noisy
  environments, so any warning messages to them
  should be visually distinct and highlighted on
  the screens.

12
Creating an Evaluation Strategy

• Quantitative Usability Requirements/Usability
  Metrics
• Explicit measures used percentages, timings, or
  numbers are used.
• It should be possible for the users to load any
  page of a web site in 10 seconds using a 56K
  modem.
• It should take no more than two minutes for an
  experienced user (one who has domain knowledge
  and has undergone the prescribed level of
  training when the new system is introduced) to
  enter a customers details in the hotels
  database

13
Creating an Evaluation Strategy

• Prioritizing Usability Requirements and Concerns
• The usability requirements most important to the
  success of the system are given priority.
• Assign values to the five dimensions of
  usability, the Five Es.

14
Creating an Evaluation Strategy

• What Type of Data Do I Want to Collect?
• Quantitative data
• Numeric content
• Qualitative data
• Non-numeric content

15

• What Am I Evaluating?
• What Constraints Do I Have?
• Money
• Timescales
• Availability of usability equipment
• Availability of participants and the costs of
  recruiting them
• Availability of evaluators
• Documenting the Evaluation Strategy

16
The Evaluation Strategy

• Deciding What to Test
• Do You Have to Work within Any Constraints?
• Writing Up Your Evaluation Strategy for the
  Global Warming Evaluation