Usability Evaluation

1
Usability Evaluation

• or, I cant figure this out...do I still get the
  donuts?

2
Purposes of Evaluation

• Which design is better?
• Are there any problems with the design as it now
  stands?
• Does the design meet usability targets?

3
When do you evaluate?

• Formative evaluation (e.g. scenarios)
• Helps with form of the solution
• Deciding between competing designs
• At early and intermediate design stages
• Summative evaluation
• Provides summary of usability
• Often compares new design with existing or
  alternative solutions
• When design is complete

4
What do you evaluate?

• Prototypes
• During design iterations
• Can be very low-fidelity
• Wizard of Oz technique
• Working systems
• System can be prototype
• Can also evaluate at end of design cycle

5
Types of Evaluation

• Depends on range of formality and completeness of
  system
• Informal user studies
• Usability studies
• Formal experiments

6
User Studies

• Early stages of design
• Usually only a few users
• Non-structured tasks
• Collect comments, observations, suggestions,
  preferences, ...

7
Usability Studies

• Usually more complete prototype
• Structured tasks
• Collect timings, errors, verbal protocol, ...
• Issues
• Finding users
• Testing environment
• Compensation

8
Formal Experiments

• Working system or piece of system
• Number of users determined by desired difference
• Tightly controlled tasks
• Precise measurements
• Statistical analysis of hypotheses

9
Ethics

• Testing can be a distressing experience
• pressure to perform, errors inevitable
• feelings of inadequacy
• competition with other subjects
• Golden rule
• subjects should always be treated with respect

10
Managing Subjects Ethically

• Before the test
• Dont waste the users time
• Use pilot tests to debug experiments,
  questionnaires etc
• Have everything ready before the user shows up
• Make users feel comfortable
• Emphasize that it is the system that is being
  tested, not the user
• Acknowledge that the software may have problems
• Let users know they can stop at any time

• Maintain privacy
• Tell user that individual test results will be
  kept completely confidential
• Inform the user
• Explain any monitoring that is being used
• Answer all users questions (but avoid bias)
• Only use volunteers
• user must sign an informed consent form

11
Managing Subjects Ethically

• During the test
• Dont waste the users time
• Never have the user perform unnecessary tasks
• Make users comfortable
• Try to give user an early success experience
• Keep a relaxed atmosphere in the room
• Coffee, breaks, etc
• Hand out test tasks one at a time
• Never indicate displeasure with the users
  performance
• Avoid disruptions
• Stop the test if it becomes too unpleasant
• Maintain privacy
• Do not allow the users management to observe the
  test

12
Managing Subjects Ethically

• After the test
• Make the users feel comfortable
• State that the user has helped you find areas of
  improvement
• Inform the user
• Answer particular questions about the experiment
  that could have biased the results before
• Maintain privacy
• Never report results in a way that individual
  users can be identified
• Only show videotapes outside the research group
  with the users permission

13
Components of a Study

• Informed consent
• User familiarization
• User questionnaire
• Background testing
• System exploration
• Specific tasks
• Post-testing
• Debriefing

14
Informed Consent

• Form advising users of their rights
• Tell them you are studying the system, not them!
• Tell them if there are any known risks
• Tell them they can stop at any time
• Get signature, give them a copy

15
User Familiarization

• Make the user comfortable
• Tell them what youre doing without giving
  everything away
• Show them the facilities and equipment
• Give them written instructions
• Tell them approximately how much time the
  evaluation will take
• Ask if they have any questions

16
User Questionnaire

• Obtain demographic information
• Age
• Gender
• Occupation or major
• Computer experience
• Domain experience
• Eyesight
• Handedness

17
Background Testing

• You may want to correlate your findings with some
  standardized test score
• Examples
• Spatial ability tests
• Memorization tests
• Domain knowledge tests
• Example result Users with high spatial ability
  preferred interface X, while others preferred
  interface Y

18
Pre- and Post-Testing

• Some systems have a specific learning goal (i.e.
  education or training)
• Giving users the same test or type of test both
  before and after usage of the system is a way to
  measure learning
• Only used with very well-developed prototypes

19
System Exploration

• User gets free play time with the system
• Observe what they seem to understand easily and
  what is troublesome
• See if they find all the features or parts of
  the system
• Good with verbal protocol

20
Specific Tasks

• Give the user a specific goal
• Ex Buy a burger with extra pickles and no onions
  using this interface
• Observe problems
• Record performance, errors

21
Post-Questionnaire

• Get users reaction to the system
• Subjective levels of satisfaction, perceived ease
  of use, usefulness, etc.
• Objective comments, thoughts, and questions

22
Debriefing

• Talk to the users about the session
• Assure them they did well
• Give more details about what youre doing if they
  are interested
• Thank them
• Give them donuts ?

23
Other Evaluation Issues

• Recruiting users
• Testing facilities
• Measurements and observations
• Compensation

24
Recruiting Users

• Attempt to match the proposed user population
• Perhaps divide into several user groups
• Techniques
• Posted advertisements
• Internet/email/newsgroup
• Colleagues/friends/classmates
• People already using existing system
• Make sure your users are not overly knowledgeable!

25
Testing Facilities

• For informal studies, a simple setup
• In all cases, privacy is important
• Replicate the usage environment?
• Evaluator present or not?
• More formal studies may use a special usability
  lab (e.g. McBryde 102 A or C)

26
Measurements Observations

• User comments
• General observations
• Specific critical incidents
• Task timing
• User errors

27
Session management

• May need multiple evaluators
• Use checksheets or pre-printed tables for filling
  in results
• Video/audio as backup if something is missed
• Dont ask the user to stop while you catch up!

28
Compensation

• Most studies give the user something for their
  time and effort
• Doesnt have to be monetary can also be
• Food
• Extra credit in a class
• Special discounts on the companys products
• Tour of your facility