Analyzing and Presenting Results Establishing a User Orientation

1
Analyzing and Presenting ResultsEstablishing a
User Orientation

• Alfred Kobsa
• University of California, Irvine

2
Tabulating and analyzing data

• Tabulate data in spreadsheet(s) per user and per
  task
• Both quantitative and qualitative data (e.g.,
  comments)
• Compute totals per user and averages per task
• Find outlier values in the raw data
• Try to explain them
• get back to the original data source to check for
  transcription errors
• look at time sheet / protocol and video recording
• Outliers may point to infrequent usability
  problems, or they may derive from accidental
  characteristics of the respective test user. In
  the latter case
• Disregard outlier values if this can be
  justified, or use median instead of average
• Remove subjects with many outlier values
  completely if this can be justified (very few
  subjects only!)
• Look at means/medians and possibly standard
  deviations to
• determine whether usability concerns are
  confirmed by the data
• discover surprises in the data, and determine
  whether they point to usability problems

3
Analyzing video and audio recordings

• Unless subjects were asked to think aloud, it
  is generally easier to analyze video data with
  concrete questions in mind rather than merely
  watching out for usability problems
• This does not so much apply to audio, since
  subjects often verbalize the problem they
  encounter
• Observations should be noted down (with time
  stamps)
• Categories for observations may already exist, or
  can be created in the observation process
• Often it is advisable to use two independent
  observers who afterwards compare their notes (and
  get back to the recordings to resolve disputes)

4
Statistical presentation and analysis

• Results of usability tests are usually presented
  using
• tabulated raw values
• descriptive statistics (means, medians, standard
  deviations)
• visualizations of raw values and statistical
  values
• In rare cases, inferential statistics can be used
• Specifically for comparing two competing
  prototypes, or the old and the new system
• Should be done with extreme caution, since
• Preconditions for the applicability of
  statistical tests are often not met (randomness
  of subject sampling and assignment to conditions,
  normal distribution of data)
• Sample sizes are often very small
• Statistical significance of a difference does not
  mean that the difference is important
• Decision makers do not know how to interpret the
  results of a statistical test (and are not
  familiar with the preconditions and limits of
  such tests)
• Testers are not well trained in statistics and do
  not know which test is appropriate

5
Identifying usability problems

• Involve the designers / programmers (particularly
  if they are going to perform the revisions)
• Focus on global problems since they often affect
  many aspects of an interface
• Global problems are more difficult to pinpoint
  and to correct
• Rank problems by level of severity
• Level 1 problem may prevent the successful
  completion of a task
• Level 2 problem may create significant delay and
  frustration
• Level 3 problem has minor effect on usability
• Level 4 possible enhancement that can be added
  in the future
• Recommend changes (and test those changes later)

6
Communicating the results

• Preparing a report / reports
• See Dumas and Reddish, Chapter 22
• Courage and Baxter, Chapter 14
• Preparing a Powerpoint presentation
• Preparing a stand-alone video/multimedia
  presentation
• See Dumas and Reddish, Chapter 23

7
Changing the product and process

• Collaborate with designers/developers throughout
  the evaluation process (and possibly with
  management)
• Prioritize and motivate your recommendations for
  re-design
• Collaborate on finding feasible ways to fix the
  problems
• Make suggestions to improve the design process,
  such as
• earlier involvement of users
• earlier testing of designs and prototypes
• hiring HCI staff
• developing design guidelines