Developing Web Design and Usability Guidelines: An EvidenceBased Approach November 4, 2002 Craig Laf

1
Developing Web Design and Usability Guidelines
An Evidence-Based Approach November 4, 2002
Craig LafondMary Frances TheofanosCommunicati
on Technologies BranchOffice of Communications
National Cancer Institute
2
Presentation Objectives

• Discuss why evidence-based web design is
  important
• Describe the process used to create the
  guidelines
• Highlight the expert review process used to
  create stronger guidelines

3
Evidence-Based Solutions

• We should base website design decisions on facts
• Attempt to
• Move away from country doctor model
• Move toward
• University-based Physician or
• Teaching Hospital Model

4
Evidence-Based Web Design

• Make the best web design decisions by putting the
  latest research (evidence) into practice early
• This leads to ensuring that sites always start at
  the highest level
• This prevents
• Exposure of users to unusable sites
• Wasting of resources
• Application of ineffective design ideas or design
  processes

5
Steps in the Process

• Identify and translate research into guidelines
• Create Relative Importance ratings
• Generate Strength of Evidence ratings

6
Step 1 Identify and Translate Research into
Guidelines

• Identify existing guidelines
• Yale Web Style Manual
• Ameritech Web Page User Interface Standards and
  Design Guidelines
• Sun Microsystems Guide to Web Style
• IBMs Web Design Guidelines
• The Library of Congress World Wide Web Style
  Guide
• Problems with style guides
• Contained conflicting guidance
• Yale Use longer pages to ease page maintenance
• Sun Use shorter pages to make site more
  maintainable
• Guidance too general
• No references

7
Guideline Limitations

• May not be specific
• Audiences
• Contexts/Situations
• May not be explicit
• May not consider usability/design experience
• Evidence from other disciplines may not be
  reflected

8
Practitioner Review

• Purpose
• Determine which guidelines practitioners think
  are most important, Create Relative Importance
  Ratings
• Reduce and strengthen the list
• 16 Reviewers
• 8 web designers
• 8 usability specialists
• December 11-31, 2001

9
(No Transcript)
10
Step 2 - Phase 1 Results

• No reliable difference between usability
  specialists and web designers in how they rated
  the guidelines
• Relative Importance and Strength of Research
  ratings are correlated

11
Step 2 - Phase 2

• Allowed reviewers to confirm or change their
  ratings with knowledge of
• Their previous ratings
• The group mean for Relative Importance
• Attempted to arrive at consensus

12
Step 2 - Phase 2

• 287 guidelines
• Same 16 reviewers
• Allowed reviewers to confirm or change their
  ratings with knowledge of
• Their previous ratings
• The group mean for Relative Importance
• Attempted to arrive at consensus

13
Reviewers ratings from Phase I
New for Phase 2
14
Step 2 - Phase 2 Results

• Again no reliable difference between ratings of
  web designers and usability specialists
• No reliable difference between Relative
  Importance and Strength of Research ratings
  from Phase 1 to Phase 2
• Relative Importance and Strength of Research
  are still correlated in Phase 2 (r.81 in both
  phases)

15
Basis for Strength of Research Rating

• Exclusively on experience
• Mostly on experience, though somewhat familiar
  with the research
• Mostly on experience, though very familiar with
  the research
• Half on experience, half on knowledge of the
  research
• Mostly on knowledge of the research, with strong
  confirmation from experience
• Mostly on knowledge of the research, with weak
  confirmation from experience
• Exclusively on knowledge of the research

16
Researcher Review

• Purposes
• Learn whether researchers will rate the
  guidelines differently than practitioners
• Assess the level of agreement on an
  evidence-based rating
• Collect additional references
• Identify where there is no evidence to support
  common web practices
• Establish an ongoing process for future reviews

17
Step 3 Participants

• Nine usability experts
• All with
• Strong educational background
• Expert understanding of the usability literature
• Good understanding of experimental design issues

18
Researcher Review

• Step 1
• Each reviewer put each of the 225 guidelines into
  one of 3 categories
• strong research support
• weak research support
• no research support
• If any one of the reviewers indicated that the
  guideline had any research support, it was kept
  in the process
• Reviewers also added new references

19
Researcher Review

• Step 2
• There was little agreement on guidelines with no
  research support
• Researchers met on August 8 to establish a common
  framework for rating the guidelines
• Decided to classify references into 9 types
  (hypothesis-based, expert opinion, survey data,
  literature review, etc.)

20
Researcher Review

• Step 3
• Researchers rated each of the guidelines on a
  common scale
• Strong Research Support
• Cumulative and compelling, supporting
  research-based evidence
• At least one formal, rigorous study with
  contextual validity
• No known conflicting research-based findings
• Expert opinion agrees with the research
•  
• Moderate Research Support
• Cumulative research-based evidence
• There may or may not be conflicting
  research-based findings
• Expert opinion
• Tends to agree with the research, and
• Consensus seems to be building

21
Researcher Review

• Weak Research Support
• Limited research-based evidence
• Conflicting research-based findings may exist
• and/or
• There is mixed agreement of expert opinions
• Strong Expert Opinion Support
• No research-based evidence
• experts tend to agree, although there may not be
  a consensus
• Multiple supporting expert opinions (presented in
  textbooks, style guides, newsletters, etc.)
• Generally accepted as a 'best practice' (or
  reflects current 'state of practice')
• Weak Expert Opinion Support
• No research-based evidence
• Limited or conflicting expert opinion

22
Results to Date
Results to Date

• References by type
• Observational study 17
• Experiment 190
• Model-based 15
• Expert opinion 57
• Literature review 78
• Survey 7
• Textbook 27
• Usability test 24
• Exploratory study - 20

23
Conclusions

• Research helps us better understand what works
• We need
• More research
• Better research
• Greater use of research findings
• Final goal To help practitioners make design
  decisions that achieve higher-quality,
  professional user interfaces!