Measuring Website Usability: Instrument Development, Validation, and Application

1
Measuring Website Usability Instrument
Development, Validation, and Application
Big XII IS Research Symposium April 5,
2003 Younghwa Gabe Lee University of Colorado
at Boulder leey_at_colorado.edu
2
Agenda

• Background
• Previous Studies
• Research Objective
• Research Design
• Expected Contribution
• Discussion

3
Background

• Online business failures are increasing as
  customers turn away from unusable or unfriendly
  sites. Build it and they will come mentality
  has led to the demise of e-commerce sites when
  sites are too late, too buggy, or too complex
• - Becker and Mottay, 2001
• In a poorly designed EC environment, users might
  be uncomfortable with the uncertainty and
  ambiguity caused by lack of interaction with
  websites.
• Jahng et al, 2000
• The number of shoppers and total sales are still
  marginal, mainly because of poor interfaces
• - Jarvenpaa and Todd, 1997

4
Background

• Building a usable website is important since
  website is the
• only source for online customers to touch, feel,
  search,
• communicate, and experience the products or
  services
• available at the online store
• Usable websites
• Build positive attitude (Singh and Dalal, 1999)
• Increase stickiness (Rettie, 2001)
• Increase revisit rates (Klein, 1998)
• Increase online purchase (Palmer, 2002)
• Increase performance (Nielsen, 2000)
• Provide more satisfaction (Lund, 1999)

5
Website Usability Research
Website Usability
How to measure?
How to verify?
Instruments Development
Design and Testing

• HCI
• Company-specific
• Industry Gurus
• Few IS researchers

• HCI
• MIS SIG HCI

6
Previous Studies

• Measurement problems of website usability
• No consensus on the definition and dimensions of
  website usability
• A number of single-item constructs
• Intuition and experience-based Few efforts to
  develop measurement using scientific methods
• HCI-oriented objective variables (error rate and
  download time)
• No Investigation of the relationship between
  website usability constructs
• No process model

7

• Agarwal and
• Venkatesh (2002)
• Content
• Ease of Use
• Promotion
• Made-for-the-medium
• Emotion

• Palmer (2002)
• Download Delay
• Navigability
• Information Content
• Interactivity
• Responsiveness

• Kim et al. (2002)
• Firmness
• Convenience
• Delight

• Zhang and
• von Dran (2002)
• Content
• Enjoyment
• Privacy
• User empowerment
• Visual appearance
• Technical Support
• Navigation
• Credibility
• Organization

• McKinney et al.
• (2002)
• Access
• Usability
• Entertainment
• Hyperlinks
• Navigation
• Interactivity

8
Download Delay
Navigability
Information Content
Perceived Success
Interactivity
Responsiveness
Palmer (2002)
9
Motivation of the Study

• Current inconsistency and incompleteness among
  website usability measurement is the crucial
  problem of website usability studies.
• There is very little in the way of concrete
  measurement that tells us how good a website
  really is. Current guidelines, methods, and
  metrics do help to design better websites, but
  there is room for improvement
• - Tarasewich (2000)

10
Research Objective

• Develop measurement of website usability
• 18 constructs and 62 instruments have been
  identified
• Investigate the causal relationship between
  website usability constructs
• Revealed Causal Mapping approach (Nelson et al.,
  2000)
• Examine the effects of website usability
  constructs to multiple dependent variables
• Satisfaction, purchase intention, revisit
  intention, actual purchase, affect, and loyalty
• Investigate generalizability of the new
  measurement and identify different causal maps
  under different boundary conditions
• Gender, Product, Industry and Culture

11
Research Design

• Instruments Development
• Literature review
• Interviews with web usability experts
• SUN, IBM, 37 Signals.com website designers
• A Focus Group Study (IS-majored master-level
  subjects)
• A Major survey to 400 Business undergraduate
  students
• Exploratory Factor Analysis
• Causal Relationship Between Website Usability
  Constructs
• Interviewed with experts (n 20)
• Interviewed with experienced online customers (n
  40)
• Data Analysis suggested by Nelson et al. (2000)

12
Research Design

• Effects of Website Usability Factors to Diverse
  Dependent Variables
• The effects of Price, Time, Scarcity,
  Convenience, Fun, Usefulness will be examined
  together
• CFA and Path Analysis will perform
• Data Analysis LISREL
• Boundary Conditions
• Compare websites with different gender-focused
• Compare websites with different types of products
  (Hedonic vs Utilitarian)
• Compare websites with different cultures (U.S. vs
  Japan)
• Compare websites with different stakeholders
  (Customers, Designers, and Managers)
• Data Analysis PLS

13
Website Usability Constructs
14
Causal Relationship between Website Usability
Constructs
15
Effects of Usability Constructs to Diverse
Dependent Variables
Actual Purchase
Revisit Intention
Satisfaction
Purchase Intention
Affect
Loyalty
Price Time Scarcity Convenience Fun Usefulness
16
Expected Contribution

• Develop new measurement of website usability
• provide a better means to evaluate website design
  quality
• Identify causal relationship between website
  usability constructs
• Provide a deeper understanding of how online
  customers build their usability perception
• Examine effects of website design factors to
  diverse dependent variables
• justify the investment on website usability
• Investigate measurements generalizability and
  causal maps under different boundary conditions
• help to perform future study using the
  measurement
• help designers to allocate limited resources to
  the most important usability factors

17
Discussion
18
Discussion
19
Revealed Causal Mapping Repertory
Grid
Causal Relationship Identification
Construct Identification Guided Interviews
Open Interviews No
requirement for multiple elements Multiple
elements are required Identify constructs and
causal Identify constructs using
Relationship based on repeated
Triadic Methods to identify Interviews for
identifying Mid-theory exists
No requirement for mid-theory
20
Discussion

• Alternative ways of conducting the research
• Process Model (e.g TAM) vs Multiple Dependent
  Variables
• Boundary conditions valuable to be observed