Assessing a Firm

1
Assessing a Firms Web Presence A Heuristic
Evaluation Procedure for the Measurement of
Usability

• By Ritu Agarawal and Viswanath VenkateshISR,
  Vol. 13, No. 2, June 2002, pp.168-186.
• ??????
• ??92.11.25

2
Outline

• Introduction
• Literature Review
• Study Methods and Procedures
• Results
• Limitations
• Discussion and Implications
• Conclusion

3
Introduction

• Explosive growth in EC and net-enable
  organizations
• Web is chosen as an alternative channel
• The design of the Web site is crucial determinant
• visitors are likely to return
• consumer satisfaction
• the short- and long-term success of Web site.

4
Introduction (cont.)

• Usability
• emerges from human-computer interaction (HCI)
  research
• measures the quality of Web presence
• is a key and proximal metric for evaluating the
  success of an Web presence
• Includes procedure and metric

5
Introduction (cont.)

• Important properties of usability
• The metric should be able to discriminate across
  site
• It must offer specific insights into areas of
  weaknesses in the design of the site
• Research objectives are to describe
• a method for assessing Web site usability
• An accompanying metric that is valuable to
  researchers and practitioners

6
HCI and Usability

• The notion of usability is a key theme in HCI
  literature
• ISO definition of usability
• The extent to which a product can be used by
  specified users to achieve specified goals with
  effectiveness, efficiency, and satisfaction in a
  specified context of use.
• Two important findings
• The importance of consistency in design
• The idea that prior knowledge possessed by users
  plays a key role in subsequent learning of new
  artifacts and devices.

7
Literature Review

• A variety of approaches to usability evaluation
• Nielsen (1994) identify eight distinct
  approaches heuristic evaluation, guideline
  reviews, pluralistic walkthroughs, consistency
  inspections, standard inspections, cognitive
  walkthroughs, formal usability inspections, and
  feature inspections.
• Gray and Salzman (1998) classify usability
  evaluation into analytic and empirical categories

8
Literature Review (cont.)

• Two recurrent themes
• Usability is multifaceted and must be assessed by
  using different measures
• The dependence on subjective assessment in the
  form of user judgments
• Kantner and Rosenbaum (1997)
• heuristic evaluation small group
• laboratory testing real users

9
Microsoft Usability Guidelines (MUG)

• The heuristic guidelines
• Five major categories
• Content
• Ease of use
• Promotion
• Made-for-the-medium
• Emotion

10
MUG (cont.)

• Content
• accesses the informational and transactional
  capabilities of a Web site
• Relevance the pertinence of the content to the
  core audience
• Media use the appropriate use of multimedia
  content
• Depth and breadth the appropriate range and
  detail of topic
• Current and timely information

11
MUG (cont.)

• Ease of use
• relates to the cognitive effort required in using
  a Web site.
• Goals clear understandable objectives.
• Structure the organization of the site
• Feedback the extent to which the Web site
  provides information regarding progress to the
  user.
• Promotion
• captures the advertising of a Web site on the
  Internet and other media.

12
MUG (cont.)

• Made-for-the-medium
• relates to tailoring a Web site to fit a
  particular users need
• Community provides users with an opportunity to
  be part of the online group
• Personalization technology-oriented
  customization
• Refinement the particular prominence given to
  current trends

13
MUG (cont.)

• Emotion
• affective reactions invoked by a Web site
• Challenge captures the idea of difficulty
• Plot relates to how the site piques the users
  interest, especially with a story line
• Character Strength relates to the credibility
  conveyed by the site
• Pace examines the extent to which the site
  provides users an opportunity to control the flow
  of information.

14
Study Methods and Procedures (cont.)

• A usability assessment procedure provides
  detailed information on what matters to different
  types of users when they visit Web site from
  different industries
• Develop a method for the assessment of usability
• Weights the relative importance (weights) of the
  different categories
• Ratings user provide ratings for specific Web
  sites on various subcategories.

15
(No Transcript)
16
Instrument Development

• Content validity conducted by four phases
• 1st Phase labeled items
• two experts in the domain of usability IS
• two experts in measure development statistical
  procedures
• two Ph.D. students in IS
• 2nd Phase 1st phase was repeated (40
  undergraduate students)

17
Instrument Development (cont.)

• 3rd Phase labeling procedure was conducted again
• The convergence and accuracy increased to nearly
  100.
• 4th Phase one item was chosen to represent each
  category and subcategory of weight and rating
• 30 randomly chosen individuals were asked to
  label the item

18
Instrument Development (cont.)

• Instructions
• General instructions task instructions (Table
  2, p.174)
• Weighting scheme and category weighting items
  (Table 3, p.175)
• Instructions and items for weight distribution
  across subcategories (Table 4, p.176)
• Instruction for rating sites (Table 5, p.176)

19
Instrument Development (cont.)

• Construct validity three-item scale was used
• How do you rate the overall usability of the Web
  site?
• How do you rate the overall design of the Web
  site?
• How do you rate your overall experience at the
  Web site?
• Two pilot studies were conducted

20
Participants

• Visitors to three branch of major electronic
  retail store during three-day period (Fri.-Sun.)
• Participant was given 10 gift card
• 1823 participants
• 1475 usable responses (81 response rate)

21
Web Site Studied
22
Procedure

• Industry assignment randomly
• Task assignment randomly
• To provide weights of different criteria
• To visit the Web sites
• The order of presentation of Web site randomly
• 5 minutes to browse (further browsing is allowed)
• To response to a three-item questionnaire
• To rate (1-10) different MUG attributes
• Demographic information was gathered

23
Demographic Characteristics
24
Results

• Cronbachs a over 0.8
• The correlations between the calculated usability
  rating and usability measured using the
  three-item scale were very high ranging from
  0.71-0.93

25
User Assessment of Weights
26
User Assessment of Weights (cont.)

• Content the most important
• Investors believed content to be more important
  than did customers.
• Ease of use modestly important
• Customers deemed ease of use more important than
  investors.
• Promotion importance varied across tasks,
  regardless of industry
• Investors weighted nearly twice as much as
  customers did

27
User Assessment of Weights (cont.)

• Made-for-the-medium influenced by a two-way
  interaction of product and task
• was more important to customers in three of the
  four industries ( except auto manufacturing)
• Emotion also influenced by a two-way interaction
  of product and task
• The auto manufacturing sites customers deemed
  emotion to be very important compared to
  customers in all industries
• Investors viewed emotion to be only minimally
  important

28
User Assessment of Usability
29
User Assessment of Usability (cont.)

• A comparison across industries reveals that
  bookstore sites scored highest with both
  customers and investors (?)
• Most sites were seen to be higher in terms of
  usability by customers when compared to investors.

30
User Assessment of Usability (cont.)

• Airline industry
• Customer task the least variability
• Investor task
• All sites to be equivalent
• Significantly lower than the customer task
• Bookstore industry
• To be the best and the worst across all sites in
  all four industries for the customer task
• The sites that were rated lower for the customer
  task emerged as being quite highly rated for the
  investor task.

31
User Assessment of Usability (cont.)

• Auto manufacturer
• Customer task exhibited a great deal of variance
• The investor ratings were lower than customer
  ratings.
• Car rental industry
• Customer task similar to the airline industry
• The investor ratings were lower than customer
  ratings
• Investor task showed the poorest usability
  across all sites

32
Limitations

• Selection bias incentive -- 10
• Generalizability only 4 industries
• Systematic bias
• Only the context of B-C sites were investigated
  B-B?
• Assigned roles
• customer
• investor
• Actual behavior was not measured

33
Discussion

• A useful usability metric discriminate across Web
  sites from different industries and among
  different types of users
• The importance of content was highest across all
  attribute categories
• The ability of a Web site to support promotion is
  more important for investors than it is for
  consumers

34
Discussion (cont.)

• A Web site generates emotion when users interact
  with it through judicious use of features such as
  character strength and pace
• The made-for-the-medium category from MUG, were
  similarly affected by a product-task interaction.

35
Implications for Practice

• Usability is an important metric for assessing
  Web site design, managers need systematic
  methodologies for performing usability
  assessments
• Users visit a Web site with a variety of goals,
  predispositions, and purposes in mind
• Personalization dynamic content
• Product-task interactions do exist and must be
  focused on in Web site design

36
Implications for Research

• Theory development
• To extend across more products and industries to
  determine the robustness of the instrument
• Empirical testing to compare weights and ratings
  assigned by actual users
• To increase confidence by reducing extraneous
  variance associated with other factors

37
Conclusion

• Examine the design of a corporate Web site and
  offer a metric
• As net-enabled organizations continue to increase
  investment in their Web presence, the results
  should be useful in an on-going assessment of
  potential impact
• Contribute an important metric to help managers
  understand and predict the likely success of EC