CostJustifying Usability

1
Cost-Justifying Usability

• Javad Sadeghi
• CPSC 681
• Nov 5, 2007

2
Outline

• Motivation
• Background
• Discount usability
• Samples
• Methodology
• Worked example

3
Motivation
Dennis Wixon, SIGCHI Bulletin, Volume 27, Number
2, April (1995), p94
4
View from the other side of the table

• For each dollar a company invests in developing
  the usability of a product, the company receives
  10-100 in benefits and wins customer
  satisfaction and continued business
• For each dollar spent to fix a problem during
  product design, 10 are spent to fix the same
  problem in product development, and 100 or more
  are spent to fix the same problem after product
  release

Claire Marie Karat, A business case approach to
usability cost justification. In, R. Bias and D.
Mayhew, Eds. Cost-Justifying Usability, Academic
Press, NY, 1994. 
5
Background

• Cost-benefit of usability (Marilyn Mantei and
  Toby Teorey)
• Business case approach (Clare-Marie Karat )
• Cost-justifying usability (Bias and Mayhew)
• Discount usability (Jacob Nielsen)
• Web-based applications

Bias R. G., Mayhew D. J. Cost-Justifying
Usability An Update for the Internet Age, 2nd
Edition. Morgan Kaufmann Publishers, 2005
6
Discount usability

• Le mieux est lennemi du bien
• Focuses on the good
• Reduces the costs
• Requires less experience and knowledge
• Three techniques
• Scenarios
• Simplified thinking aloud
• Heuristic evaluation

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
7
Scenarios

• Cuts down complexity by eliminating parts of full
  system
• Small and cheap to design and implement
• Paper mock-ups

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
8
Simplified thinking aloud

• Thinking aloud studies, videotape subjects,
  perform detailed protocol analysis
• Simplified version
• Take notes
• 35 test user per test

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
9
Heuristic evaluation

• Finding usability problems
• Evaluator inspect alone
• Current usability guidelines 1000 rules
• 10 basic usability principles
• Easy to learn
• Experts best
• Non-experts can find most problems
• Remaining, using simplified thinking aloud

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
10
Nielsens surveys on cost-justifying usability

• Usability costs
• 10 of the budget of each project
• Cost does not increase linearly with the budget
• Usability benefits
• 135 usability increase after redesign
• Return on investment (ROI)
• It is not linear with the costs
• The more users, the bigger ROI

Jakob Nielsen's Alertbox, http//www.useit.com/ale
rtbox/20030107.html
11
Samples of cost-justifying usability

• Australian insurance company
• Usability-enhancing project, costing lt A 100,000
• Average of 7.8 errors per form
• More than one hour to repair the errors of each
  form
• Annually saving as a result gt A 530,000
• Redesigning application forms
• IBM employees login system
• Usability project, financing 20,700
• Making sign-on attempts faster
• Savings in work time during the first day of the
  new system 41,700

Nielsen J., Usability Engineering. Academic
Press, 1993.
12
Methodology

• Cost-justifying usability is
• A conservative approach
• Part of project planning
• Mayhew and Manteis framework
• A basic framework for cost-justifying usability
• Estimating costs
• Estimating benefits
• Justifying the assumptions made for benefits

Randolph G. Bias, Deborah J. Mayhew,
Cost-Justifying Usability. Academic Press, 1994.
13
Estimating costs

• Steps
• Personnel hours and equipment costs
• Needs experience
• Additional one time setup
• Lab setup

• Usability tasks (e.g. user profile)
• Techniques (e.g. user questionnaire)

• Development of survey
• Pilot testing
• Distribution and collection
• Responding
• Coding and entering data
• Analyzing results
• Computer time
• Supplies and duplicating costs

Randolph G. Bias, Deborah J. Mayhew,
Cost-Justifying Usability. Academic Press, 1994.
14
Estimating benefits

• Increased user productivity
• Decreased user errors
• Decreased training costs
• Savings gained from making changes earlier in
  design
• Decreased user support

• Relevant audience
• Internal development organization
• Vendor company
• Relevant categories of benefits
• Estimation
• Best measurement unit
• Assuming magnitudes

• Increased sales
• Decreased customer support
• Savings gained from making changes earlier in
  design
• Reduced cost of providing training

Randolph G. Bias, Deborah J. Mayhew,
Cost-Justifying Usability. Academic Press, 1994.
15
Example

• Data entry system for 250 users
• Two to four primary screens
• Process 60 of them per day
• Users work 230 days a year
• Total cost per hour of 25
• The benefit of 1 second speedup in year
• 250 users 60 screens 230 days 1/3600 hours
  25 23,958 / year

Randolph G. Bias, Deborah J. Mayhew,
Cost-Justifying Usability. Academic Press, 1994.
16
Justifying the assumptions made for benefits

• Reasonable and likely minimum benefits
• Not precise, proven and specific
• Basing the predictions on known facts
• Published research
• Actual case histories
• Previous experience

Randolph G. Bias, Deborah J. Mayhew,
Cost-Justifying Usability. Academic Press, 1994.
17
Cost-benefit analysis of heuristic evaluation a
case study

• Case study integrating system
• For internal telephone company
• 11 evaluators
• Find 44 usability problems
• 40 core problems
• 4 from other part of the interface
• Estimating the costs (evaluation time)
• Estimating the benefits (increased usability)

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
18
Estimates of the times spent
?
?
?
Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
19
Time expenditure

• time(i) 47.8 5.2 i (Eq1)
• i number of evaluators
• Not exact for larger i
• Reduction reducing 2 observers to 1
• Revised time(i) 37.3 4.2 i (Eq2)
• Estimated hourly loaded cost 100
• Cost estimation 10,500

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
20
Benefit estimation

• Estimations of evaluators for improvements

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
21
Benefit estimation (cont.)

• Mean values
• 0.8 days -learning time reduction
• 18-expert speedup
• To be conservative
• 0.5 days - learning time reduction
• 10- expert speedup (3.3 of total work time)
• Assuming 2000 users (from 3000)
• User-days saved 1,000
• User-years saved 67 13,000 user days
• Total user days saved 14,000

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
22
Convert to monetary terms

• User-day cost of 100
• With actual saving of 50
• One-year saving 540,000
• Net present value (benefit) 500,000
• Benefit-cost ratio 48

Nielsen J., Guerrilla HCI Using Discount
Usability Engineering to Penetrate the
Intimidation Barrier http//www.useit.com/papers/g
uerrilla_hci.html
23
Thanks!

• Questions?