Task Measurement ''' Because You Cant Manage What You Cant Measure

1
Task Measurement... Because You Cant Manage
What You Cant Measure

• September 10, 2008

2
Task MeasurementTask Performance Indicator

• Welcome
• Opening Introduction Gerry McGovern
• Task Measurement Rolf Molich
• Question Answer session

3
Audio pane select audio mode. In the Audio
pane, select either Use Telephone or Use Mic
Speakers. By default, you will be joined into the
Webinar muted. If you select Use Telephone, be
sure to enter the Audio PIN noted in your Control
Panel. If you select Use Mic Speakers you can
join the Webinar via VoIP. You will need speakers
to listen to the Webinar and a microphone to
speak. Click Audio Setup to select Mic Speakers
devices. You can switch between using VoIP (Mic
Speakers) or Telephone during the Webinar
session.
4

• Questions Answer Session
• Use the Question pane.
• To submit text questions and review answers.
  Broadcast messages to attendees will also show
  here.
• Click Hand icon to raise/lower hand. If asked to
  speak you will be un-muted and you can ask you
  question.

On exiting the webinar you will be prompted to
complete a survey. We welcome your feedback.
5
Our Speakers

• Gerry McGovern

• Rolf Molich

www. dialogdesign.dk www.customercarewords.com
www.gerrymcgovern.com www.customercarewords.com
6
Task Measurement... Because You Cant Manage
What You Cant Measure

• September 10, 2008

Gerry McGovern gerry_at_customercarewords.com
7
(No Transcript)
8
Sample Task

• What are your companys rules for maternity leave
  in your country?

8
9
Search
More search options Help
Intranet Home Class B Class C About Me
About Company
Expected1st click
Homepage
10
Search
More search options Help
Intranet Home Class A Class B About
Me About Company
About Me
ABOUT ME Training Job Vacancies Travel Time and
Payroll Employee Benefits Environment (Working,
Local) Career Development
Expected 2nd click
11
Search
More search options Help
Intranet Home Class B Class C About Me
About Company
Homepage
Actual1st click
12
Search
More search options Help
Intranet Home Class A Class B About
Me About Company
About Me
ABOUT ME Training Job Vacancies Travel Time and
Payroll Employee Benefits Environment (Working,
Local) Career Development
Actual 2nd click for 6/15 users
13
Search
More search options Help
Intranet Home Class A Class B About
Me About Company
About Me
ABOUT ME Training Job Vacancies Travel Time and
Payroll Employee Benefits Environment (Working,
Local) Career Development
Only 1/15 clicked here
14
Task 7 - Employee Policies
Task What are the companys rules for maternity
leave in your country? Overall success Solved
task in first attempt - Solved task in second or
third attempt 13 Unable to solve task 1, 2, 3,
4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15 Expected
paths A. About Me gt Time Payroll 13 Alternative
paths B. About Company gt Org. gt HR gt HR
Info. 1, 15 C. About Company (somewhere) 2, 3,
4, 12 D. About Me gt Compensation Benefits 4,
7, 9, 11, 12 E. About Me gt Environment
(Working, Local) 5, 6, 7, 9, 10, 14
14
15
Task Measurement... Because You Cant Manage
What You Cant Measure

• Presentation of Method
• September 10, 2008

Rolf Molich rolf_at_customercarewords.com
16
Does this baby get even prettier over time?
Opinion based answer.
17
Does this baby get longer over time? Objective
answer. We can track progress, because we can
measure it.
18
Is this a usable intranet? Opinion based
answer. How long does it take to find the rules
for maternity leave? Objective answer.
19
Why Measurement?

• You cant manage what you cant measure.
• -- Tom DeMarco

20
Users Time is Precious

• Websites are a product which you ask people to
  buy, using their time and attention as currency.
• -- Tom du Pré

21
Managing a Website

• Managing a website is about managing users tasks
• Identify top user tasks for Internet, Intranet,
  desktop application, etc.
• Develop performance measures and benchmarks based
  on user task performance
• Assign responsibility for the success of each top
  user task to staff (task manager)
• Report to senior management on task metrics
• Set annual task performance improvement targets

22
Task MeasurementTask Performance Indicator (TPI)

• Determine top tasks
• Recruit representative users
• Measure Users carry out tasks
• Compute and report results
• Improve task performance
• Measure again.

23
Step 1 Determine Tasks

• Determine 10 (or more) frequently used tasks.
• The chosen tasks must be frequent so improving
  them significantly will have a measurable impact
  on the efficiency of the organization.

24
Step 1 Determine Tasks

• Tasks are selected based on
• Result of Customer Carewords study
• Most used pages
• Key search terms
• Customer Carewords standard tasks (this allows
  inter-company comparisons)

25
Step 1 Determine Tasks

• Sample Find Person Tasks
• Where is John McGuires office?
• Find an expert on investing in IBM.
• Who is responsible for personal banking in
  Uruguay?

26
Step 2 Recruit Participants

• Recruit 15-20 users who are a representative mix
  of your employees.
• Getting the participant mix right is vital in
  ensuring that we get results that reflect the
  true nature of carrying out the identified tasks
  within the organization.

27
Step 3 Measurement Sessions
28
Step 3 Measurement Sessions

• Ask each of the 15-20 users to carry out the
• chosen tasks one by one as quickly as possible.
• We are not measuring you
• Measure time for each task.
• Observe usability problems and successes.
• Ask about frequency.
• Similar to traditional usability test, except no
  think aloud and no discussions

29
Step 3 Measurement Sessions

• An expert observes what happens.
• Each measurement session takes 30 to 90 minutes
  and comprises 10 to 20 tasks.
• Measurements are always done in individual
  one-on-one sessions.

30
Step 4 Compute and Report Results
31
Step 4 Compute and Report Results

• Result Success
• User completes task successfully.
• We register the time it took.

32
Step 4 Compute and Report Results

• Result Failure
• User gives up.
• Task is not completed within 6 minutes (for
  small tasks).
• Technical problems prevent the user from
  completing the task.

33
Step 4 Compute and Report Results

• Result Disaster
• User completes task but the result is wrong.

34
Step 4 Compute and Report Results

• Example of disaster
• Users were asked to find out what percentage of
  customers are considered reluctant according to
  customer satisfaction studies.
• Some users found a page that said that the
  percentage of reluctant customers was 24.
  However, this figure was outdated.
• The right figure was 14.

35
Step 4 Compute and Report Results

• Use all measurements for a task to compute
• Completion rate - Effectiveness
• Completion time - Efficiency, Median time for
  task
• Disaster rate User finds wrong answer, but
  believes it is correct

36
Step 4 Compute and Report Results

• Optimal time
• Projected time for task with
• An improved but realistic user interface
• Realistic response times

37
Step 4 Compute and Report Results

• Task Performance Indicator (TPI)
• 
  Optimal time for task
  Median measured time for task
• The Success factor factors in the effect of
  failing and disastrous task attempts. It is 100
  if all participants manage to solve the task
  correctly within 6 minutes. It is 50 if only
  half of the participants manage to solve the
  task, etc.

TPI Success factor x
38
Step 4 Compute and Report Results

• TPI example
• Optimal time for task 40 secs
• Median time for users to carry out task 100 secs
• Optimal time / Median time 40

39
Step 4 Compute and Report Results

• TPI example
• Optimal time for task 40 secs
• Median time for users to carry out task 100 secs
• Optimal time / Median time 40
• 9 of 15 participants complete the task 60
• Task Performance Indicator 40 of 60 24

40
Step 5 Improve Task Performance

• You apply the usability advice offered in
  Customer Carewords TPI report
• General or specific usability advice,
• Optimal solution patterns based on best practice.

41
Step 6 Measure Again

• Measure again, for example 1 year later.
• Compute improvement in completion rate,
  completion time and disaster rate.

Measure
Improve
Analyse
42
We expected that most participants would be able
to complete the key tasks we gave them so we
could get valid measurements.
43
We see again and again that a significant number
of key task attempts fail completely. Users give
up and find other means to get the answer.
44
Main Reasons for Failures, Disasters

• Search
• Navigation
• Freshness

45
Potential Productivity Gains
Fact Existing Customers have identified
potential annual savings to management ranging
from 15,000,000 to 35,000,000. (20M-50M US
Dollars).
Potential productivity gains with 36,000
employees
46
Task Measurement Highlights

• Management tool Which task improvement
  investments will pay off?
• Focus on task performance rather than tool
  performance or system performance
• Systematic treatment of task failures and
  disasters
• All-in-one Task Performance Indicator (TPI)
• Suggested optimal task flow defines clear goal

47
Questions Answers

• There are two ways to submit a question
• Type your question in the Questions panel.
• Press Raise your Hand and you will be invited
  to speak to the panel directly.

Would you like to know more? Visit our website
www.customercarewords.com or Email
andrew_at_customercarewords.com Customer Carewords
has a network of partners across North America,
Europe and Asia/Pacific. Our partners have deep
experience with creating customer-centric,
task-managed websites utilizing Customer
Carewords. Contact us to talk to a partner.