By Jack J. Phillips, Ph.D.

1
The Bottom Line Return on Investment (ROI) Model
for Business Engagement
ByJack J. Phillips, Ph.D.
2
Agenda

• Review current status of ROI implementation
• Explore the myths and mysteries of ROI
• Identify the key steps involved in the ROI
  methodology
• Examine the key issues involved in ROI
  implementation

3
Paradigm Shift

• Results Based
• program linked to specific business needs
• assessment of performance effectiveness
• specific objectives for behavior and business
  impact
• results expectations communicated to participants

• Activity Based
• no business need for the program
• no assessment of performance issues
• no specific measurable objectives
• no effort to prepare program participants to
  achieve results

4
Paradigm Shift
Results Based
Activity Based
5
Key Measurement Issues

• Time?
• Costs?
• Data Collection?
• Management Support?
• Skills
• Other Issues?

6
Why ROI?

• ROI is the ultimate level of evaluation
• ROI is a concept familiar to most managers
• ROI has a rich history of application
• Many top executives are requiring ROI information
• The ROI issue cannot be ignored!

7
The ROI Process

• Generates six types of data
• Reaction, Satisfaction, and Planned Actions
• Learning
• Application and Implementation
• Business Impact
• Return on Investment
• Intangible Measures
• .and includes a technique to isolate the effects
  of the program or solution.

8
ROI by the Numbers

• Process refined over a 20-year period
• Thousands of impact studies conducted each year
• 100 case studies published
• 2,000 individuals certified
• 15 books developed to support the process
• 500 member professional network
• ROI Process adopted by hundreds of organizations
  in 38 countries

9
Payoff of the ROI Methodology

• Aligns programs to business needs
• Show contributions of selected programs
• Earn respect of senior management/ administrators
• Build staff morale
• Justify/defend budgets

10
Payoff of the ROI Methodology

• Improve support for human resources/
  learning/development
• Enhance design and implementation processes
• Identify inefficient programs that need to be
  redesigned or eliminated
• Identify successful programs that can be
  implemented in other areas

11
ROI Applications

• Apprenticeship Training
• Associate Relations Programs
• Career Development Programs
• Competency Systems
• Diversity Programs
• E-Learning
• Executive Coaching
• Executive Education
• Gain Sharing Programs

• Global Leadership
• Organization Development
• Orientation Systems
• Recruiting Strategies
• Safety Health Programs
• Self-Directed Teams
• Skill-Based/Knowledge-Based Compensation
• Technology Implementation
• Total Quality Management
• Wellness/Fitness Initiatives

12
Private Sector Organizations Using the ROI Process

• Allstate Insurance
• Amazon.com
• Apple Computer
• ATT
• Bristol-Myers Squibb
• Coca Cola
• Comcast
• Dell Computers
• Deloitte Touche
• Delta Airlines
• DHL Worldwide Express
• Federal Express
• General Motors
• Georgia Pacific
• Hewlett Packard

• Hilton Hotels
• Home Depot
• Intel
• Lockheed Martin
• Microsoft
• Motorola
• NCR
• Nextel
• Olive Garden Restaurant
• PricewaterhouseCoopers
• Shell Oil
• Singapore Airlines
• TD Bank
• Verizon Communications
• Wachovia Bank

13
Government Organizations Using the ROI Process

• US Department of Defense
• US Internal Revenue Service
• US National Security Agency
• US Department of Veteran Affairs
• NASA
• State of New York
• State of Mississippi

• State of Texas
• City of Richmond, Virginia
• Government of New Zealand
• Government of Singapore
• Government of Italy
• Government of Australia
• Provincial Government of Ontario, Canada

14
An ROI Quiz
15
Appropriate Interventions

• Performance Improvement
• Training and Learning Solutions
• Organization Development
• HR Programs
• Change Initiatives
• Technology Implementation
• Meetings and Events

16
Movement Within the Sectors
Manufacturing Sector
Service Sector
Health Care Sector
Non-Profit Sector
Public Sector
Educational Sector
17
Public Sector Issues

• Absence of revenues and profits
• Absence of hard data
• Whose ROI?
• Government services are essential
• Restricted range of options to correct problems

18
Organizational Characteristics

• Size of the organization
• Visibility of the organization
• Magnitude of budget
• Focus on measurement
• Internal drivers for accountability
• Pace of change

19
ROI The Pieces of the Puzzle
An Evaluation Framework
Case Applications and Practice
Implementation
Operating Standards and Philosophy
A Process Model
20
Evaluation Levels Defined
21
THE ROI PROCESS Calculating the Return on
Investment of a Business Performance Solution
Evaluation Planning
Data Collection
Level 1 Reaction, Satisfaction, and Planned
Actions
Level 3 Application/ Implementation
Develop Objectives of Solution (s)
Develop Evaluation Plans and Baseline Data
Collect Data During Solution Implementation
Collect Data After Solution Implementation
Level 2 Learning
Level 4 Business Impact
22
Tabulate Costs of Solution
Data Analysis
Reporting
Isolate the Effects
Convert Data to Monetary Value
Calculate the Return on Investment
Generate Impact Study
Level 5 ROI
Identify Intangible Measures
Intangible Measures
23
Characteristics of Levels

• Chain of Value of Customer Frequency Difficulty
  of
• Impact Information Focus of Use Assessment
• Satisfaction Lowest Consumer Frequent Easy
• Learning
• Application
• Impact
• ROI Highest Client Infrequent Difficult

Customers Consumers The customers who are
actively involved in the process. Client The
customers who fund, support, and approve the
project
24
Linking Assessment with Evaluation
Needs Program Assessment Objectives Evaluation
Business Impact Business Needs Objectives Impa
ct
4
4
Job Performance Application Application Needs O
bjectives
3
3
Skills/Knowledge Learning Learning Needs Objec
tives
2
2
Preferences Satisfaction Reaction Objectives
1
1
25
Collecting
Level 3 ? ? ? ? ? ? ? ? ?
Level 4 ? ? ? ? ?

• Follow-Up Surveys
• Follow-Up Questionnaires
• Observation On the Job
• Interviews with Participants
• Follow-Up Focus Groups
• Program Assignments
• Action Planning
• Performance Contracting
• Program Follow-Up Session
• Performance Monitoring

26
Isolating

• Use of a control group arrangement (35)
• Trend line analysis of performance data (15)
• Use of forecasting methods of performance data
  (5)
• Participants estimate of program impact
  (percent) (50)
• Supervisors estimate of program impact (percent)

27
Isolating

• Managements estimate of program impact
  (percent)
• Use of previous studies
• Subordinates report of other factors
• Calculating/Estimating the impact of other
  factors
• Use of customer input

28
Converting

• Converting output to contribution - standard
  value
• Converting the cost of quality - standard value
• Converting employees time
• Using historical costs
• Using internal and external experts

29
Converting

• Using data from external databases
• Linking with other measures
• Using participants estimates
• Using supervisors and managers estimates
• Using staff estimates

30
Tabulating

• Needs Assessment (Prorated)
• Development Costs (Prorated)
• Program Materials
• Instructor/Facilitator Costs
• Facilities Costs

31
Tabulating

• Travel/Lodging/Meals
• Participant Salaries and Benefits
• Administrative/ Overhead Costs
• Evaluation Costs

32
Intangible Benefits

• Increased Job Satisfaction
• Increased Organizational Commitment
• Improved Teamwork
• Improved Customer Service
• Reduced Complaints
• Reduced Conflicts
• Reduced Stress

33
Return on Investment
Benefits / Program Benefits Costs
Ratio Program Costs Net
Program Benefits ROI
Program Costs
34
Calculating ROI

• BCR ???
• ROI X 100 ???

240,000 80,000 Total Benefits Program
Costs Program Costs
35
Calculating ROI

• BCR 3.00
• ROI X 100 200

240,000 80,000 240,000 - 80,000 80,000
36
ROI Target Options

• Set the value as with other investments, e.g.
  15
• Set slightly above other investments, e.g. 25
• Set at break even - 0
• Set at client expectations

37
When properly implemented, high ROI values can be
achieved with programs on

• Leadership
• Team Building
• Management Development
• Supervisor Training
• Sales Training

. . . 100 to 700 ROI is not uncommon
38
ROI Best Practices

• Implemented as a process improvement tool not a
  performance evaluation tool
• Impact studies are conducted involving 5-10 of
  the programs
• A variety of data collection methods is used
• The effects of learning/development are isolated
• Business impact data are converted

39
ROI Best Practices

• ROI evaluation targets are developed
• The ROI methodology generates a micro level
  scorecard
• Data are being integrated to create a macro
  scorecard
• Implemented for about 3-5 of the budget
• Routine ROI forecasting
• Used as a tool to strengthen/improve

40
Building a Scorecard

• Provides macro-level perspective of success
• Serves as a brief report versus detailed study
• Shows connection of trainings contribution to
  business objectives
• Integrates various types of data
• Demonstrates alignment between programs,
  strategic objectives, and operating goals

41
Micro Level Scorecard
Macro Level Scorecard
0 1 2 3 4
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3
42
Seven Categories of Data

• Indicators
• Reaction and Planned Action
• Learning
• Application
• Business Impact
• ROI
• Intangibles

43
Scorecard ReportingFirst University
44
Key Implementation Actions

• Determine/establish responsibilities
• Develop skills/knowledge with ROI
• Develop transition/implementation plan
• Conduct ROI studies
• Prepare/revise evaluation policy/procedures/
  guidelines
• Train/brief managers on the ROI Process
• Communicate progress/results

45
Cost-Saving Approaches

• Plan for evaluation early in the process
• Build evaluation into the process
• Share the responsibilities for evaluation
• Require participants to conduct major steps
• Use short-cut methods for major steps
• Use sampling to select the most appropriate
  programs for ROI analysis

46
Cost-Saving Approaches

• Use estimates in the collection and analysis of
  data
• Develop internal capability to implement the ROI
  process
• Streamline the reporting process
• Utilize web-based software to reduce time

47
Recommended Targets
100
Level 1 - Reaction
60
Level 2 - Learning
30
Level 3 - Application
10
Level 4 Business Impact
5
Level 5 - ROI
48
Criteria for Levels 4 5

• Expected life cycle of the program
• The importance of the program in meeting the
  organizations goals
• Cost of the program
• Visibility of the program
• The size of the target audience
• Extent of management interest

49
What Happens If We Do Nothing?

• Budget?
• Influence?
• Support?
• Other Issues?

50
ROI Reality

• ROI information is desired by clients/executives
• The ROI process provides a balanced, credible
  approach with six types of data
• All types of organizations are routinely using
  the ROI process
• The ROI process can be implemented without
  draining resources
• The ROI process is a long-term goal for many
  organizations

51
Status of Major Learning and Development Metrics
52
In Training Magazines 2004 Top 100 survey, 96
measure through Business Impact and 75 measure
through ROI
53
Program Analysis Dominates
88
78
80
63
27
11
Satisfaction of Training Participants
Impact of Training on Performance
ROI of Training
Organizations Utilizing Specific LD
Metrics Organizations Wanting Specific LD Metrics
54
ROI in training is a fast growing metric . . . .
78 of companies have ROI on their wish list
Corporate Leadership Council 2002
55
Sample of Published ROI Studies