Bottomline Evaluation: Measuring Results from Training and Performance Improvement

1
Bottom-line EvaluationMeasuring Results from
Training and Performance Improvement

• Tuesday May 19, 2003
• 800-930 a.m.
• ASTD International Conference
• Donald J. Ford, Ph.D., CPT
• Training Education Management
• Redondo Beach, CA 90277

2
Biography

• Donald J. Ford, Ph.D., C.P.T., is president and
  owner of Training Education Management, a
  training and performance improvement consulting
  firm in Redondo Beach, California. Dr. Ford also
  teaches graduate courses in Human Resource
  Development for Cal State Northridge and Antioch
  University, L.A. His consulting clients include
  ASTD, Toyota, Nissan Motors U.S.A., Rockwell
  International, U.S. Marines Corps, Orange County
  Transportation Authority, Glendale Memorial
  Hospital, and National Education Corp., among
  others. He has worked in the field of human
  resource development for eighteen years,
  including training management positions at
  Southern California Gas Company, Magnavox,
  Allied-Signal and Texas Instruments. He possesses
  another eight years of experience in public
  education at the secondary, adult and college
  levels.
• He holds a B.A. and M.A. in history and a Ph.D.
  in education, all from UCLA. He has published 35
  articles and three books on topics in training,
  education and business. He edited a book
  entitled, In Action Designing Training Programs
  published by ASTD in 1996. His third book
  entitled, Bottom-Line Training How to Design and
  Implement Programs That Boost Profits, was
  released by Gulf Publishing in July, 1999 and
  forms the basis for this presentation. Dr. Ford
  has presented at the ASTD International
  Conference four times, and at ASTD L.A.
  conferences and workshops.
• He may be reached at
• 1874 Pacific Coast Highway, Suite 205
• Redondo Beach, CA 90277
• Telephone/Fax [phone]
• E-mail donaldford_at_earthlink.net
• Web www.trainingeducationmanagement.com

3
Bottom-Line Training Design Model
Results

• Learning
• Performance
• Financial
• Strategic

4
Kirkpatrick Evaluation Model
Training Environment

• Level 1 Reactions
• Learner
• Client

• Level 2
• Learning
• Learner
• Organization

Learning Event
Work Environment

• Level 4
• Results
• Performance
• Financial

• Level 3
• Job Behavior
• Learner
• Organization

5
Typical Evaluation Points
Analyze
Design/ Develop
Implement
Evaluate

• Evaluation Design
• Tests
• Surveys
• Formative Evaluation

• Skills Transfer
• Performance
• Business
• Results
• Data Analysis
• Evaluation Reports

• Baseline Data
• Learners Performance
• Skill Gaps
• Business Measures

• Reactions
• Pre-Test
• Post-Test

6
Two Types of Evaluation

• Formative
• Improve the process the program
• Focus on process
• Measures
• Acceptance
• Satisfaction
• Quality

• Summative
• Sum up the impact, Go-No Go
• Focus on results
• Measures
• Performance change
• Business results
• ROI

7
HPI Evaluation Model
HPI Process
Business Analysis
Intervention Implementation
Performance Analysis
Cause Analysis
Intervention Selection
Evaluation
Business Goals
Performance Gaps
Root Causes
Project Management
HPI Outputs
Change Management
Solutions
Evaluation Outputs
Business Measures
Performance Measures
Evaluation Design
Performance and ROI
Analysis Evaluation
Customer Satisfaction
Evaluation Baseline
Summative Evaluation
Formative Evaluation
Evaluation Process
8
(No Transcript)
9
Figure 21-7Interrupted Time Series Design
T
M2
M5
M1
M4
M6
M3
M1- 6 measures at regular intervals T
training

• Examples
• Accident rates pre and post safety training
• Sales revenues pre and post sales training
• Error rates pre and post quality training
• Turnover rate pre and post management training

10
A Training Parable

• One day in Training Land, the Senior
  Vice-President of the companys largest division
  came to request training for all of his 2,000
  employees. His reason is that the company is
  about to launch a new line of products and
  services and he fears employees will not be able
  to deliver these successfully without being
  retrained. Training needs to start in six months.
  
• The content must include product features and
  benefits, sales techniques, customer service and
  a customer information system.
• Your assignment is to design an evaluation plan
  for this request.

11
Figure 24-1Bottom-Line Impact of Training
Strategic Growth
Revenues
Savings
Cost Avoidance
Training
12
Sales Training Impact
Cancer Drug Sales as of Market Share
Monthly Sales

• What was the impact of the sales training on
  market share?
• If the total annual market for these drugs is 10
  million, what was the financial benefit of the
  sales training?

13
Figure 23-4 Productivity Gains Caused by
Workplace Literacy Training Program
Productivity Level ( of Ideal)

• What conclusions can you draw from from this
  chart about the impact of the training?
• If each percentage increase in productivity was
  worth an average of 5,000 what would be the
  total value (benefit) of this training?

Training Group Monthly Productivity Average
Jan Feb Mar Apr May Jun Jul
Aug Sep Oct Nov Dec
14
Training Return on Investment (ROI)

• Total benefits total costs
• Example
• 750,000 total benefits
• 187,500 in total costs
• 750,000/187,500 41
• Company receives 4 in return for each 1 spent
  or a return of 400

15
ROI Example

• What is the ROI of this training?

16
Evaluating Behavior, Performance Skill Transfer

• Success factors
• Learned new skills
• Desire to change
• Supportive work climate
• Supportive supervision
• Lack of barriers
• Rewards and incentives

17
Performance Measurement Methods

• Action planning
• Pre-post surveys
• Pre-post interviews
• Job observation
• Performance documentation
• Output
• Quality
• Cost
• Time
• Behavior

18
Figure 23-1 Sample Action Plan for Customer
Service Training

• Summary of Skills Learned
• In this course, I learned the following new
  customer service skills
• How to greet customers.
• How to identify customer needs.
• How to locate customer account information in the
  customer database.
• How to make simple changes to an account, like
  updating the customers name, address, applying
  payments, waiving late charges and fees.
• How to handle customer complaints and document
  problems.
• Ways the New Skills Can be Applied
• Once I return to work, I can apply these new
  skills in the following ways
• Use the standard customer greeting I learned in
  class with every customer.
• Ask questions of the customer, following the
  Inquiry Matrix I learned about in class.
• Listen actively to identify the customers real
  need.
• Use the Customer Database (CD) to access account
  number in the system.
• Use the system features to update the customers
  account.
• Handle any complaints by using the Customer
  Complaint Resolution Job Aid.

19
Sample Action Plan for Customer Service Training,
continued

• Obstacles or Barriers to Skills Application
• Barriers include
• Lack of time to practice on the job.
• Waiting time for system to access account
  information.
• Some customers do not speak English well and are
  hard to understand.
• Some customers are already so upset by the time
  they call that it is difficult to solve their
  problem.
• Enablers for Skill Application
• Enablers include
• My supervisor wants met to succeed and is willing
  to give me time to learn the new skills.
• My co-workers are willing to help me and answer
  questions when they have time.
• I like helping people and will try to satisfy
  upset customers.
• The computer system is being improved to speed up
  access time.

20
Sample Action Plan for Customer Service Training,
continued

• Specific Steps To Be Taken and Deadlines
• Step Deadline
• Use standard greeting on all calls. Tomorrow
• Begin using the Inquiry Matrix on all
  calls. Tomorrow
• Practice looking up customer information. Tomorrow
  
• Become proficient on the Customer Database
  System. One month from now
• Use the Customer Complaint Resolution Job Aid to
  One Month from now
• resolve all customer complaints.
• Follow up Actions to be Taken to Review Progress
• Meet with my boss to discuss this action plan
  within one week.
• Begin using the job aids immediately to help me.
• Ask co-workers for help when I need it.
• Register to attend another customer service class
  next month.

21
District Manager Training Pre-Post Survey

• What conclusions would you draw about the
  trainings impact on supervisors job behavior?
• What recommendations would you make about changes
  to the program based on this data?
• How could you translate this data into financial
  results?

22
Balanced Scorecard

• Training Activity
• PayrollUS Avg.1.9
• /Employee/yearUS Avg.569
• Hrs./Empl./yearUS Avg. 36
• Training Efficiency
• Cost/HourUS Avg.28
• Billable Hrs.US Avg.70

• Training Results
• Positive Course RatingsAvg.91
• Learning GainAvg.45
• Behavior ChangeAvg.25
• Revenues/EmployeeAvg.100K
• Market/Book Avg.101

SourceAuthors Study of High Tech firms
Source ASTD Training Data Book, 1996.
23
Your Scorecard

• Training Activity
• Payroll_____
• Training budget/Payroll
• Hrs./Empl.____
• Training hours/Employees
• /Empl. _____
• Training budget/Employees
• Training Efficiency
• Cost/Hr. ______
• Training budget/ training hours
• Billable ____
• Billable hours/staff hours

• Training Results
• Positive Ratings
• Positive ratings/total ratings
  _________________
• Learning Gain
• Post-pre / Pre-test scores
• _________________
• Performance Gain
• Post-pre / Pre-performance
• _________________
• Revenues/Employee
• Total revenues/employees
• _________________
• Market/Book Ratio
• Total market worth/ total physical assets

24
Training Design Job Aid

• Analysis Phase
• Needs Analysis
• Performance Analysis
• Needs Assessment
• Job Analysis

• Task Analysis
• Learner Analysis
• Context Analysis
• Skill Gap Analysis

• Design Phase
• Training Objectives
• Deliverables/Strategies
• Budgets/Schedules

• Project management
• Blueprints/Prototypes

• Development Phase
• Draft Materials
• Tests/Assessment

• Quality Control
• Full-scale Production

25
Training Design Job Aid, continued

• Implementation Phase
• Train the Trainer
• Classroom Delivery

• Non-Classroom Delivery

• Evaluation Phase
• Formative
• Reactions
• Learning

• Transfer to Job
• Results/ROI

26
Closing commentsand QA

• Donald J. Ford, Ph.D., CPT
• Training Education Management
• Redondo Beach, CA 90277
• Telephone/Fax [phone]
• E-mail donaldford_at_earthlink.net
• Web www.trainingeducationmanagement.com