Evaluation of Training

1
Evaluation of Training

• Chapter 8

2
Learning Objectives

• By the conclusion of this discussion you should
• Understand the different types of evaluation that
  should be collected and how to collect them.
• Know the basic design issues associated with the
  four levels of outcome evaluation.
• Be aware of the various threats to internal and
  external validity of data.
• Be able to construct a cost/benefit analysis and
  an evaluation plan for your proposed training
  program

3
Why Evaluate?

• To justify the existence of the training
  department
• To decide whether to continue or discontinue
  training program
• To gain information on how to improve future
  training programs.

4
Evaluation Data

• Process (Formative) evaluates the training
  design, development, and delivery.
• Before
• During
• Outcome Data (Summative) evaluates how well
  training accomplished its objectives.
• Reaction, Learning, Behavior, Results

5
Reaction Data

• Determine what you want to find out
• Design a form (quantify)
• Encourage written comments/suggestions
• Get 100 immediate response
• Get honest responses
• Develop acceptable standards
• Measure reactions against standards and take
  appropriate action
• Communicate reaction as appropriate

6
Learning

• Use a control group if practical
• Evaluate knowledge, skills, and/or attitudes both
  before and after program
• Design differently for declarative, procedural
  and strategic knowledge as well as skill and
  attitude
• Get 100 response
• Use the results of the evaluation to take
  appropriate action
• Communicate results as appropriate.

7
Behavior

• Use a control group if practical
• Allow time for behavior change to take place
• Evaluate both before and after program if
  practical
• Survey and/or interview, documentation review,
  etc.
• Get 100 response or a sampling
• Repeat the evaluation at appropriate times
• Consider cost versus benefits
• Communicate as appropriate

8
Organizational Results

• Use a control group if practical
• Allow time for results to be achieved
• Measure both before and after the program if
  practical
• Repeat the measurement at appropriate times
• Consider cost versus benefits
• Be satisfied with evidence if proof is not
  possible.
• Communicate as appropriate

9
Evaluating Costs

• Determine if Results where worth the Costs!!
• Cost-Benefit
• Cost-Effectiveness
• Cost savings analysis
• Utility analysis

10
Actual Costs

• Development Costs
• Materials (20 trainees x 100) 2,000
• Trainer/developer time (40hrs _at_ 80) 3,200
• Direct Costs
• Trainers time (24hrs _at_ 80) 1920
• Food/beverage (20 trainees _at_ 20 each) 400
• Material/equipment
• (projector rental, markers, etc.) 250
• Indirect Costs
• Marketing 100
• Administration (20 hrs _at_ 10) 200
• Participant Compensation
• 20 employees (24hrs _at_ 10) 4,800
• Evaluation Costs
• Evaluators time (40 hrs _at_ 80) 3,200
• Materials 250
• Total 16,320

11
Cost/Benefit Analysis

• Results Increased productivity by 20
• In an regular 8 hour day, each employee now
  produces 96 parts instead of 80 parts.
• (8hrs 10)/80 parts 1.00 per part
• (8hrs 10)/96 parts .83 per part
• .17 addition profit annualize over 1 years
  production of 26,880 4,570 per employee
• (20 4570) 91,400

12
Return on Investment

• ROI benefit/cost
• ROI 91,400/16,320
• 5.6 or 560 during 1st year
• Whenever ROI is greater than one you achieved a
  positive return on your investment.

13
Who is Interested in Outcomes?
14
Validity

• Internal proof that performance improvements
  are related to training program
• External proof that training will be equally
  effective for other groups yet to attend the
  training
• Must have internal validly first

15
Threats to Internal Validity

• History
• Maturation
• Testing
• Instrumentation
• Statistical Regression
• Selection

• Mortality
• Diffusion of Training
• Compensation Treatments
• Compensatory Rivalry
• Demoralized Control Group

16
Threats to External Validity

• Testing
• Selection
• Reaction to Evaluation
• Multiple Techniques

17
Basic Evaluation Designs

• Post-test Only
• Limited uses
• Hard to assess change
• Pre-test/Post-test
• Can demonstrate change

18
Complex Evaluation Designs

• Post-test with Control Group
• Pre-test/Post-test with Control Group
• Time Series (with or without control group)
• Multiple Baseline
• Solomon 4 Group

19
Conclusion

• Evaluation of training is necessary and if done
  correctly can provide internally and externally
  valid data.
• Evaluation is complex and must be designed to
  provide the data your organization requires from
  it.
• Cost analysis is a necessary part of training
  evaluation.