Training Implementation and Evaluation

1
Training Implementation and Evaluation

• Chapters
• 6-8

2
Training Methods

• Consider
• Program objectives or learning outcome (e.g.,
  skill vs. knowledge acquisition)
• Transfer of learning to job
• Development and use costs
• Trainee level of expertise novice vs. expert
• Effectiveness for certain types of learning and
  skill acquisition

3
On the Job Training (OJT)

• Job Instruction Training
• Preparing trainee, presenting tasks, practice
  time, and follow-up
• Job Rotation
• Coaching and Mentoring

4
Advantages and Disadvantages of OJT

• Advantages
• Facilitates transfer of learning to job
• Reduced training costs
• Generally well supported by organizations

• Disadvantages
• Physical constraints
• Disruption of production
• Customer inconvenience
• Potential safety concerns
• Potential damage to equipment

5
Off the Job Training Approaches

• Lecture
• Discussion
• Audiovisual media
• Experiential Methods
• Computer-based or self-paced learning
• Other E-learning Methods

6
Lecture Method

• Oral presentation of material
• Most frequently used method (gt90)
• Efficient in transferring factual information
  (e.g., theories, concepts, procedures) to a large
  group of trainees in a short amount of time

7
Problems with the Lecture Method

• One-way communication
• Trainees can become passive and bored
• Not good for changing attitudes and behaviors
• Trainees dont share ideas with each other
• How can lectures be improved?

8
Discussion Method

• Two-way communication between trainer and learner
  and between learners guided by questioning
  techniques
• Requires trainer skills in developing the right
  questions to ask, gate-keeping, and reinforcing
  learner responses
• Time and common reference point among learners
  needed to be effective

9
Audiovisual Methods

• Static media printed materials and slides
• Dynamic media videos and audios presented
  through film, tape, CD, and DVD
• Telecommunication satellite or cable programs,
  instructional TV, teleconferencing, and
  videoconferencing

10
Considerations in Use of Audiovisual Methods

• Costs production, distribution, equipment
• Content is important!
• Equipment needs and quality
• Trainer knowledge in use of these media
• Trainee needs two-way interaction, possession
  and use of appropriate equipment, trainee
  intimidation by methods

11
Experiential Methods

• Require active engagement of learners
• Examples
• Cases
• Business games and simulations
• Role plays
• Behavior Modeling
• Outdoor/team building activities

12
Considerations in Use of Experiential Methods

• Effective for developing behavioral skills (e.g.,
  interpersonal, decision making, problem solving
  skills)
• Require trainer skill in using the methodology
  and appropriate material preparation
• Learners may be intimidated by methods
• Require active participation, preparation,
  serious attitude, and respect for others among
  trainer and learners

13
E-Learning Methods

• CD-ROM DVD (32)
• Self-paced web or computer courses (60)
• Internet and intranet training (54)
• Intelligent computer-assisted instruction (AI or
  expert systems or intelligent tutoring)
• Other E-learning methods multimedia, virtual
  education networks, virtual classroom, virtual
  reality, group and project-based learning
  technologies, electronic support systems

14
Characteristics of Computer-based Learning
Technologies

• Interactivity and self-pacing
• On-demand nature anytime, anywhere
• Diagnostic assessment and evaluation
• Logistics Accessible through networks/servers
  no need for travel to training location
• Instructional management and reporting system
  (e.g., electronic portfolios)
• Administrative issues are managed and stored
  electronically

15
Impact of new technology on the learning
environment

• Technology has allowed learning to become a more
  dynamic process
• The learning environment has expanded to include
  greater interaction between learners and the
  training content
• There is greater interaction between learners and
  the instructor
• Instruction is primarily delivered to the
  learners online using the internet
• Instructor is more of a coach and resource person

16
Considerations in Use of Computer-Based Methods

• Costs development and use
• Effectiveness and efficiency of learning
• Technical issues bandwidth, browser
  compatibility, access
• Trainee attitude toward this learning technology

17
Commission on Technology and Adult Learning (2000)

• ASTD and National Governors Association convened
  commission to study adult e-learning
• Mission To define and encourage a
  technology-enabled learning environment that will
  result in an engaged citizenry and a skilled
  workforce for the digital economy

18
Commission Recommendations

• Create the highest-quality e-learning experiences
  possible
• Implement new measures and methods for assessing
  and certifying what individuals know and are able
  to do
• Ensure broad and equitable access to e-learning
  opportunities

19
Other Training Implementation Concerns

• Arrange appropriate physical environment (space,
  seating, temperature, noise, interruptions)
• Initial meeting break the ice and establish
  learning climate
• Remember the focus is on the learners, not on the
  trainer!

20
Training Methods Exercise

• Work with your training project group to assess
  the effectiveness of different kinds of methods
  to meet different kinds of training objectives
• Based on the objectives of your training program,
  what methods are most appropriate?

21
Training Evaluation

• Collection of descriptive and evaluative
  information to determine the effectiveness of
  training decisions and outcomes
• Needs to be planned out before the training takes
  place (i.e., when training objectives are being
  developed

22
Purposes of Evaluation

• Is the program accomplishing its objectives?
• What are the strengths and weaknesses of the
  program?
• What is the cost/benefit ratio?
• Who should participate in future programs?
• How should we market the program?

Goal Build credibility and establish worth of
HRD programs
23
Current HRD Evaluation Efforts

• 92 of companies perform evaluations, but these
  mostly just measure trainee reaction
• Why arent evaluations done?
• 1. When done correctly, evaluation is difficult
• 2. Afraid to find out training is not effective
• 3. Difficult to separate out the effects of
  other factors (resources, changes in the
  organization and/or environment, policies, etc.)
  on training effectiveness

24
Steps in Training Evaluation
Select evaluation criteria
Base on training objectives
Determine evaluation design
Conduct the evaluation
Interpret and post results
25
Training Evaluation Models

• Kirkpatricks Famous Framework
• 1. Reactions (Did trainees like the program?)
• 2. Learning (Did trainees learn?)
• 3. Behavior (Did trainees behavior on the job
  change as a result of training?)
• 4. Results (Has the training improved
  organizational effectiveness?)

26
Elements of All Evaluation Models

• Measure reactions of trainees (affective,
  perceived usefulness and difficulty)
• Measure learning of trainees (cognitive
  constructs such as declarative knowledge, new
  mental models, meta-cognitive skills)
• Measure behaviors of trainees (skills such as
  procedures, speed of performance, error rates,
  automaticity)
• Measure results (tangible outcomes, ROI, indirect
  outcomes, contribution to community/society)

27
Training Evaluation Practices
Percentage of Courses Using Outcome
Outcomes
28
Steps in Training Evaluation
Select evaluation criteria
Base on training objectives
Determine evaluation design
Conduct the evaluation
Interpret and post results
29
Research Design Issues

• Specifies the expected result of evaluation
  study, methods of data collection, and how data
  will be analyzed
• Quality of training evaluation results depends
  upon the quality of the research design

30
Types of Research Design
I
Training
Post test
Post test
Training
II
Pretest
Training
Pretest
III
Post test
Post test
Pretest
Control Group
31
Types of Research Design
IV Solomon Four Group
Pretest
Training
Post test
Group 1
Pretest
Post test
Group 2
Training
Group 3
Post test
Post test
Group 4
32
Other Factors that Improve Research Design

• Random assignment of training and control groups
• Matching trainees and control group members
• Collect data over multiple points in time (time
  series)
• Pool data to increase sample size for statistical
  analysis

33
Steps in Training Evaluation
Select evaluation criteria
Base on training objectives
Determine evaluation design
Conduct the evaluation
Interpret and post results
34
Data Collection Issues

• Interviews
• Questionnaires
• Direct observation and rating
• Tests and simulations
• Archival performance data

Collect data at individual, unit, and
organizational levels based on objectives
Ensure validity, reliability, and practicality
of methods
35
Ethical Issues in Training Evaluation

• Confidentiality
• Informed consent
• Withholding training
• Use of deception
• Pressure to produce positive results

36
Steps in Training Evaluation
Select evaluation criteria
Base on training objectives
Determine evaluation design
Conduct the evaluation
Interpret and post results
37
Assessing the Utility of Training

• Cost-benefit analysis compare costs of training
  program with the benefits received (both monetary
  and non-monetary)
• Costs direct costs, indirect costs, overhead,
  development costs, and participant compensation
• Benefits improvement in trainee attitudes, job
  performance, quality of work, creativity

38
Return on Investment Calculation
Return on Investment Results Training
Costs Training Costs
100,000 - 75,000 25,000 25,000 75,000
33
ROI
If ROI is negative, then training costs more than
it returns to the organization
If ROI is positive, then returns to the
organization are more than the training costs
39
Pros and Cons of Calculating ROI for Training
Programs

• Pros
• Understandable to line managers
• Credibility of HRD programs enhanced
• Useful measure if reliable sources of data and
  conservative estimates are used

• Cons
• Time and costs involved in doing ROI
• Can we quantify all results of HRD programs?
• Some may question assumptions in the calculation
• Balanced scorecard approach better (financial,
  customer, internal process, and innovation and
  learning)

40
Utility Analysis
U (N)(T)(dt)(Sdy) - C
N Number of trainees T Length of time
benefits last Dt Effect size (difference
between job performance of trained and untrained
groups, expressed in standard deviation
units) Sdy Dollar value of job performance of
untrained employees (expressed in standard
deviation units) C Costs of training
U Change in utility in terms
41
Considerations in Utility Analysis

• May be difficult to estimate values of SDy and
  length of time benefits of training last
• Assumes line managers use rational methods to
  evaluate training programs
• Need involvement of managers to estimate these
  values
• Use reasonable assumptions and conservative
  estimates
• Recognize that non-monetary benefits are not
  being measured