Human Resource Development: Principles and Practice

1
13.1
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
2
13.2
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
3
13.3
MISCONCEPTIONS ABOUT EVALUATION
While evaluation is seen as professionally and
theoretically desirable, some have doubts about
its worth. 1. It is too costly. This
misconception is based on a very short-term view
of costs. The cost of risk is also part of the
equation. These risks include not knowing if
mistakes were made and not knowing how to avoid
these mistakes in the future. 2. The measures
are not exact. This misconception does not
recognise that any complex decision is based on
less than perfect information. 3. It is too
difficult. True, most evaluation recommendations
apply more obviously to the legitimate system.
However, the developmental efforts of the shadow
system can be evaluated.
Human Resource Development Principles and
Practice
By Brian Delahaye
4
13.4
THE ROLE OF EVALUATION
The role of the evaluation stage is
four-fold 1. to measure what change has
occurred 2. to improve the other three stages
investigation, design and implementation 3. to
see if the change is attributable to the learning
episode 4. to see if the amount of change is
worthwhile.
Human Resource Development Principles and
Practice
By Brian Delahaye
5
13.5
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
6
13.6
ASSESSMENT OF LEARNING

• Assessment addresses the question What changes
  have occurred in the mind of the learner as a
  result of the learning episode?. To measure this
  change, the learning has to be converted into a
  behaviour that can be observed and this has some
  weaknesses
• The behaviour is only a sample of the learning.
• The behaviour represents explicit knowledge
  only.
• The learning generation processes may take some
  time.
• The test has to tap into the appropriate
  knowledge.
• The test has to tap into the same potential
  knowledge each time.
• There are several issues that need to be
  considered when planning assessment.

Human Resource Development Principles and
Practice
By Brian Delahaye
7
13.7
TYPES OF ASSESSMENT

• Assessment can be divided into six types
• skill tests are used for procedural skills
• objective written tests e.g. multiple choice
• subjective written tests e.g. essays
• performance tests used to assess complex
  processes. The examiner may asses the process
  itself or the product often using a pre-designed
  observation form (see Fig.13.1 on page 346).
• learning diaries Journals written by the
  learner good for assessing externalisation
• portfolio assessment collections of the
  learners work takes a long term view, focuses
  on quality and the learner reflects on own work.

Human Resource Development Principles and
Practice
By Brian Delahaye
8
13.8
THE MEANING OF SCORES

• When a test has been examined and given a
  quantitative score the result is called a raw
  score.
• This raw score can be interpreted in a number of
  ways
• criterion-referenced scoring the learners
  score is compared to a pass mark
• norm-referenced scoring the raw score is
  compared to the average of a nominated group
• formative assessment the score is used for
  developmental feedback
• summative assessment the score is used for
  evidence of learning.

Human Resource Development Principles and
Practice
By Brian Delahaye
9
13.9
OVERLAP OF THE FOUR HRD PROCESSES
The assessment of learning plays a dual role
being critical to both the implementation stage
and the evaluation stage of HRD. In the
implementation stage, assessment provides dynamic
feedback. Firstly, this feedback is often
immediate. Secondly, it provides irrefutable
evidence that the learner finds difficult to
ignore. In the evaluation stage, assessment
provides initial evidence of the success or
otherwise of the learning experience. This
feedback may be taken into account immediately
(e.g. the HR developer changes the learning
processes) or later (e.g. recommendations in the
Evaluation Report).
Human Resource Development Principles and
Practice
By Brian Delahaye
10
13.10
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
11
13.11
KIRKPATRICKS FOUR LEVELS
Kirkpatricks model is based on four levels of
ascending order 1. reaction measures the
reactions of the learners to the learning
episode 2. learning measures the knowledge
gained by the learners. Has been discussed under
Assessment of Learning. 3. behaviour examines
the change of behaviour of the learner on the
job 4. results measures the impact of the
learning on the organisation as a whole Several
comments can be made about Kirkpatricks model.
Human Resource Development Principles and
Practice
By Brian Delahaye
12
13.12
COMMENTS ON KIRKPATRICKS MODEL

• The strengths of Kirkpatricks model include
• simple and seemingly robust model
• recognises the impact of a learning episode
  should extend beyond the individual learner
• provides an easily remembered checklist
• each level provides a unique examination of HRD.
• The criticisms of Kirkpatricks model include
• the levels are not co-dependent
• the model concentrates on processes and outputs,
  not inputs
• the initial level is easier but measurement of
  the deeper levels are more susceptible to
  contamination.

Human Resource Development Principles and
Practice
By Brian Delahaye
13
13.13
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
14
13.14
THE PRESAGE FACTORS
Brinkerhoffs model proposes six levels Stage
I Evaluate needs and goals ( this is similar to
the HRDNI). Stage II Evaluate the HR
design Stage III Evaluate implementation Stage
IV Evaluate learning (similar to Kirkpatricks
level 1) Stage V Evaluate usage and endurance of
learning (similar to Kirkpatricks level
3) Stage VI Evaluate pay-off (similar to
Kirkpatricks level 4). Brinkerhoff, then, adds
two that occur prior to the learning episode
evaluate HR design and evaluate implementation.
These are called presage factors.
Human Resource Development Principles and
Practice
By Brian Delahaye
15
13.15
THE PRESAGE FACTORS (Continued)

• Stage II, evaluating the HR design focuses on the
  quality of the HR plan (i.e. the training or
  workshop program).
• A stage II evaluation should occur when
• the design is unique or experimental
• the costs of mounting the program are high
• the HRD issues are crucial
• the participant groups are volatile, influential
  or demanding.
• The design should be reviewed by a variety of
  stakeholders.

Human Resource Development Principles and
Practice
By Brian Delahaye
16
13.16
THE PRESAGE FACTORS (Continued)
Stage III, evaluating during the implementation
provides data that will help shepherd the
learning event to a successful conclusion. This
highlights the covert activity of the conducting
HR developer where the program is continually
monitored to compare reality with the program
plan. The HR developer can also be monitored,
usually by another HR developer. However, the
big brother syndrome has to be avoided.
Human Resource Development Principles and
Practice
By Brian Delahaye
17
13.17
TIME OUT
To this stage of the discussion, we have examined
assessment of learning, Kirkpatricks model and
Brinkerhoffs model. This has answered two of the
roles of evaluation 1. to identify what change
has occurred 2. to improve the other three
stages. The other two roles will now be
addressed 3. to see if the change is
attributable to the learning episode. This will
be examined by looking at the SCIENTIFIC
MODELS. 4. to see if the amount of change was
worthwhile. This will be examined by looking at
COST BENEFIT ANALYSIS.
Human Resource Development Principles and
Practice
By Brian Delahaye
18
13.18
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
19
13.19
THE SCIENTIFIC MODELS

• The scientific models are based on the
  experimental methods used in research
  laboratories and are used to demonstrate
  causality. They also assume that changes can be
  measured.
• Post-test learning experience/evaluation
• Pre-test - Post-test evaluation/learning
  experience/evaluation
• Time series evaluation pre-test/pre-test/pre-te
  st/learning experience/post-test/post-test/post-t
  est
• Control group
• Experimental group pre-test/learning
  experience/post-test
• Control group pre-test/usual duty/ post-test
• The Solomon four uses three control groups and
  the experimental group
• The HR developer has to compare the costs of the
  designs compared to the expected benefits gained.

Human Resource Development Principles and
Practice
By Brian Delahaye
20
13.20
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
21
13.21
COST-BENEFIT ANALYSIS

• Cost-benefit analysis (CBA) is used to see if the
  amount of change was worthwhile. It is based on
  the premise
• identify the costs, in dollar terms, for the
  learning experience
• identify the the benefits accruing from the
  learning experience, in dollar terms
• the ratio between the costs and benefits should
  be in favour of the benefits.
• There are two problems often encountered
• converting the benefits into monetary values
• deciding on the cut-off points.
• There are two good reasons for conducting a CBA.
  Firstly, the survival of the HR section may
  depend on proving the value of development.
  Secondly, it helps in the decision of selecting
  the learning programs that have the most impact.

Human Resource Development Principles and
Practice
By Brian Delahaye
22
13.22
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
23
13.23
THE EVALUATION PLAN

• Planning for evaluation commences during the
  design stage. This ensures that the appropriate
  evaluation occurs at the appropriate time. It
  also allows the coordination of developmental and
  judgemental evaluation. Planning or evaluation
  should include
• develop the assessment for developmental
  purposes first
• incorporate this into the evaluation plus any
  further assessment required
• decide what presage variables will be evaluated
  and when
• incorporate appropriate HRDNI investigation
  instruments
• design daily and course/workshop reaction sheets
• design pre-test and post-test instruments, if
  appropriate
• identify and plan methods to be used at Stage V
  and VI
• if CBA is to be used, arrange collection of
  costs and benefits
• prepare a budget
• send evaluation plan to staff who are affected.

Human Resource Development Principles and
Practice
By Brian Delahaye
24
13.24
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
25
13.25
THE EVALUATION REPORT

• Once the evaluation has been completed a report
  should be completed for the key stakeholders. The
  evaluation plan has a communication role,
  decision making role and becomes an historical
  document. This report should include
• an executive summary
• a findings/recommendations section
• a contents list
• the main body, which should include
• reasons for the evaluation
• list of personnel involved
• discussion of the various types of evaluation
  undertaken
• a discussion on the findings, the options and
  the conclusions
• a list of, and discussion of, the
  recommendations
• appendices.

Human Resource Development Principles and
Practice
By Brian Delahaye
26
13.26
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
27
13.27
THE LEGITIMATE SYSTEM
Most of what has been discussed in this chapter
applies directly to the legitimate system. The
methods mainly rely on negative feedback systems.
The methods also emphasise the legitimate systems
focus on the efficient use of resources and the
immediate survival of the organisation. For the
legitimate system, one of the advantages of
competency-based learning has been the
recognition of the strong link between learning
in a planned program and the application of that
learning in the workplace. It should also be
recognised that, if the assessment process fails
to detect a lack of comprehension on the part of
the learners, the HR developer could be held
liable.
Human Resource Development Principles and
Practice
By Brian Delahaye
28
13.28
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
29
13.29
THE SHADOW SYSTEM
Evaluation in the shadow system is a very
delicate affair. The use, by a third party, of
the methods described in this chapter results in
negative feedback loops and this kills the
creativity needed in the shadow system. The main
evaluation method for the shadow system is the
use of values systems. The manger instills a
particular value in the system and checks for
evidence for this value. The actors in the
shadow system, however, should use the evaluation
methods on themselves and their own endeavours.
Human Resource Development Principles and
Practice
By Brian Delahaye
30
13.30
CHAPTER 13 EVALUATION

• MISCONCEPTIONS ABOUT EVALUATION
• ASSESSMENT OF LEARNING
• KIRKPATRICKS FOUR LEVELS
• THE PRESAGE FACTORS
• THE SCIENTIFIC MODELS
• COST-BENEFIT ANALYSIS
• THE EVALUATION PLAN
• THE EVALUATION REPORT
• THE LEGITIMATE SYSTEM
• THE SHADOW SYSTEM
• THE NEED FOR DIALECTIC THINKING

Human Resource Development Principles and
Practice
By Brian Delahaye
31
13.31
THE NEED FOR DIALECTIC THINKING

• Some of the misperceptions about evaluation can
  be explained by the concept of dialectic thinking
  (see Chapter 2).
• Evaluation has a number of opposing
  characteristics
• evaluation can be both developmental and
  judgemental
• the more objective the measure used, the less
  rich the insights and vice versa
• liberal education vs. vocational education.
• Evaluation serves and supports a number of dual
  roles. Balancing these conflicting roles is the
  new challenge as knowledge is a valuable but
  delicate resource.

Human Resource Development Principles and
Practice
By Brian Delahaye