Unit 8

1
Unit 8 How to Evaluate

• Aim
• To provide an overview of how to effectively
  evaluate learning programmes
• Learning outcomes
• Develop evaluation strategies for your learning
  programmes
• Use a range of evaluation tools appropriate to
  your learning programmes

2
Evaluation

• Training evaluation is a bit like eating five
  portions of fruit and vegetables a day everyone
  knows that they are supposed to do it, everyone
  says they are planning to do better in the future
  and few people admit to having got it right.
• Tamkin P, Yarnall J, Kerrin M (2002)

3
What is evaluation ?

• To ascertain or fix the value or worth of
  something
• To assess or appraise
• To examine a subject or rate it
• Make a judgement

4
Why do we evaluate ?

• To identify if educational solution meets
  outcomes
• To determine the value of educational solution
• To determine what worked well and what could be
  improved
• To demonstrate the impact and value of what you
  are doing

5
Test for the Best www.test4best.scot.nhs.uk)
6
Checklist(www.test4best.scot.nhs.uk)

• Evaluating the impact of a new educational
  solution can be complex, as a wide range of
  factors may need to be considered
• Outcomes and outputs
• Think about the original analysis of the issue
  which prompted the learning programme to be
  developed.
• Think about the links between
• the issue
• the outcome i.e. the anticipated change in
  behaviour, quality, performance and efficiency of
  a service or individual
• the output i.e. the activity developed to achieve
  the outcome

7
Reviewing the links between the issue, the
outcome and the outputs

• Issue
• There has been an increase in complaints about
  the way telephone calls to a clinic are handled
• Outcome
• Elimination of complaints about call handling
• Output
• Telephone skills training for staff who take and
  make calls
• How will output address outcome?
• Clearer frameworks for call handling will help
  staff deal more effectively with routine calls
• Improved skills will help staff deal with more
  complex difficult calls

8
Checklist(www.test4best.scot.nhs.uk)

• Stakeholder consultation
• What do they want to find out from the evaluation
  ?
• This will help establish and agree the aims of
  any evaluation of impact
• Information needs
• Do you have information which will act as a
  baseline against which you will measure
  development ?
• If you dont have this information, what effect
  will it have on the aims of your evaluation ?

9
Checklist(www.test4best.scot.nhs.uk)

• Approaches to evaluation
• What type of information is needed ?
• Do you need small scale evaluation (self
  assessment, user feedback) or likely to require
  external evaluation ( strategic importance)
• How will the information be gathered ?
• Frequency and timing of evaluation ( one
  evaluation or a series of evaluations)
• Dissemination of findings
• Who needs to know about the findings of this
  evaluation ?
• How will you disseminate the evaluation report ?

10
How do we evaluate ?

• Who is the evaluation for?
• What do you need to know?
• Why do you need to know it?
• Will the evaluation make a difference?

11
Evaluation Models
12
Kirkpatricks Four-level approach(1959)
Level 4 Impact the impact of learning on
practice in terms of results within the
organisation
Level 3 Application change in job behaviour and
transfer of learning to the workplace
Level 2 Learning - the resulting increase in
knowledge or capability before and after
training
Level 1 Reaction - what they thought and felt
about the learning programme
13
Kirkpatricks Four level approach examples
Level 4 Follow up participants and managers
,productivity data look at costs, quality,
patient experience
Level 3 Observation, reflective log by
participant, CPD evidence
Level 2 Pre and post course knowledge
questionnaire, Self assessment of knowledge,
observation of performance,
Level 1 Reaction questionnaires, post its
14
Learning Programme-Mentor Update Training for
Nurses
LEVEL EVALUATION TOOL AND STRATEGY
4 Discuss with participant, students and charge nurse - How learning experience impacted on service and quality of service CPD portfolio
3 3 month post learning to evaluate theory into practice
2 Pre and post course knowledge questionnaire
1 Post it feedback 3 structured questions to whole group
15
Evaluation models Hamblins five - level
approach ( 1974)

• Level 1 Reactions
• Level 2 Learning
• Level 3 Job behaviour
• Level 4 Organisation the effects on the
  organisation, from participants job to
• performance changes
• Level 5 Ultimate value the financial effects,
  both
• on the organisation and the economy
• Hamblin AC (1974) Evaluation and Control of
  Training, Maidenhead, McGraw Hill.

16
Summary

• Evaluation is an essential component of any new
  educational development.
• Evaluation needs to be considered carefully at
  the start of any learning programme development
  and well before implementation.
• There is no right or wrong way to evaluate
• Remember
• Who is the evaluation for ?
• What do you need to know ?
• Why do you need to know it ?
• Will the evaluation make a difference ?

17
Group work

• Think about a learning programme you want/are
  delivering
• Worksheet 1
• Stakeholders
• Information needs
• Evaluation approach
• Dissemination
• Worksheet 2
• Evaluation strategy