Principles of Performance Monitoring

1
Principles of Performance Monitoring
Evaluation

• Steven Marwick
• Director
• Evaluation Support Scotland

2
What I will cover

• Views on evaluation the alternative survey
• What is evaluation?
• The evaluation plan steps to success
• Evaluation Support Scotland areas of work and how
  we can help

3
The alternative survey

• A short presentation to illustrate attitudes to
  evaluation and common problems in a humorous
  way.
• The slides are not in your pack because it would
  spoil the impact.
• If you really want to see them again you can get
  them on our website.

4

• What do you think?
• And what do you know?!

5
What is evaluation?

• Evaluation is a process of asking questions
• ?
• gathering evidence
• ?
• analysing it
• ?
• being prepared to act on the results.

6
Evaluation quiz!

• 1. What difference or changes are you hoping to
  make?

7
Quiz - 2

• What things would tell you that youve achieved
  your outcomes?

8
Quiz - 3

• How will you measure your indicators?

9
Thinking about evaluation

• Decide why I need to evaluate (and for whom).
• Clarify the changes I want to make (outcomes) and
  how they link to activities.
• Agree what will show were making those changes.
• Decide what information to collect about my
  activities and how to collect it.
• Decide what information to collect about outcomes
  and how to collect it.
• Decide what the information tells me about what
  weve have done and whether it made a difference.
• Decide what should we do with that information.
  Are there any learning points and who else I need
  to tell?

10
Why evaluate?

• Evaluation is for
• ? LEARNING
• ? ACCOUNTABILITY

11
Whats in it for me?

• ?SORP ?Good governance
• ?Making the service better ? VFM
• ? Help fundraising ? Feedback to users
• ? Feedback to funders ? Meeting risk
• ? Motivation ? Influencing others

12
Weavers Triangle
13
McDeeshire Carers Project
14
Indicators

• The things that help you decide if you are
  delivering your plan.
• Output indicators about
• activities / numbers / money
• Outcome indicators about
• outcomes measure before and after

15
Carers Project indicators

• Support and rights Level of income from benefits
• No of successful referrals
• Aids and adaptations accessed
• Coping Level of expressed stress
• Ability to say what they feel
• New things able to do
• Isolation How alone they say they feel
• No of new friends made
• Repeat contact
• New things able to do

16
Collecting information

• Info about what you do and for whom.
• Info to show whether these things are delivering
  your outcomes.
• What collecting already?
• What new information to collect?
• How build info gathering into activities?

17
Collecting outcome info

• Observation
• Group discussions
• Wall chart what I can do better
• Interviews
• Third party feedback (social workers, teachers)
• Pictorial representations of progress
• (Video) diaries
• Mapping
• Mystery shopping

18
Carers activity measures

• How many people supported - breakdown
• How often the advice service available
• How many group sessions organised
• How many pieces of advocacy
• Attendance records
• Database
• Files

19
Carers outcome measures

• Observation
• Group discussions
• Carer 1-1 interviews
• Third party feedback (social workers)
• Outcomes wheel
• Progress trees or charts
• Video diaries

20
Analysing information

• Thematic analysis what are the common issues?
  What progress individually and collectively?
• Did outcomes change and if so why?
• What does the information say to others?
• What will you do differently in the future?
• What methods worked best?
• How does it compare to research?

21
Reporting

• One internal core report! Adapt for different
  audiences
• Users
• Funders
• Other stakeholders
• Annual report
• Internal future planning

22
Self evaluation

• Advantages
• Make use of your knowledge and expertise.
• Likely to lead to change or improved practice.
• Might provide a truer picture.
• Cheaper?
• Disadvantages
• Make take up a lot of time or resources.
• May be seen as less objective or credible.
• You may not have all the skills you need.
• Difficult to step back / see bigger picture.

23
External evaluation

• Advantages
• Brings research and analytical skills.
• Can set project in context and against evidence.
• Has no vested interest (??)
• Seen as robust and valid by funders and others.
• Disadvantages
• Needs budget, good specification, good people.
• Cant be there all the time.
• You still need to collect information.
• Who is in charge? (funder or you?)
• Not a PR exercise (but might become one).

24
Internal vs External

• Not either /or
• Always self-evaluate!
• Could use externals for technical support or for
  critical or context analysis.
• Ethos of evaluation should fit the ethos of your
  project.

25
Approaching funders

• Set your outcomes BEFORE you apply.
• Build in the cost of evaluation / technical
  support.
• Set evaluation plan and systems.
• Ask the funder what their requirements are as
  soon as you get funding (or before).
• Be strong only collect whats relevant!
• If funder is commissioning an external
  evaluation, ask to be involved from start.

26
Evaluation Support Scotland

• We provide specialist support to voluntary
  organisations and funders
• to enable them to evaluate and learn
• and so provide better services for communities.

27
What we do

• Evaluation support on outcomes, collecting
  information, planning and doing evaluation.
• One stop shop for resources, good practice and
  list of consultants.
• Work with funders to build evaluation and
  learning into programmes.
• Help join up monitoring evaluation
  requirements.
• Partnership with other organisations to share
  learning about evaluation.
• Promote evaluation language, practice learning.

28
Getting support

• Check out our website.
• Fill in an enquiries form on web or phone us.
• Sign up for a support clinic.
• Sign up for a workshop.
• Send us material to comment on.
• Pay us for a longer piece of support?

29
Evaluation Support Scotland

• info_at_evaluationsupportscotland.org.uk
• Tel [phone]
• 6th floor Riverside House, 502 Gorgie Road,
  Edinburgh EH11 3AF
• www.evaluationsupportscotland.org.uk

30
Personal action plan

• Did I learn anything?
• What will I do with this knowledge?
• What will I feedback to colleagues?
• Is there anything I didnt understand what
  questions do I want to ask?