Building Evaluation Capability

1
Building Evaluation Capability

• North of Scotland Public Health Network
• 3rd May 2007

Acknowledgement Associate Professor Patricia
Rogers (RMIT, Melbourne)
2
Content

• Evaluation
• What, why, scope, evaluation needs
• Evaluation capability
• What
• Building evaluation capability
• Seven principles
• Need for a strategy
• Building evaluation capability in the north of
  Scotland

3
Evaluation
4
What is Evaluation a definition
Evaluation assists sense-making about policies
and programs through the conduct of systematic
inquiry that describes and explains the policies
and programs operations, effects, justifications
and social implications .. (Mark, Henry, Julnes)
to improve decision making and enhance
organisational learning.
5
Why is evaluation important?

• An increased demand for accountability
• Evaluation promotes organisational learning and
  improved performance
• Evaluation helps us understand customers
  expectations and satisfaction
• Evaluation provides key approaches that support
  outcome planning
• Evaluation helps us identify, describe, attribute
  and report impact
• Evaluation develops knowledge

6
What is the Scope?Forms of Evaluation

• Any time in the life cycle of a programme or
  project
• Pro-active (ex-ante) - happens before a
  project/program is designed
• Clarification - happens during project/program
  design
• Interactive - happens during the life of a
  project/program
• Monitoring - happens during the life of a
  project/program
• Impact - usually happens at the end of a
  project/program

7
Criteria for Doing Evaluation- Which and to What
Level?

• Size of the evaluand
• Pilot or Trial
• Need to or good to or directed to
• Feasible
• Potential usefulness for future decisions?
• The level
• Go for higher quality levels (Mark, Henry,
  Julnes)
• Costs no more
• High risk of getting the answer wrong
• Difficult to observe / reverse an intervention in
  a timely fashion.
• Except where there is good prior evidence
• Depending on the level of uncertainty or dissent

8
Evaluation Capability
9
A Common Mental Map of Developing Evaluation
Capacity



10
An enlarged mental map of developing evaluation
capacity
11
Building Evaluation Capability
12
The 7 Principles
1
Address supply demand
Think big but start small
2
3
Work all levels of the organisation / system
4
Establish a model of behaviour change
5
Develop common context-specific frameworks
Build your own knowledge of evaluation
6
7
Walk the talk
13
2
Think big but start small

• Grow evaluation capability
• Trial new approaches and frameworks
• Work with volunteers
• Apply action learning principles
• Example -1 project 12 months
• - 10 projects 18 months
• - Mandating for a new initiative
• - Mainstreaming

14
3
Work All Levels of the Organisation

• Senior executive level
• Middle and project managers
• Evaluation champions
• Evaluation specialists
• The profession

15
4
Model of Behaviour Change
Adapted from Target 10 Project, DPI Victoria
Australia
16
A theory of change for building evaluation
capacity
Better outcomes for the public
Improved programs (through improved
implementation, better resource allocation and/or
improved selection of programs)
Use evaluation capability to undertake or oversee
proactive, claricative, monitoring, process,
impact evaluations as discrete studies or ongoing
work
Opportunities to deploy the capacity
Various activities
17
5
Develop context specific frameworks

• The importance of defining success
• Theories for action (Bennetts Hierarchy,
  Canadian Model)
• Templates guidelines
• Evaluability assessment
• Training in design, data collection, analysis,
• interpretation, and so on

18
Build your knowledge of evaluation
6

• Secondments
• Evaluation research
• Visiting Experts
• Publications presentations
• Communities of Interest
• Participation in professional bodies
• It is important to both develop knowledge and
  contribute knowledge.

19
Walk the talk
7

• Systematically and visibly evaluate your
  capability.
• Plan evaluation from the beginning
• Undertake evaluability assessment
• External review / mentoring/ other
• Practice good project management.
• Project planning (Outcomes, outputs, activities,
  resources finance, HR, infrastructure, and so
  on).
• Project implementation
• Project reporting accountability

20
Thinking About Demand
21
The Importance of Demand

• Keith Mackay (1994) World Bank,
• He has argued that supply is not as crucial as
  demand. (World Bank, 1994).
• Bob Williams (2001)
• The danger of focusing on just supply is that of
  ending up all dressed up with nowhere to go
  capable of producing evaluations but unable to
  use them, or even worse, producing evaluations
  that are treated as irrelevant

22
Under-Utilisation
I see a major challenge to the continuation of
the momentum of your evaluation program. The
challenge is that of adequate utilization of the
completed evaluation reports -- utilization by
provider middle managers, purchasers, and funding
sources. Improved evaluation does not occur at
the completion of an evaluation report. It must
be utilised. As overall evaluation efforts
widen, there are associated risks. If project
staff begin to believe their completed
evaluations are ignored / not acted upon /
misuse, morale associated with conducting
evaluations will plummet. Claude Bennett
23
Challenge 1
Balancing supply demand
Supply
Demand
24
Challenge 2
Growing both supply demand
25
Challenge 3

• Develop a demand for negative findings (heroic
  failures) and to use these as learning
  opportunities.

26
Challenge 4

• Monitoring changes in demand

27
Strategies for Working on Demand

• Undertake an analysis of current evaluation-like
  activities
• Hold values clarification exercises across
  stakeholder groups
• Use appreciative inquiry to analyse demand
• Regular facilitated workshops/ meetings to
  monitor demand
• Rehearse reports
• Include utilisation strategies in evaluation
  plans
• Regular reviews of utilisation
• Create supportive organisational culture

28
Where is evaluation needed?Analysing your system
Note Not all evaluative information is labelled
evaluation
29
Thinking About Supply
30
Ways of building supply

• Examples
• External evaluation courses and workshops
• Internal workshops/resources, linked to practice
  (just-in-time)
• Developing local resource material manual,
  guidelines, templates
• Learning from networks conferences, communities
  of interest
• Mentoring, apprenticeships, advisory services
• Reading library, resource centre, book reading
  club
• Learning circle, mentoring
• External review of evaluations

31
(No Transcript)
32
Strategy Plan

• Pulling it all together

33
Developing an Evaluation Capability Strategy

• Use robust planning processes
• Vision / description of success
• Reach
• Senior managers
• Project / Programme managers
• Project staff
• Evaluation champions
• Evaluation specialists
• Logic
• Principles

34
Contents of the Policy Evaluation and Appraisal
BEC Strategy

• Contents
• Introduction
• What is Building Evaluation Capability
• Scope of the Approach in Scottish Health
  Improvement
• Underpinning Values
• Theory of Action (Indicative)
• Activities and Outputs
• Educational Opportunities
• Peer Support
• One to one Support
• Community / Organisational Capability
• Reporting and accountability, evaluation risk
  management and communication plans

35
Some Examples of Activities
36
Summary

• Evaluation Capability Building
• It takes a long time. You need a long term vision
• Executives need to both nurture and insist on
  evaluation
• Getting rid of fear is paramount - capability
  building is about learning
• Clients need to be open-minded, capability
  builders flexible
• Needs to be guided by principles and not driven
  by prescription
• Collaborate rather than compete