Is everyones dog noones dog And other challenges in moving from individual evaluations to evaluation

1
Is everyones dog no-ones dog? And other
challenges in moving from individual evaluations
to evaluation systems

• 
  Utvärderingssystem
• i Sverige
• till vad och för vem?
• Swedish Evaluation Society conference 2005

2

• What do we mean by evaluation systems?
• What ways of thinking about systems are most
  useful for developing evaluation systems?
• What are some of the issues and challenges in
  developing and maintaining evaluation systems?

3
Examples of evaluation systems

• Scoring system for evaluating tenders/job
  applicants/proposals
• Framework for Reviews in Schools (Education
  Review Office, New Zealand)
• (ERO, 2002)
• Public Service Commission Monitoring and
  Evaluation system (South Africa) (Naidoo, 2005)
• Federal evaluation strategy 1988-1997 (Australia)
  (Mackay, 2004)

4
1. What do we mean by an evaluation system?

• A system is composed of regularly interacting or
  interdependent groups of activities/parts that
  form the emergent whole.
• Wikipedia
• (For other useful definitions, see Williams
  (2002) which discusses whether things are
  actually systems or whether it is simply useful
  to think of them as if they were)

5
1. What do we mean by an evaluation system?

• A systematic evaluation framework used across a
  number of separate evaluations
• Connecting separate evaluations
• An explicit process linking evaluations with
  decision making

6
a) A systematic evaluation framework used across
a number of separate evaluations

• Evaluation logic
• Evaluative criteria
• Standards
• Data collection methods
• Synthesis methods
• Reporting formats
• Evaluation design template
• Purpose
• Audiences
• Questions
• Methods

7
b) Connecting separate evaluations

• The different parts of a multi-level evaluation
  (local, regional and overall)
• Synthesis of separate evaluations to inform
  evidence-based policy and practice
• Connected computerised information systems

8
c) An explicit process linking evaluations with
decision making

• Linking an evaluation with an intervention
• Linking a number of evaluations with a range of
  interventions

9
Purposes of evaluation systems
10
Everyones dog no-ones dog?

• Do individual evaluations need to identify and
  engage intended users and uses?
• Do evaluation systems need to do this?
• Is the concept of ownership necessary to
  achieve commitment?
• Do evaluation systems need to pay more attention
  to the people involved in producing and using
  evaluations and those whose interests the
  evaluation is meant to serve?

11
Example 1 Public Service Commission, South
Africa

• Evaluates the performance of national and
  provincial departments in terms of nine
  Constitutional principles
• Standardised data collection processes by a
  central agency
• Computerised knowledge management system to
  combine data
• Explicit attention to intended use (Public
  Service Commission, 2003)

12
The underlying programme logic of the PSC
monitoring and evaluation system
Overall result Better governance and service
delivery in South Africa
Problems are addressed
Achievements are affirmed and promoted
Departments focus on priority areas
Learning from Good practice examples takes place
FOLLOW UP
Problem areas identified
Good practice by others is identified and promoted
Priority areas in public administration Are
communicated
Departments reflect on their own performance
REPORTING
Public service monitoring
Public Service Commission, 2003
13
Example 2 Evidence-based policy and practice as
an evaluation system

• Steps in EBPP
• Generating/locating evidence
• Validating evidence
• Synthesising evidence
• Making synthesis accessible and meaningful
• Using evidence in policy and practice
• Contributing to the evidence base
• Rogers Williams (2005)

14
EBPP as a cycle
6. Contributing to evidence base
Rogers Williams (2005)
15
Alternative strategies for EBPP

• Meta-analysis of experimental and
  quasi-experimental studies
• Realist best evidence synthesis
• Replicating proven practice
• Corporate and community memory of what works
• Local performance monitoring

16
Meta-analysis

• Generating/locating evidence
• Published research
• Validating evidence
• Experimental or quasi-experimental designs
• Synthesising evidence
• Meta-analysis
• Making synthesis accessible and meaningful
• Publishing results of systematic reviews What
  works e.g. Campbell Collaboration
• Publishing policy briefs e.g. evidence nuggets
• Using evidence in policy and practice
• Apply findings about what works and what does not

17
Meta-analysis

• Advantages
• Simple, clear messages that can be readily
  communicated
• Transparent method for validation and synthesis
• Disadvantages
• Restricts acceptable evidence in a way that
  excludes significant information
• Dependence on existing research makes it
  difficult to include consideration of the impact
  of context on what works
• Doesnt address situations where an intervention
  is contributes to an outcome but only in
  combination with other factors (e.g. Increased
  school funding)

18
Limitations of drawing only on a narrow range of
evidence

• Smith, G. and J. Pell (2003). "Parachute use to
  prevent death and major trauma related to
  gravitational challenge systematic review of
  randomised controlled trials." BMJ 327(7429)
  1459-61.

19

• OBJECTIVES To determine whether parachutes are
  effective in preventing major trauma related to
  gravitational challenge.
• DESIGN Systematic review of randomised
  controlled trials.
• RESULTS We were unable to identify any
  randomised controlled trials of parachute
  intervention.

20

• CONCLUSIONS As with many interventions intended
  to prevent ill health, the effectiveness of
  parachutes has not been subjected to rigorous
  evaluation by using randomised controlled trials.
  

21

• CONCLUSIONSAdvocates of evidence based medicine
  have criticised the adoption of interventions
  evaluated by using only observational data.
• We think that everyone might benefit if the most
  radical protagonists of evidence based medicine
  organised and participated in a double blind,
  randomised, placebo controlled, crossover trial
  of the parachute.

22
2. Ways of thinking about systems of evaluation

• Simple linear systems
• Systems archetypes, such as the impact of delays
  (e.g. Senge, 1992)
• Complex adaptive systems (e.g. Eoyang and Berkas,
  1998, Zimmerman, 2001)
• Critical systems thinking (e.g. Midgely, 1996,
  Flood Jackson, 1991)

23
Thinking about an intervention and its evaluation
as two simple systems
   
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
24
Thinking about an intervention and its evaluation
as two simple systems
   
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
EVALUATION (INCLUDING ALL FORMS OF EVALUATION
FROM NEEDS ASSESSMENT, PROGRAM DESIGN, PROCESS
EVALUATION, MONITORING AND IMPACT ASSESSMENT)
25
Thinking about an intervention and its evaluation
as two simple systems
   
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
EVALUATION (INCLUDING ALL FORMS OF EVALUATION
FROM NEEDS ASSESSMENT, PROGRAM DESIGN, PROCESS
EVALUATION, MONITORING AND IMPACT ASSESSMENT)
26
Challenges

• Delays in feedback
• Uncertainty
• Disagreement
• Lack of trust to share information

27
a. Consequences of a delay in the system
   
DELAY
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
EVALUATION (INCLUDING ALL FORMS OF EVALUATION
FROM NEEDS ASSESSMENT, PROGRAM DESIGN, PROCESS
EVALUATION, MONITORING AND IMPACT ASSESSMENT)
28
Uncertainty and disagreement
Staceys (1996) Agreement and Certainty Matrix,
from Zimmerman, 2001
29
Zimmerman, 2001
30
b. Dealing with uncertainty

• Some elements of Complex Adaptive Systems
• Continual change but not necessarily continuous
• Nonlinear and discontinuous causal factors
• Emergent or self-organizing behaviour
• Eoyang and Berkas (1998)

31
Evaluation in a Complex Adaptive System
Eoyang and Berkas (1998)
32
c. Dealing with disagreement
Critical Systems Theory CST asserts that all
boundaries affecting what and who to consider or
involve are contestable, and should be contested.
The intention is to make decision-making more
explicit and transparent. The CST researcher does
this by directly asking for input from the people
whose voices are not normally heard. (Rogers
Williams, 2006)
33
d. Lack of trust to share information

• Knowledge management is not just a technical
  issue of combining data
• Issues of trust and relationship

34
Modified Johari Window
Hase, Davies, Dick (1999)
35
Iterative development of evaluation systems
Better outcomes for the public
Improved programs (through improved
implementation, better resource allocation, or
improved selection of programs)
Development of systems to apply evaluation
capacity to undertake, oversee and use discrete
evaluations, ongoing evaluative activity or
monitoring
Capacity (types of capital)
Opportunities to deploy the capacity
Human
Economic
Social
Institutional
Various activities
36
Final thoughts Overloading the ship

• Adding more and more can lead to poor performance
  and instability
• What does this mean for an evaluation system

37
Learning from the sinking of the Vasna
Fairley and Willshire (2000)
38
Learning from the sinking of the Vasna
Fairley and Willshire (2000)
39
Ethical imperative 1 Acknowledge limitations of
evaluation systems

• Appropriateness for a particular situation
• Trade offs between accuracy and timeliness
• Trade off between detail and simplicity

40
Ethical imperative 2 Identify and address
barriers

• Technical sufficiently accurate answers to the
  important questions
• Learning getting beyond assumptions, learning to
  do things differently
• Emotional defensive routines in response to
  shame, fear, and grief
• Organisational- incentives that support or
  restrict use of information for improvement
• (Rogers and Williams, 2006)

41
Ethical imperative 3 Identify and manage
possible risks

• Distraction from actually getting the job done
• Reducing the quality of evaluation by focusing on
  narrow targets or inappropriate models of the
  system
• Creating no-ones dog
• Overloading the ship
• Do a negative program theory how could building
  an evaluation system actually make things worse?

42
References

• Checkland, Peter (1981) Systems Theory, Systems
  Practice Chichester John Wiley
• Education Review Office, New Zealand (2002)
  Framework for Reviews in Schools. Available at
  lthttp//www.ero.govt.nz/EdRevInfo/Schedrevs/School
  Framework.htmgt
• Eoyang, G and Berkas, T.(1998) Evaluation in
  Complex Adaptive Systems. Paper available at
  lthttp//www.winternet.com/eoyang/EvalinCAS.pdfgt
• Fairley, Richard E., and Willshire, Mary Jane.
  "Why the Vasa Sank 10 Problems and Some
  Antidotes for Software Projects," IEEE Software,
  vol. 20,  no. 2,  pp. 18-25,  March/April,  2003.
  
• Flood R. L. (1999). Rethinking the 5th
  Discipline. London Routledge
• Flood, Robert and Jackson, Michael (eds) (1991)
  Critical Systems Thinking Directed Readings.
  New York John Wiley and Sons.
• Hase, S., Davies, A. and Dick, B (1999) The
  Johari Window and the Dark Side of Organisations.
  Southern Cross University. Available at
  lthttp//www.itslifejimbutnotasweknowit.org.uk/file
  s/johariwind.pdfgt
• Naidoo, Indran (2005) Monitoring and Evaluation
  as an Effective Strategy for Transformation.
  Presentation to the Senior Management Conference,
  Free State. Available at lthttp//www.fs.gov.za/IN
  FORMATION/Documents/SMS20Conference/free20state
  20SMS20conference.pptgt
• Mackay, Keith (2004) Two Generations of
  Performance Management and Evaluation Systems in
  Australia. ECD Working Paper Series No. 11.
  World Bank Operations Evaluation Department.
  Available at http//lnweb18.worldbank.org/oed/oedd
  oclib.nsf/DocUNIDViewForJavaSearch/14163969A1A709B
  D85256E5100013AA8/file/ecd_wp_11.pdf
• Midgley, G. (1996). What is this thing called
  Critical Systems Thinking? Critical Systems
  Thinking Current Research and Practice. R. L.
  Flood and N. R. A. Romm. New York and London,
  Plenum Press 11-24.

43
References - continued

• Public Service Commission (2003) The PSCs Public
  Administration Monitoring and Evaluation System
  First Consolidated Report. Available at
  www.pmg.org.za/docs/2003/appendices/031112psc.ppt
  
• Rogers, Patricia and Williams, Bob (2005)
  Evidence-Based Policy and Practice. Presentation
  to the Australian Government Department of Family
  and Community Services as part of the evaluation
  of the Stronger Families and Communities Strategy
  2000-2004.
• Rogers, Patricia and Williams, Bob (2006)
  Evaluation for Practice Improvement and
  Organisational Learning, in Ian Shaw, Jennifer
  Greene and Mel Mark (eds) The Handbook of
  Evaluation. London Sage Publications (UK).
• Senge P. M. (1992) The Fifth Discipline. Sydney.
  Random House
• Smith, G. and J. Pell (2003). "Parachute use to
  prevent death and major trauma related to
  gravitational challenge systematic review of
  randomised controlled trials." BMJ 327(7429)
  1459-61. gt
• Stacey, Ralph (1996) "Emerging Strategies for a
  Chaotic Environment." Long Range Planning, Vol.
  16, pp. 182-189.
• Williams, Bob (2002) Evaluation and Systems
  Thinking. Available at http//users.actrix.co.nz/
  bobwill/evalsys.pdf
• Zimmerman,B., Ralph Stacey's Agreement
  Certainty Matrix, 2001 . Available at
  http//www.plexusinstitute.org/edgeware/archive/th
  ink/main_aides3.htm

44
Patricia.Rogers_at_rmit.edu.au CIRCLE at RMIT
University Collaborative Institute for Research,
Consulting and Learning in Evaluation, Royal
Melbourne Institute of Technology Melbourne,
AUSTRALIA