Inaugural Aotearoa New Zealand Evaluation Association conference

1

• Inaugural
  Aotearoa New Zealand Evaluation
  Association conference
• Where is the value in evaluation?
• Masterton,
• July 2007

Patricia J. Rogers CIRCLE at Royal Melbourne
Institute of Technology, Melbourne, Australia
2
Thanks for pointing out my failings in public
and other things youll never hear in
evaluation

• Inaugural
  Aotearoa New Zealand Evaluation
  Association conference
• Where is the value in evaluation?
• Masterton,
• July 2007

Patricia J. Rogers CIRCLE at Royal Melbourne
Institute of Technology, Melbourne, Australia
3

• How evaluation is intended to work
• Challenges in leading, learning, and
  participating in change
• Some ways forward

4
1. Visions and Plans
5
1. Visions and Plans
The vision for evaluation in leading, learning
and participating in change
6
Ways evaluation can contribute to change
7
Lao Tzu

• Great nations are like great people
• when they make a mistake, they realize it
• having realized it, they admit it
• having admitted it, they correct it
• they consider those who point out their faults
  as their most benevolent teachers.

8
2. What actually happens
The reality of evaluation in leading, learning
and participating in change
9
Challenges to using evaluation for change

• Technical limitations of available evidence,
  delays in feedback, uncertainty/disagreement
  about what is needed in the intervention and the
  evaluation
• Cognitive taking in new information, overcoming
  assumptions
• Emotional defensive routines in response to
  shame, fear, and grief
• Organisational- incentives that support or
  restrict generation and use of information for
  improvement including organised self-interest,
  dysfunctional accountability systems and
  adversarial politics

10
Technical challenges delays
   
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
11
   
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
EVALUATION (INCLUDING ALL FORMS OF EVALUATION
FROM NEEDS ASSESSMENT, PROGRAM DESIGN, PROCESS
EVALUATION, MONITORING AND IMPACT ASSESSMENT)
12
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
EVALUATION (INCLUDING ALL FORMS OF EVALUATION
FROM NEEDS ASSESSMENT, PROGRAM DESIGN, PROCESS
EVALUATION, MONITORING AND IMPACT ASSESSMENT)
13
Consequences of a delay in the system
   
DELAY
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
EVALUATION (INCLUDING ALL FORMS OF EVALUATION
FROM NEEDS ASSESSMENT, PROGRAM DESIGN, PROCESS
EVALUATION, MONITORING AND IMPACT ASSESSMENT)
14
Uncertainty and disagreement
Staceys (1996) Agreement and Certainty Matrix,
from Zimmerman, 2001
15
Zimmerman, 2001
16
Cognitive challenges
17

• "It is impossible for someone to learn what they
  think they already know."
• Epictetus (AD 55?-135?), Greek Stoic philosopher

18
Lack of trust to share information

• Knowledge management is not just a technical
  issue of combining data
• Issues of trust and relationship

19
Modified Johari window Hase, Davies, Dick, 1999
20
Saving face

• How to survive in Australia
• 1. Never criticise.
• 2. Dont praise, except when it is expected.
• 3. Never explain.

21
3. Some ways forward
Some ways forward
22
Authentic evaluation

• Caring more about the quality of the intervention
  and of the evaluation- than saving face
• Focused and feasible evaluation given situation
• Transparent and defensible methods and judgements
• Iterative design and implementation
• A safe space for naming problems and addressing
  them
• Corporate memory documenting decisions
• Team work insider/outsider, team, association

23
Aragorn approach to evaluation

• Are you frightened?
• Yes.
• Not nearly frightened enough I know what hunts
  you.

24
Gandalf approach to evaluation

• What we have to do is to make the most of the
  time that is allotted to us.

25
Challenges and some responses

• Caring more about the quality of the intervention
  and of the evaluation- than saving face
• Focused and feasible evaluation given situation
• Transparent and defensible methods and judgements
• Iterative design and implementation
• A safe space for naming problems and addressing
  them
• Corporate memory documenting decisions
• Team work

• Technical limitations of available evidence,
  delays in feedback, uncertainty and disagreement
  about what is needed in the intervention and the
  evaluation
• Cognitive taking in new information, overcoming
  assumptions
• Emotional defensive routines in response to
  shame, fear, and grief
• Organisational- incentives that support or
  restrict generation and use of information for
  improvement including organised self-interest,
  dysfunctional accountability systems and
  adversarial politics

26
References and further reading

• Hase, Davies and Dick (1999) The Johari Window
  and the Dark Side of Organisations. Ultibase
  article. Available at http//ultibase.rmit.edu.au
  /Articles/aug99/hase1.htm
• Rogers, P. and Williams, B. (2006) Evaluation
  for performance improvement and organisational
  learning in I. Shaw, J. Greene and M. Mark (eds)
  Sage handbook of Evaluation, London Sage
  Publications
• Stacey, R. (1996) Complexity and Creativity in
  Organizations. Berrett-Koehler.
• Tavris, C. and Aronson , E. (2007) Mistakes Were
  Made (But Not By ME) Why we justify foolish
  beliefs, bad decisions, and hurtful acts
  (Harcourt)
• Treborlang, R. (1991) How to survive Australia.
  Major Mitchell Press. Also available at
  http//www.majormitchell.com.au/contents.htmlsurv
  ive.
• Zimmerman, B. (2001) Ralph Stacey's Agreement
  Certainty Matrix . Available at
  http//www.plexusinstitute.org/edgeware/archive/th
  ink/main_aides3.html