System Review of Evaluation in German Development Cooperation

1
System Review of Evaluation in German Development
Cooperation

• DAC Network on Development Evaluation meeting, 18
  November 2008
• Michaela Zintl, Head of Evaluation and Audit
  Divison, BMZ

2
Rationale

• First review in 1998 / implementation of
  recommendations monitored in 2001
• Significant changes since
• International level MDGs Paris Declaration
  new developments also in the field of evaluation
• National level changes in the budget code
  joint up German development cooperation

3
Purpose

• ...to assess whether the evaluation system of
  German development cooperation performs
  adequately with a view to strategic and
  conceptual demands and in terms of
  organisational, managerial and methodological
  benchmarks (DAC norms and standards,
  international good practice)
• ...to develop recommendations for improvements of
  the system

4
Scope evaluation systems of

• BMZ (Ministry)
• Six implementing agencies (KfW, GTZ, ...) and the
  private sector arm (DEG)
• 12 CSOs receiving funds from BMZ
• further 30 CSOs (online questionnaire)
• ...covering 80 of bilateral aid funded by BMZ

5
Methodology

• Guided self-assessment
• Document analysis (including assessment of
  evaluation reports)
• Face to face interviews with staff and management
  of all organisations involved, external experts
  and MoP 170 interviews

6
Caveat

• Not included
• Partner countries views
• DC managed by other federal ministries and
  federal states
• Voluntary contributions to international
  organisations
• Light review of evaluation reports

7
Findings Independence and credibility

• Significant improvements since last review
• Most organisations acknowledge DAC norms and
  standards
• - Evaluation underfunded almost everywhere
• - Coverage appears wanting (insufficient data)
  
• - Lack of ex post evaluations, impact
  assessments and comprehensive comparative studies

8
Findings Quality

• Has generally improved, due to
• Acceptance of standards training
• Results orientation
• Improvement in process quality
• - Insufficient methodological awareness mix
  of methods more rhetoric than practice
• - Almost no impact evaluations
• - Evaluation knowledge not considered a
  priority for selection of staff and consultants

9
Findings Participation

• o No significant changes since last review
• - Evaluations continue to be donor centered
  except for some NGOs
• - Insufficient use of local independent
  consultants
• - (Too) little evaluation capacity building

10
Findings Utility

• Learning is usually the primary objective
• Evaluations are considered valuable by
  intended users
• o Mostly used for management of individual
  programms little conceptual learning
• - Little inter-institutional learning
• - Focus on learning by donor agencies not
  partners

11
Conclusions I Improvements on many dimensions
but challenges remain

• Institutional independence increased but
  insufficient use of external consultants
• Credibility increased because of better quality
  but limited by lack of transparency
• Quality improved but limited range of methods
  used
• Participation no significant improvement
• Utility internal and instrumental learning good
  and improved inter-ageny and conceptual
  learning poor

12
Conclusions II System development

• Heterogenity between agencies has rather
  increased over time
• System tilt between BMZ and implementing agencies
  
• Steering capacity of BMZ too low
• Most pressing problem overcoming institutional
  fragmentation

13
Recommendations

• General recommendations along the various
  dimensions in synthesis report specific
  recommendations in case study reports
• On the system level
• Massive strengthening of BMZ
• Independent evaluation agency (SADEV model)
• Independent evaluation advisory board (DFID model)

14
Lessons learned/ follow-up too early to say but
in any case

• Combination of external experts and peers
  essential
• Task to review 20 organisations is overwhelming
• Partly a very political process, partly benign
  neglect
• Change triggered already during the process in
  almost all organisations, although some prefer to
  say otherwise