Study commissioned by the DAC Network on Development Evaluation

1
JOINT EVALUATIONS RECENT EXPERIENCES, LESSONS
LEARNED, OPTIONS FOR THE FUTURE

• Study commissioned by the DAC Network on
  Development Evaluation
• June 2005

2
Background

• 2000 Publication of Effective Practices in
  Conducting a Joint Multi-Donor Evaluation in the
  DAC Evaluation and Effectiveness Series
• 2004 DAC Network on Development Evaluation
  commissions a new study on joint evaluations
• To review past experience,
• To distil lessons learned, and
• To explore options for the future

3
Background

• The report is based on
• An extensive review of literature and written
  material, including official documents, manuals,
  evaluation reports, etc.
• More than 100 interviews with evaluators and aid
  officials, representatives of the research and
  NGO community, consultants, etc.
• The Nairobi workshop with representatives of
  partner country governments, NGOs and consultants

4
Background

• The report consists of three parts
• First, an analysis of key aspects of joint
  evaluation work carried out between 1990 and
  today
• Second, a review of recent experience with joint
  evaluations, and a presentation of lessons
  learned and recommendations
• Third, a chapter on options for the future and
  issues for discussion, primarily addressed to
  the DAC Evaluation Network

5
Background

• Annex 1 contains an annotated table of about
  50joint evaluations that were identified during
  the research phase. This overview provides a
  fascinating picture of the variety of joint
  evaluations that DAC members and others have
  implemented since 1990
• However, this summary table is unlikely to be
  fully complete. Members are invited to provide
  any additional information on joint evaluations
  of which they are aware

6
Findings

• The report contains a large number, and a wide
  range, of findings. This presentation highlights
  a selection of the most important ones
• Joint evaluations are a dynamic area of
  development co-operation, with an increasing
  number of joint evaluations undertaken during the
  last 4-5 years
• The DAC Evaluation Network has been instrumental
  in leading and forming the debate on this
  important aspect of evaluation work

7
Findings

• However, DAC members are displaying different
  levels of involvement in joint evaluation work.
  Denmark, the Netherlands and Norway are clearly
  in the lead, followed by a large middle group and
  then others who are demonstrating less commitment
  to joint working
• There is no evidence of a clear pattern that
  would explain why particular joint evaluations
  are undertaken and others are not

8
Findings

• The wide variety of configurations of actors,
  activities, focus, etc, in different joint
  evaluation work means it is difficult to develop
  a typology of joint evaluations that remains
  simple and workable. However, the following
  categorization, based on the mode of how actors
  work together, is proposed

9
Findings

• TYPE OF EVALUATION MODE OF WORKING TOGETHER
• Open to all stakeholders. All evaluation
  partners participate andcontribute equally
• Open to those who meet certain requirements,
  such as membership in a particular group of
  countries, participation in a SWAp, etc.
• A wide range of mostly complex arrangements for
  joint work with a general inclination to keep the
  number of partners down, usually to two or three

Classic multi-partner
Qualified multi-partner
Hybrid multi-partner
10
Findings

• There is a trend over the last 4-5 years towards
  fuller partner country participation. However,
  partner countries are too often invited to
  participate at a relatively advanced stage of the
  process when key decisions have already been
  taken
• A clear message came out of the Nairobi workshop
  that partner countries want to be involved in
  joint evaluation work more prominently, upstream
  and with stronger ownership

11
Findings

• Chapter 1 explores the wide range of advantages
  of joint working, along with the challenges and
  costs
• Chapter 2 explores more detailed experiences,
  challenges and possible solutions in joint
  evaluation work. A wide range of lessons learned
  and practical recommendations for the future are
  put forward.

12
Findings

• Strengths of joint working include
• A high degree of flexibility in forming and
  organising evaluation partnerships
• Knowledge sharing and mutual learning and
  capacity building
• High quality of products, usually due to the
  pooling of resources and approaches
• Enhanced credibility, legitimacy and impact

13
Findings

• Reduction of transaction costs for partner
  countries
• Reduction of number of conflicting messages
  emanating from a multitude of donor evaluations
• Shared financing of large evaluations - which
  individual donors might not be able to afford
• Promotion of donor harmonisation and alignment
• Response to the evaluation challenges emanating
  from the new modes of harmonised aid delivery
  such as GBS, SWAps, basket financing, etc.

14
Findings

• Potential weaknesses and challenges include
• Reliance on individual initiative as the prime
  mover of joint evaluation work
• Uncertain and uneven commitment of partners to
  the outcomes of the joint evaluation
• Too little time allocated at the beginning of the
  process to clarify and agree the objectives,
  concepts, methodologies and ground rules
• Complex supervisory and management structures,
  with too little delegation of authority, and too
  much micro-management and control, resulting in
  bureaucratisation and delayed processes and
  delivery

15
Findings

• Selection, guidance and supervision of
  consultants
• High direct and indirect costs for the lead
  donor(s)
• Long gestation periods for evaluation results to
  become available, entailing the risk of untimely
  delivery of products and outcomes
• Risk of conflict between the official and
  unofficial agendas of the partners involved in a
  joint evaluation

16
Findings

• Insufficient upstream participation of partner
  countries - resulting in a lack of ownership and
  a risk of sidelining the evaluation results
  within those countries
• Relatively weak reflection, in joint evaluation
  activities so far, of the new aid modalities and
  of the DAC agenda on Aid Effectiveness

17
Conclusions

• The picture that has emerged from this study is
  one of strong and invaluable leadership of the
  DAC Evaluation Network in
• - Putting joint evaluations on the agenda
• - Rooting them firmly in the donor community
• - Providing practical inputs and guidance
• Without the contributions of DAC Evaluation
  Network members and observers who have been
  willing to take up the challenges of joint
  evaluations, to invest resources, creativity and
  leadership, and to stay with this idea right from
  the first experiments in the 1990s, we would be a
  long way from where we are today.

18
Conclusions

• However, there are a number of new and emerging
  challenges that need to be addressed - by donors
  individually, by partner countries, and by the
  DAC Evaluation Network collectively.
• These challenges are listed in Chapter 3 which
  focuses on options for the future. Many of the
  issues raised need to be addressed urgently.

19
Conclusions

• These options and challenges for the future are
  focused around three basic considerations
• Improving the efficiency and effectiveness of
  joint evaluations
• Working towards enhanced developing country
  participation and ownership
• Streamlining the role of the DAC Evaluation
  Network and focusing on strategic issues

20
Next Steps

• Network Members are invited to give feedback and
  comments on the report, by a deadline of 21 June
• Feedback and comments should be sent directly to
  Hbreier_at_t-online.de with copy to
  Sebastian.Ling_at_oecd.org of the Secretariat.