World Bank: An Approach to ResultsBased CountryLevel Evaluations

1
World Bank An Approach to Results-Based
Country-Level Evaluations

• Ajay Chhibber Director
• Independent Evaluation, World Bank
• Incorporating Managing for Development Results in
• Monitoring and Evaluation
• Bangkok, February 1, 2006

2
Outline of Presentation

• Methodological Approach
• Challenges
• Experience with Joint Evaluations
• Expanding the Approach

3
Methodological Approach Three Basic Steps

• Determining the objectives
• Assessing the achievement of each objective
• Contribution

4
Country Assistance Evaluation Framework
5
Step 1 Determining the Objectives of the Banks
Assistance Strategy

• Main Country-level Program Objectives are
  derived from
• The country assistance strategy agreed with the
  Client
• The evaluators ex post assessment of fundamental
  development constraints.

6
Country Assistance Evaluation Framework
What are the main objectives of the Bank program
intended to accomplish? Relevant? Selective?
Evaluable?
Bank objectives are a subset of country
development objective.
7
Step 2 Organize by ObjectivesProducts and
Services

• For each separate objective
• What was the Bank trying to accomplish?
• Were the instruments used
• (i) Loans, Credits, and Grants
• (ii) Analytical and Advisory Services and,
• (iii) Partnerships, Aid Coordination and Resource
  Mobilization
• Coherent? Relevant to achieving the objective?

8
Country Assistance Evaluation Framework
9
Illustration of CAE Framework Tunisia
10
Outcomes Was the objective achieved?

• Mauritania CAE Progress On Infant Mortality.

11
OutcomesA Second Example

• In Brazil, the Bank focused on improving
  socio-economic outcomes in the North East.

12
Step 3 Overall Outcome of the Country
Assistance Program

• In aggregate, were the objectives of the program
  met?
• An Example from the Tunisia CAE

13
Country Assistance Evaluation Framework
14
Country Assistance Evaluation Experience
Tunisia CAE Summary
Objective
Indicator
Performance
15
Step 4 Contribution

• Bank Performance
• Borrower Performance
• Partner Performance
• Exogenous Factors

16
Country Assistance Evaluation Framework
17
Assessing the Banks Contribution

• Relevance and implementation of the strategy
• Design and supervision of the Banks lending
• Scope, quality, and follow-up of non-lending work
• Consistency of Banks lending with its
  non-lending work
• Did the Bank work well with partners, was the
  program complementary and consultative

18
Assessing Other Contributions

• Client Performance key issuesOwnership of
  assistance program Support for national and
  international development priorities (MDGs,
  national development plan, etc.) and respect for
  safeguards
• Partner Performance key issuesImpact on design
  of assistance program and Impact on
  implementation of assistance program
• Exogenous Factors what was the impact on
  outcomes of (a) World economic shocks, (b) Events
  of nature, (c) War/civil disturbances, and (d)
  other exogenous shocks.

19
Challenges to Assessing Country Assistance
Effectively

• Clarifying the object of evaluation.
• Reaching agreement on counterfactuals.
• Attributing program results correctly.

20
Challenges 1. Distinguishing between

• Country development performance and the outcome
  of the assistance program performance
  Distinguishing country outcomes from outcomes of
  Bank assistance programs can be especially
  problematic for large middle-income borrowers.
  In Brazil, as noted earlier the Bank focused on
  the poor in the Northeast and
• The outcome of the assistance program performance
  and Donor (World Bank) performance
  Unsatisfactory outcome ratings do not imply poor
  Bank performance (e.g., Bulgaria Haiti Rwanda
  are examples of this).

21
Challenges 2. Finding the Right Counterfactuals

• Ideal would be to measure Client development with
  and without assistance program.(Not feasible, so
  we have to use proxies.)
• Client development before and after assistance
  program.
• Client development relative to countries at
  similar stage of development.
• Client performance relative to the development
  indicators specified in country assistance
  strategy.

22
Challenges 3. Attributing Program Results
Correctly

• Individual donor attribution not possible using
  our current counterfactuals, as attribution
  involves establishing a causal link between a
  change and a specific intervention.
• We have moved to a concept of most likely
  association.
• This involves establishing to the degree
  possible how and to what extent the Banks
  interventions were linked to the results
  achieved.

23
OED Evaluation Partnerships

• CDFMulti-donor evaluation
• PRS and FSAP (ongoing)joint evaluation with
  Independent Evaluation Office (IEO) of IMF
• Global Program Evaluations
• Country Evaluations with other IFIs

24
OED Multilateral Partnerships in Country
Assistance Evaluations
Source OED Working Paper Partnership in Joint
Country Assistance Evaluations A Review of World
Bank Experience, OED, World Bank, 2005.
25
Expanding the Approach

• Rationale
• Lessons from Limited Experience
• Issues in Extending the Approach

26
Rationale

• Donor community moving towards more coherence in
  assistance programs
• From a conditionality approach to a
  country-ownership approach
• From project aid approach to a budget-support/deve
  lopment policy lending approach
• From a separate procedures and policies approach
  to a more harmonized approach and,
• From a single donor-to-aid recipient approach to
  more cooperation among donors.

27
Lessons Learned

• Can identify overall impact of donor assistance,
  as well as key constraints, contradictions and
  gaps
• Promote understanding of institutions
  evaluation methods and encourage the use of
  common evaluation standards, consistent with the
  broader agenda of harmonization.
• Lower transaction costs for aid recipients by
  reducing the burden of multiple, separate
  evaluations for client country institutions, but
• Take longer to prepare and cost more than
  anticipated, as differences in organizational
  mandates and methodologies can impose
  constraint/delays on joint evaluations.
• Delays are likely to be more pronounced when a
  single document is produced.

28
Extending the CAE Approach Some Issues

• Establishing the Objectives of the Assistance
  Programs
• Key question is what are the objectives/results/
  outcomes that donor programs are attempting to
  accomplish?
• Country development program?
• CG agenda of reforms?
• PRSP?
• MDGs?
• Essential to defining the organizing principles
  of the evaluation

29
Extending the CAE Approach Some Issues

• Attribution/Contribution will present many
  challenges.
• Donor programs may reflect many different
  objectives, e.g. domestic political concernshow
  will we measure relevance of donor programs
  against objectives
• Will it be possible to assess the contribution of
  the individual donors, e.g. this would involve
  assessing project versus program assistance, as
  well as assessing in some cases small
  interventions.
• Is it necessary? Can we focus more on learning
  what has worked and what has not?

30
Extending the CAE Approach Some Issues

• Country participationhow do we manage it?
• The active participation of the recipient country
  seems to be necessary for an evaluation of this
  nature.
• But, what about Independence questions?
  Inevitably we will be assessing the Governments
  (and past Governments?) contribution to outcomes.
• Do we use the MOF, a Supreme Audit Agency, or
  Experts independent of the Government?