Evaluating General Budget Support

1
Evaluating General Budget Support

• Stephen Lister
• Development Evaluation Workshop
• IDS Sussex, 3 October2006

2
Introduction

• This presentation
• will raise
• scope and methods of the Joint Evaluation of
  General Budget Support (GBS)
• issues in linking policy to evaluation
• wont properly explain
• the evaluation findings and conclusions --see
  Reports and Summaries at
• http//www.oecd.org/document/51/0,2340,en_21571361
  _34047972_36556979_1_1_1_1,00.html
• will be superficial!

3
The Study

• joint study, building directly on previous
  DFID-initiated work on evaluability of GBS
• large Steering Group, long TOR, large consultancy
  team, 18-month study
• seven countries, 19942004
• Burkina Faso, Malawi, Mozambique, Nicaragua,
  Rwanda, Uganda, Vietnam
• TOR to what extent and under what circumstances
  is GBS relevant, efficient and effective for
  achieving sustainable impacts on poverty
  reduction and growth

4
Methodology Some Basics

• Identifying the evaluation subject
• GBS as a package of inputs
• (funds, conditions, dialogue, harmonisation and
  alignment, capacity building and TA)
• Subset of Partnership GBS (PGBS)
• (new style of conditionality(?), support to PRSs)
• In practice, overlap between GBS and SBS (Sector
  Budget Support)
• Country-level inventories to document episodes of
  PGBS
• Counterfactual?
• Illustrative sample of countries

5
What did we find to evaluate?
Partnership GBS flows

• large volume, but recent, uneven distribution of
  PGBS
• no scope for econometric approach
• useful contrasts in penetration (duration,
  amounts, number of donors involved)
• (e.g. Uganda vs. Vietnam Rwanda vs. Mozambique)

6
Theory-Based Evaluation

• Enhanced Evaluation Framework (EEF) below and
  hand-out developed from earlier EF by
• addition of level zero entry conditions
• policy effects, as well as flow-of-funds and
  institutional effects
• more recognition of parallel and complementary
  inputs (other aid, government resources)
• disaggregation of poverty effects (income,
  services, empowerment)
• feedback loops
• Hypothetical effects spelt out in Causality Map
  below and hand-out

7
Enhanced Evaluation Framework (EEF)

8
Causality Map
9
How did this help?

• It did not
• Remove the attribution problem
• Remove the problems of time scale and data
  (especially for poverty impact)
• Alter the fact that PGBS is an approach not a
  strategy (and a moving target)
• But it did
• Clarify and disaggregate hypotheses about effects
• Enable disaggregation (and use!) of
  counterfactuals
• Enable all 7 country studies to be symmetrically
  organised
• identical country report structures
• same standard Evaluation Questions
• interlocking evaluation teams
• crude rating system to standardise assessments
  across countries and distinguish general trends
  from effects attributed to PGBS

10
Overall assessment

• Overall positive assessment in 5 of 7 cases.
• Principal findings
• Relevant response to problems in aid
  effectiveness.
• Efficient, effective and sustainable way of
  supporting national poverty reduction strategies.
• Positive systemic effects on capacity by
  providing discretionary funds to national budget
  system.
• Spill-over effects enhance quality of aid as a
  whole.
• Initial effects on poverty mainly through
  expanding public services. Ultimate effects will
  depend on the quality of the national poverty
  reduction strategy.
• Capacity for learning suggests instrument can
  become more effective over time.
• Did not find unintended effects or side-effects
  that would outweigh benefits.
• Sustainability requires more attention to
  mitigation of risks.
• Findings are more widely relevant to
  programme-based approaches which share PGBS
  design principles.

11
Evaluation Issues and Reflections

• demand for policy advice vs. evaluation rigour
• logical consistency of the evaluation question?
• counterfactuals and the interaction of aid
  modalities
• terminology! but appearances matter
• implications for development evaluation agenda?