Delegation constituted in La Paz

1
European CommissionJoint Evaluation Unit common
to EuropeAid, Relex and Development
Methodology for Evaluation of Budget support
operations at Country levelConceptual and
Methodological Note
2

• CURRENT SITUATION

EXISTING BUDGET SUPPORT EVALUATIONS 2004
Tanzania 2006 Joint Evaluation of 7 countries
from 1994 to 2004 budget support 2007
Ghana

• CRITICISM

• Evaluations addressing mainly THE PROCESS
• And not enough the CHAIN OF RESULTS
• IMPACTS AT THE GLOBAL LEVEL
• IMPACTS ON SECTOR POLICIES

3

• Expected Outputs of
• the Budget Support Evaluation

• An inventory of the inputs provided through
  budget support during a given period.
• An assessment of the performance of these inputs
  in terms of direct outputs and induced outputs.
• An identification of the changes which have
  occurred during the period in the partner country
  and the key causal factors driving those changes
  related to the budget support.
• An assessment of the extent to which budget
  support may have contributed to the results
  identified at the outcome and impact levels.
• An assessment of the overall relevance of budget
  support arrangements in the light of the evolving
  country and sector specific context, the aid
  policy and the related goals.

4

• Comprehensive Evaluation Methodology

• A Three components approach methodology combining
• An evaluation of the budget support itself
• An assessment of the budget support related
  outcomes and impacts of the government strategy
• An exploration of the linkages between the
  budget support process and some outcomes and
  impacts of the government strategy
• Focused on the main topics of the budget support
• Each topic is associated with one or several
  Evaluation Questions, Judgement Criteria and
  Indicators
• To ensure harmonisation of the evaluations in
  various contexts

5
GBS/SBS COMPREHENSIVE EVALUATION FRAMEWORK
6

• FIRST COMPONENT

EVALUATION OF THE BUDGET SUPPORT
Overview of the national context and its evolution
Entry conditions for budget support
Volume of funds, policy dialogue and
conditionalities, technical assistance and
capacity building, harmonisation and alignment
Assessment of the inputs provided by the budget
support
Assessment of the direct outputs of the budget
support
Effects of the identified inputs including
effects on transaction costs
Assessment of the induced outputs and the
corresponding government actions

Changes that budget support has induced in
comparison to what was expected
7

• SECOND COMPONENT

ASSESSMENT OF IMPACTS AND OUTCOMES
Government

• Economic growth
• Incidence of income and non income poverty
• Degree of empowerment and social inclusion
• Evolution of MDG indicators
• Achieving sector development objectives for SBS

Review of the impacts of the government
strategy specifically those related to the
budget support deliverables
e.g.
8

• SECOND COMPONENT

Causal factors

• Changes in business confidence and
  competitiveness
• Changes in the use and appreciation by the
  beneficiaries of goods and services provided by
  the public sector
• Changes in the degree of confidence of the
  general public sector

Identification of the factors that have
contributed to the key changes
e.g.
Identification of the factors external to the
government strategy which could have had an
impact on the mentioned changes

• Regional and international economic
• Political environment
• Other unforeseen event

e.g.
9

• SECOND COMPONENT

Relationship

• Positive linkages between government actions and
  improved outcomes
• and
• Failure or inappropriate actions which can have
  jeopardised the quality of the outcome

Identification of the outputs of the government
strategy which appear to have been of greatest
importance in generating change at outcomes level
10

• THIRD COMPONENT

ASSESSMENT OF THE INFLUENCE OF BUDGET
SUPPORT  INTEGRATION AND CONSISTENCIES BETWEEN
COMPONENTS 1 2 

• Comparison of the results of component 1 and
  component 2
• Improved understanding of the influence of
  budget support on the institutional and policy
  changes
• Increased consideration of the role of dominant
  factors in the determination of outcomes and
  impacts

If consistencies indicate causal links showing
positive conclusion for budget support
If no consistency find a possible explanation
11
GBS/SBS COMPREHENSIVE EVALUATION FRAMEWORK
Component 1
Component 3
Component 2
12
HOW TO MANAGE ANDIMPLEMENT THIS
EVALUATION?CHALLENGES AND PROPOSALS

• We must take into account
• THE STAKEHOLDERS
• THE LEAD OF THE EVALUATION
• THE CONSISTENCY, HARMONISATION AND DISSEMINATION
  ISSUES
• THE IMPLEMENTATION
• THE CAPACITY BUILDING ISSUE

13
STAKEHOLDERS

• To whom this evaluation is for?
• Are the donors the only stakeholders?
• Proposal 1
• All donors, the government and other main
  stakeholders involved in the budget support
  evaluated should be part of the evaluation.
• The evaluation of budget support operation in a
  designed country should be part of a multi-donor
  evaluation programme.

14
EVALUATION LEAD

• Who will take the lead for managing the
  evaluation?
• From where will the evaluation be managed?
• Proposal 2
• One of the main donors in the country should be
  chosen by the other donors to take the lead and
  consequently for contracting with the evaluators.
  Special arrangements should be set up to involve
  representatives in the country.

15
CONSISTENCY, HARMONISATION AND DISSEMINATION
ISSUES

• Proposal 3
• To ensure consistency, the headquarter
  Evaluation Service of the lead donor should be
  responsible for the evaluation methodology and
  quality assurance. Each donor can have dedicated
  persons in his Evaluation Service to go into the
  countries where the evaluation takes place, at
  crucial moments.
• Proposal 4
• A network of donors should be set up to ensure
  harmonisation and dissemination.

16
IMPLEMENTATION

• Who will do the evaluation?
• Who has currently the competencies to evaluate
  budget support and how to manage it?
• Proposal 5
• A team of external independent consultants
  familiar with the main topics of the evaluation
  should be contracted for each evaluation.

17
CAPACITY BUILDING

• Do we need a specific training on budget support
  evaluation?
• Proposal 6
• Make sure that few consultants are immediately
  operational and have been already trained in
  existing trainings
• Proposal 7
• Train the identified people within the
  Evaluation services of the Member States, some
  external independent consultants and the main
  governmental stakeholders of the recipient
  countries.

18

• Where are we ?

• Issue paper (May 2008)
• Steering group at OCDE / DAC level (June 2008)
• 1st meeting of a core group (September 2008)
• DAC Evaluation Network (18 19 November 2008)
• 2nd meeting of core group (December 2008)
• Common methodological document and standard terms
  of reference (early 2009)
• Testing methodological approach with partners
  agreement (2009)