Phase 2 Evaluation of the PD

1
Phase 2 Evaluation of the PD

• Evaluation
• Methodology
• Reference Group Workshop/Meeting 11 13
  February 2009

2
Methodological Issues

• The broad choice of evaluation methodology, as we
  discussed yesterday, follows on from the
  evaluation questions chosen. However there are
  more specific methods-related issues that need to
  be thought about at this stage.

3
Methodological Issues

• One way of focussing on methodology is to
  consider what methodological standards or
  qualities we expect this evaluation to have.
  (You will notice a certain overlap with the
  previous workshop on quality standards!)

4
Methodological Issues

• For example, it can be argued that this
  evaluation should end up with the following
  qualities
• A balanced sufficient sample of countries
• Sufficient coverage of sectors and themes
• Information of good quality
• Offering the possibility of explanation and
  attribution

5
Methodological Issues

• If an evaluation has these qualities, it will
  allow us to generalise to some extent across
  different PD settings (external validity) be
  confident that our measurements or descriptions
  are consistent (reliability) and have
  confidence in the strength or power of findings
  that they are sufficiently supported by the
  evidence collected and analysed

6
Methodological Issues

• Making the right decisions about methodology at
  the beginning will make an evaluation defensible,
  able to withstand criticism when it eventually
  reports
• Lets take each of the attributes in turn

7
Methodological Issues

• A balanced sufficient sample of countries
• When policy makers ask about the PD they want to
  know whether we are able to say something about
  countries in different geographical regions
  those that are more and less aid dependent and
  both those that have strong institutions and
  governance and others that have some elements of
  fragility perhaps because they are still
  recovering from wars or conflicts. We also need
  enough countries to be able to support
  conclusions for the PD as a whole

8
Methodological Issues

• Sufficient coverage of sectors and themes
• Countries selected need to cover the main policy
  areas and sectors recognised as important for
  development for example healthcare, encouraging
  small businesses, education, progress towards the
  MDGs, international trade support. They also need
  to include important themes such as capacity
  development, civil society participation, donor
  harmonisation, improving governance reducing
  fragility. This will allow sensible comparisons
  to be made across cases

9
Methodological Issues

• Information of good quality
• This requires
• Available information one rationale for
  selecting some sectors/themes
• Willingness to use innovative data sources
• Ensuring that all pre-existing sources are
  synthesised, reviewed and exploited
• Cross-checking (triangulating) across multiple
  sources of information
• Expending more time relatively on data collection

10
Methodological Issues

• Offers the possibility of explanation
  attribution
• There are two classic ways we can attempt to
  explain. First through longitudinal analyses that
  follow a causal chain over time this is the
  basis of time-series data and panel studies as
  well as causal modelling, tracker-studies
  theory-based evaluations. Second we can compare
  across places, settings or time periods
  including before after studies comparison
  groups quasi experiments and full controlled
  experiments

11
Methodological Issues

• On this basis, we can put together a possible
  starting list of methods that could be used in
  Phase 2.
• The list would include

12
Methodological Issues

• Synthesis reviews of existing evaluations,
  research and indicator systems
• Comparative in-depth case studies (of country
  partnerships) which are chosen to contain a good
  cross-section of common themes/sectors
• Longitudinal studies either forward looking
  (theory-based mapping of plausible directions
  of travel) or backward looking tracking back to
  PD-like, longer established policies
• Targeted comparative studies to supplement
  country based case comparisons

13
Methodological Issues

• Methodologies have to be understood as more than
  analytic tools. How they are resourced and
  steered will determine their value as much as
  their technical sophistication. To take two
  examples

14
Methodological Issues

• Putting together the best team of experts to
  undertake the evaluation at country central
  levels will be challenging. It may require
  bringing together public sector and civil society
  expertise national and possibly regional
  resources skills.

15
Methodological Issues

• National reference groups will need to open up
  access for information and cooperation
  institutionally and across government to
  safeguard the independence and credibility of the
  evaluation build bridges so as to make it more
  likely that evaluation outputs will be used and
  useful

16
Methodological Issues

• These are some of the issues that needs more
  discussion before the Terms of Reference for this
  evaluation are prepared and will therefore be
  taken up in the group discussion session that
  follows
• ..after any points of clarification