Thinking about Impact Assessment - from an International Development Cooperation standpoint

1
Thinking about Impact Assessment - from an
International Development Cooperation standpoint

• Professor Elliot Stern, Lancaster University UK
• Presentation to ALNAP 24th Biannual
• Berlin December 2nd 2008

2
Thinking about Impact Assessment

• The argument I want to make
• Outcomes, results, effects and impacts
  are important
• But we need to think clearly about why this is
  so what methods are appropriate and in what
  circumstance

3
Thinking about Impact Assessment

• Evaluators have always been interested in
  outcomes, results, effects and impacts
• The balance between summative and formative,
  process and outcome evaluations has been
  argued about , and
• There have been legitimate criticisms that too
  much attention given to process not outcomes

4
Thinking about Impact Assessment

• There are often weaknesses in evaluation
• The balance of evaluative effort can be skewed
  towards processes unconnected to outcomes
• Methods adopted make little effort to disentangle
  what works from what is spurious from what is
  due to a particular intervention/initiative or to
  other causes
• Evaluators have been known only to be concerned
  for beneficiaries ignoring those who have
  missed out
• Initial success is privileged not longer term
  results

5
Thinking about Impact Assessment

• In development cooperation the OECD/DAC
  definition has emphasised duration in defining
  impacts impacts are
• long term effects produced by a development
  intervention
• Not all have accepted this distinction even in
  this particular policy domain, thus EU defines
  impact as
• A general term used to describe the effects of
  an intervention on society

6
Thinking about Impact Assessment

• There has been a general upsurge of interest in
  experimental, scientific which have informed
  the discourse about impact across many domains.
• This has been linked to medical trials -Cochrane
  Collaboration and similar moves in human services
  Campbell Collaboration and reinforced by US
  legislation requiring evaluations to be
  scientific

7
Thinking about Impact Assessment

• Impact has in this context been given a
  narrower methods-led meaning, to paraphrase Mohr
• A comparison of what happens with what would have
  happened had the intervention not been
  implemented
• From this perspective - the one advocated by
  Howard White at 3ie and the GDC - impact has
  become identified with attribution and the
  counterfactual and experimental methodologies
  associated with that understanding of science
  research
• (see GDC Report When Will We Ever Learn?
• Improving Lives through Impact Evaluation 2006)

8
Thinking about Impact Assessment

• This is not the first time this model has been
  advocated - it recurs. It is not generally
  accepted in the evaluation community as the only
  or superior approach but it is important.
• The battles that have gone on in the NONIE group
  and elsewhere have forced some acceptance of a
  mix of methods, fit for purpose and
  circumstance.
• But we would be wise to continue to distinguish
  between this and other approaches to impact

9
Thinking about Impact Assessment

• Why does attribution matter?
• Mainly because we need to disentangle what makes
  a difference, what works in the jargon, from
  changes that have nothing to do with our efforts
• Not simply was this initiative successful?
• Also
• Did this initiative/intervention make a
  difference that would not otherwise have
  happened?
• For example

10
Thinking about Impact Assessment
C
C
Figure 1
Figure 3
A
B
B
A
B(2)
C
Figure 2
A
B(1)
D
11
Thinking about Impact Assessment

• There are two complementary approaches to this
  problem
• Comparative methods, including before/after
  comparisons quasi experiments and full
  (randomised) experiments
• Theory-based methods including Theories of
  Change, causal modelling and realist analyses
• We usually need a mix!

12
Thinking about Impact Assessment

• There are 4 sets of considerations I would use
  when considering how to construct an approach to
  the evaluation of impacts
• The political agendas of the actors
• Technical issues of what is possible
• Arguments from the philosophy of science
• Ethical considerations

13
Thinking about Impact Assessment

• The narrow approach to impact does have political
  drivers although they are very diverse and the
  alliances are sometimes strange. Advocates want
  obviously to better meet social and economic
  needs. But they also may want to
• Legitimate (or de-legitimate) institutional and
  policy goals
• Simplify policies/find the silver bullet/reduce
  costs/risks
• Reduce public expenditure the nothing works
  agenda, again.. and on a smaller scale
• Occupational politics or careerism

14
Thinking about Impact Assessment

• Technical considerations are well-rehearsed. They
  include
• Problems constructing and maintaining control
  groups
• The practicalities of random allocation (central
  control, administrative capacity, resources)
• The risks of contamination
• The tendency to reductionism a focus on
  limited outcomes of interest
• The statistical power of measures (sample size)
• Trade-offs between internal validity and external
  validity hence our ability to generalise

15
Thinking about Impact Assessment

• I would want to distinguish between practical
  risks and logistical difficulties on the one hand
  and fitness for purpose
• Many objections to experimental
  quasi-experimental methods- when they are
  appropriate - can be overcome with careful
  attention to procedures and protocols but
  sometimes difficulties are rooted in the object
  and its context as well as in methods

16
Thinking about Impact Assessment

• We can compare three scenarios
• S1 Standardized interventions in identical
  settings with common beneficiaries
• S2 Standardized interventions in diverse
  settings, possibly with diverse beneficiaries
• S3 Customized interventions in diverse settings
  with diverse beneficiaries

17
Thinking about Impact Assessment

• These scenarios necessarily pull for different
  methodologies
• Scenario 1 is better adapted to experiments
• Scenario 2 is better adapted to quasi experiments
  comparisons (contingent and realist) and
  combinations of methods
• Scenario 3 is better adapted to case studies or
  narrative/qualitative approaches that build
  plausible theories

18
Thinking about Impact Assessment

• Experiments do tend to favour single inputs and
  outcomes that can be delivered in discrete
  packages (not embedded) and that have a
  relatively short implementation chain in the
  sense both of time and complexity/ease of
  implementation and where the intervention is
  repeated often (large n)
• They are best for projects that deliver a known
  service to large numbers of recipients are
  possible for relatively simple programmes
  unsuited to complex multi-measure strategies/
  policies.

19
Thinking about Impact Assessment

• This is acknowledged even by protagonists of
  RCTs. For example Esther Duflo of the MIT Poverty
  Lab has noted
• randomised evaluations are not suitable for all
  types of programmes. They are suitable for
  programmes that are targeted to individuals or
  communities, and where the objectives are well
  defined. For example, the efficacy of foreign aid
  disbursed as general budget support cannot be
  evaluated in this way.

20
Thinking about Impact Assessment

• There is however a danger that advocates of
  narrower impact approaches will press to
  redefine policy measures so that they become
  evaluable through their preferred methods. As
  Duflo went on to say
• It may be desirable, for efficiency or political
  reasons, to disburse some fraction of aid in this
  form GBS, although it would be extremely costly
  to distribute all the foreign aid in the form of
  general budget support, precisely because it
  leaves no place for rigorous evaluation of
  projects. (Italics added)

21
Thinking about Impact Assessment

• In international development cooperation, there
  is a tendency for advocates of impact
  approaches to also favour sectoral, targeted
  programmes (sometimes called vertical
  interventions) rather than policies that seek to
  address wider issues of governance and
  institution-building such as General Budget
  Support or the Paris Declaration - arguing that
  sectoral programmes are both likely to be more
  effective and are often cheaper to deliver of
  timely relevance given MDG goals

22
Thinking about Impact Assessment

• Philosophical objections to experiments (and
  randomisation in particular) go to the heart of
  hard-fought debates about causality in the social
  sciences. These are variously
• Epistemological what we know and how
• Ontological - the nature of knowledge
• Methodological the possibilities of data
  collection and analysis
• To pick up on a few examples of these debates .

23
Thinking about Impact Assessment

• Newtonian science assumed that we can observe
  regularities or patterns of individual phenomena
  from the outside this allows for consistent
  explanations explanation can be derived
  empirically
• Most contemporary understandings of causality are
  theory based assume we cannot observe causal
  mechanisms we need to open up the black-box
  because a) causal mechanisms are often hidden and
  b) are often unstable e.g. are context specific
• Hence difficulty in finding Humean general laws!

24
Thinking about Impact Assessment

• If we follow this line of arguments it is
  unlikely we will ever be able to consistently
  demonstrate what works even for relatively
  straightforward projects and programmes across
  all contexts and circumstances evidence remains
  a matter of probability and estimation not
  certainty or truth

25
Thinking about Impact Assessment

• Which is why there is a need for
• Multi-methods that can be triangulated
• Theory based approaches to understand
  mechanisms that cannot be fully observed
• Distinguishing between causality explanation
• Recognising the limits of proof and certainty
• Understanding and typologising contexts
• Linking process evaluations with outcome/impacts
  so as to understand a) what is being implemented
  and b) what accounts for divergence/diversity

26
Thinking about Impact Assessment

• The ethical difficulties that all applied
  research faces are also well documented
• Treating people as actors with agency and will
  rather than as passive objects
• Denying an intervention from some if it is needed
  in order to achieve randomisation
• Although the latter can be an unfounded it would
  be consistent with counterfactual logic to offer
  alternatives in terms of service rather than
  something/nothing the focus of experiments are
  often modes of delivery not the actual service

27
Thinking about Impact Assessment

• To conclude
• We do need to focus more on outcomes/effects/impac
  ts
• Comparisons (including experiments) are important
  as are model/theory building
• We need to accept that as initiatives become more
  complex and multi-measure so certainty and
  predictability about what works will diminish
• We should neither be put-off nor seduced by the
  promises of experimentalists they offer many
  things but in a limited set of circumstances
  as the wise ones among them admit!

28
Thinking about Impact Assessment