Combining different methods in impact evaluation

1
Combining different methods in impact evaluation

• Presentation by Steen Folke,
• Senior researcher at the Danish Institute for
• International Studies
• International Workshop on Impact Evaluation
• Paris, November 15 2006

2
Important banalities

• Impact evaluation is not an exact science and
  it is dangerous to pretend that it is
• The use of a combination of different
  quantitative and qualitative methods can
  highlight different dimensions of impact
• Impact is the result of development processes
  that depend on the context as well as the
  intervention

3
Definition of impact

• Impact assessment is the systematic analysis of
  the lasting or significant changes positive or
  negative, intended or not in peoples lives
  brought about by a given action or series of
  actions
• (Chris Roche Impact Assessment for Development
  Agencies, Oxford 1999)

4
Classical effect evaluations

• Quasi-experimental survey design
• Ceteris paribus assumptions
• Before/after and/or with/without
• Quantitative methods
• Attempted objectivity
• Dubious assumptions

5
Participatory impact assessments

• Involvement of beneficiaries
• External facilitators
• Participatory techniques
• Qualitative methods
• Subjective
• Problems of reliability

6
Wider impact studies

• Heterogeneous category of in-depth studies
• Contextualised, tailor-made approach
• Unintended consequences as well as stated
  objectives
• Extensive fieldwork
• Combination of quantitative and qualitative
  methods
• Development interventions and societal processes

7
Model of development intervention, context,
process and impact
8
Noakhali Rural Development Project, Bangladesh

• Danida funding 1978-92 390 m. DKK (gt 50 m.)
  flagship in Danish aid
• 2 phases, 15 components irrigation,
  cooperatives, rural poor prog., mass education
  etc.
• gt60 expatriate advisers, staff gt1000
• Waning enthusiasm, many implementation problems
  (complexity)
• 3rd phase planned, but aborted

9
Ex-post impact evaluation

• Did the flagship float or sink?
• No terminal evaluation
• Ex-post impact study 9 years after
• Contextualised, tailor-made approach
• Combination of quantitative and qualitative
  methods
• 8 researchers, 15 assistants, 4 months fieldwork

10
Methods I Project focus

• Documentary study (project documents)
• Archival work in Danish embassy, Dhaka
• Questionnaire survey with former advisers and
  Danida staff
• Stakeholder interviews (Danida staff, former
  advisers, Bangladeshi staff)
• Quantitative analysis of project monitoring data

11
Methods II Study of context

• The national context books, articles, statistics
  (economy, policies)
• The local context census data, other sources
  (environment, population, socio-economic
  development)
• Institutional mapping (esp. NGOs)
• Extensive village studies (12 villages, 9 with
  and 3 without NRDP)
• Intensive village studies (4 NRDP villages,
  restudy 20 years later)

12
Methods III Quantitative

• Surveys of 5 important project components
  irrigation, infrastructure, fisheries,
  cooperatives for rural poor, mass education
• Random sampling (beneficiaries)
• Questionnaire-based interviews with beneficiaries
• Some interviews with non-beneficiaries (control
  group)

13
Methods IV Qualitative

• Assessment of roads, buildings and irrigation
  canals (function, maintenance)
• Key informant interviews
• Focus group interviews
• Observation
• In-depth interviews (issue-based and life
  stories)

14
Example of findings Irrigation IEconomic
situation of household
15
Example of findings Irrigation II

• Most important reason for improvement in
• economic situation compared to 15 years
• ago (total 59 respondents)
• Increased production (irrigation) 20 resp.
• Remittances 20 resp.
• Petty trade/business 8 resp.
• Other reasons 11 resp.
• Most important reasons for deterioration (17
  resp)
• Illness, large family, loss in business

16
Conclusion The project

• The Danida flagship did not sink the evaluation
  team found substantial impact in many areas 9
  years after
• The impact was primarily in the form of marginal
  improvements for the rural population, mainly the
  poor and women
• The socio-economic inequalities and the local
  power structure were not challenged

17
Conclusion The methods

• The combination of a range of different methods
  contributed to unravel different dimensions of
  impact from a complex project
• Study of the context was essential to understand
  impact of the project versus other factors
• Quantitative and qualitative methods were
  complementary
• Attempts to assess the counterfactual brought
  limited insight