Methodologies for Impact Evaluation: Quantitative vs Qualitative Methods

1
Methodologies for Impact Evaluation
Quantitative vs Qualitative Methods

• Prepared for the course on Evaluating the Impact
  of Projects and Programs, April 10-14, 2006,
  Beijing China.
• Contact Ywang2_at_worldbank.org

2
Outlines

• Multi-dimensional nature of poverty, and
  methodologies for Impact evaluation
• Quantitative methods were introduced earlier
• Introduction to Qualitative methods
• The Why and the What
• The how to issues Evaluation methods and data
  requirements
• The benefits and drawbacks
• Examples

3
Methodologies of Impact Evaluation

• Poverty and inequality are multi-dimensional
  income poverty, income inequality, also
  deprivation of basic needs food, nutrition,
  shelter, and clothing, and access to basic social
  services health and education
• The MDGs have 11 goals which are mostly
  quantifiable, but others are difficult to
  quantify
• This requires that both quantitative and
  qualitative methods must be used for impact
  evaluation.
• Quantitative methods were discussed yesterday.
• This module focuses on qualitative methods.

4
Qualitative methods the what

• Qualitative methods determine causal relationship
  by methods other than establishing
  counterfactuals
• They rely on understanding processes, behaviors,
  and conditions as perceived by individuals or
  groups
• Examples how a project is perceived to have
  affected peoples lives, how a training program
  is perceived to affect learning

5
Questions to be answered by Qualitative methods

• How was the program/project perceived to have
  affected the individual/group?
• Has the individual /group had the opportunity to
  participate in the project?
• Was this social service perceived accessible to
  this individual/ group?
• Was this information perceived to be available to
  this individual/ group?
• Has the benefit reached the intended target group?

6
The how to use issues Please note.

• Qualitative data can also be quantified, and
  analyzed using quantitative methods (such as
  logit, tobit and probit regressions)
• For Impact evaluation, establishing the
  counterfactual is so essential, thus, qualitative
  methods are often used as a complementary methods
  for quantitative analysis.
• e.g. Rapid rural assessment which rely on
  participants knowledge of the conditions
  surrounding the project/program being evaluated
• Participatory evaluation in which stakeholders
  are involved in all stages of evaluation. See
  World Bank 1996, The World Bank Participation
  Sourcebook.

7
Main data collection instrumentsfor Qualitative
methods

• These methods are often used in qualitative
  analysis
• Case studies pros and cons
• Focus groups pros and cons
• Interviews pros and cons
• Observations pros and cons

8
Benefits of Qualitative methods

• Benefits they are flexible, can be tailored to
  the needs using open-ended approaches,
• can be carried out quickly using rapid techniques
  and
• can provide a better understanding of
  stakeholders perceptions and priorities and
  conditions and processes that may have affected
  program impact

9
Drawbacks of this method

• Subjectivity involved in data collection
• The lack of a comparison group not possible to
  establish counterfactual, and thus the causality
• The lack of statistical robustness given small
  sample sizes
• Thus the results cannot be generalized to a
  larger population
• The validity and reliability depend on the skill
  and capacity of the evaluator /interviewer

10
Summary

• Objective of IE is to determine the causal
  relationship between the program/project and the
  improvements in the livelihood of the
  beneficiaries.
• The key problem is to establish a good
  counterfactual net out the influence of other
  factors such as growth
• Qualitative methods are not perfect they have
  many problems because of the difficulties in
  establishing counterfactuals.
• They can often be used in conjunction with
  quantitative methods.
• Dr. Zhang will elaborate on the analytical tools
  and methodological issues when using qualitative
  methods.