Title: Using Impact Evaluation for Results Based Policy Making
1Using Impact Evaluation for Results Based Policy
Making
- Arianna Legovini
- Impact Evaluation Cluster, AFTRL
Slides by Paul J. Gertler Sebastian Martinez
2Answer Three Questions
- Why is evaluation valuable?
- What makes a good impact evaluation?
- How to implement evaluation?
3IE Answers How do we turn this teacher
4into this teacher?
5Why Evaluate?
- Need evidence on what works
- Limited budget forces choices
- Bad policies could hurt
- Improve program/policy implementation
- Design eligibility, benefits
- Operations efficiency targeting
- Information key to sustainability
- Budget negotiations
- Informing beliefs and managing press
6Allocate limited resources?
- Benefit-Cost analysis
- Comparison of choices
- Highest return investment
- Benefit
- Change in outcome indicators
- Measured through impact evaluation
- Cost
- Additional cost of providing benefit
- Not accounting cost
7Traditional M E
- Monitoring
- Outcome trends over time
- e.g. poverty, school enrollment, mortality
- Process Evaluation
- Implementation
- Efficiency
- Targeting
- Administrative Data
- Management Information Systems
8Impact Evaluation Answers
- What is effect of program on outcomes?
- How much better off are beneficiaries
because of the intervention? - How would outcomes change under alternative
program designs? - Does the program impact people differently (e.g.
females, poor, minorities) - Is the program cost-effective?
- Traditional ME cannot answer these
9For Example IE Answers
- What is the effect of Job Training on employment
and earnings? - How much do cash transfers lower poverty?
- Do scholarships increase on school attendance for
girls more than boys? - Does contracting out primary health care to
private sector lead to an increase in access? - Does replacing dirt floors with cement reduce
parasites improve child health? - Do improved roads increase access to labor
markets raise income for the poor?
10Types of Impact Evaluation
- Efficacy
- Proof of Concept
- Pilot under ideal conditions
- Effectiveness
- Normal circumstances capabilities
- Impact will be lower
- Impact at higher scale will be different
- Costs will be different as there are
economies of scale from fixed costs
11So, Use impact evaluation to.
- Scale up pilot-interventions/programs
- Kill programs
- Adjust program benefits
- Inform (i.e. Finance Press)
- e.g. PROGRESA/OPORTUNIDADES (Mexico)
- Transition across presidential terms
- Expansion to 5 million households
- Change in benefits
- Battle with the press
12Next question please
- Why is evaluation valuable?
- What makes a good impact evaluation?
- How to implement evaluation?
13Assessing impact
- examples.
- How much does an anti-poverty program lower
poverty? - What is beneficiarys income with program
compared to without program? -
- Compare same individual with without programs
at same point in time - Never observe same individual with and without
program at same point in time
14Solving the evaluation problem
- Counterfactual what would have
happened without the program - Need to estimate counterfactual
- i.e. find a control or comparison group
- Counterfactual Criteria
- Treated counterfactual groups have identical
characteristics on average, - Only reason for the difference in
outcomes is due to the intervention
152 Counterfeit Counterfactuals
- Before and after
- Same Individual before the treatment
- Non-Participants
- Those who choose not to enroll in program
- Those who were not offered the program
16Before and After Examples
- Agricultural assistance program
- Financial assistance to purchase inputs
- Compare rice yields before and after
- Find fall in rice yield
- Did the program fail?
- Before is normal rainfall, but after is drought
- Could not separate (identify) effect of financial
assistance program from effect of rainfall - School scholarship program on enrollment
17Before and After
- Compare Y before and after intervention
- A-B Estimated Impact
- B counterfactual Estimate
- Does not control for time varying factors
- C True counterfactual
- A-C True impact
- A-B is under-estimate
Y
After
Before
A
B
C
t-1
t
Time
18Non-Participants.
- Compare non-participants to participants
- Counterfactual non-participant outcomes
- Problem why did they not participate?
19Job training program example
- Eligible group offered job training
- Compare employment earning of those who sign up
to those who did not - Who signs up?
- Those who are most likely to benefit,
i.e. those with more ability - Would have higher earnings than non-participants
without job training - Poor estimate of counterfactual
20Health Insurance Example
- Health insurance offered
- Compare health care utilization of those who got
insurance to those who did not - Who buys health insurance?
- Expect large medical expenditures
- Less healthy
- Who does not buy? The healthy!
- Cannot separately identify impact of insurance
from health on utilization
21What's wrong?
- Selection bias People choose to participate for
specific reasons - Many times reasons are directly related
to the outcome of interest - Job Training ability and earning
- Health Insurance health status
and medical expenditures - Cannot separately identify impact of the program
from these other factors/reasons
22Program placement example
- Govt offers family planning program to villages
with high fertility - Compare fertility in villages offered program to
fertility in villages not offered - Program targeted based on fertility, so
- Treatments have high fertility
- Counterfactuals have low fertility
- Cannot separately identify program impact from
geographic targeting criteria
23Need to know
- Why some get program and others not
- How beneficiaries get into treatment versus
control group - If reasons correlated w/ outcome
- cannot identify/separate program impact from
- other explanations of differences in outcomes
- The process by which data is generated
24Possible Solutions
- Guarantee comparability of treatment
and control groups -
- ONLY remaining difference is intervention
- In this seminar we will consider
- Experimental design/randomization
- Quasi-experiments
- Regression Discontinuity
- Double differences
- Instrumental Variables
25These solutions all involve
- knowing how the data are generated
- Randomization
- Give all equal chance of being in
control or treatment groups - Guarantees that all factors/characteristics will
be on average equal btw groups - Only difference is the intervention
- If not, need transparent observable criteria
for who is offered program
26The Last Question
- Why is evaluation valuable?
- What makes a good impact evaluation?
- How to implement evaluation?
27Implementation Issues
- Policy relevance
- Political Economy
- Finding a good control group.
- Retrospective versus prospective designs
- Making the design compatible with operations
- Ethical Issues
- Relationship to results monitoring
28The Policy Context
- IE needs answers policy questions
- What policy questions need to be answered?
- What outcomes answer those questions?
- What indicators measures outcomes?
- How much of a change in the outcomes
would determine success? - Example teacher performance-based pay
- Scale up pilot?
- Criteria Need at least a 10 increase in test
scores with no change in unit costs
29Political Economy
- Is IE needed for some policy purpose?
- Ex ante build into institutions of government
decision-making - Stakeholders Collaboration btw country,
stakeholders evaluation team - How will negative results affect program
managers, policy makers stakeholders? - Job performance vs knowledge generation
- Reward for using IE to change/close
weak programs
30Two paths to Control Groups
- Retrospective (very hard)
- Try to evaluate after program implemented
- Statistically model how governments individuals
made allocation choices - Cannot alter treatment or control group
- Prospective
- Can introduce some reasons for participation that
are uncorrelated with outcomes - Complement operational objectives
- Easier and more robust
31Easier in prospective designs
- Generate good control groups
- Most interventions cannot immediately deliver
benefits to all those legible - Budgetary limitations
- Logistical limitations
- Typically phased in
- Those who go first are potential treatments
- Those who go later are potential controls
- Use Rollout to find control groups
32Who goes first among equals?
- Cost considerations
- What is most efficient scale to deliver program
- Operations, social and political costs
- Individual/household or community?
- e.g. welfare program, roads, health insurance
- Eligibility criteria
- Are benefits targeted?
- How are they targeted?
- Can we rank eligible's priority?
- Are measures good enough for fine rankings?
33Ethical Considerations
- Do not delay benefits Rollout based on
budget/administrative constraints - Equity equally deserving beneficiaries deserve
an equal chance of going first - Transparent accountable method
- Give everyone eligible an equal chance
(e.g. Colombia School Vouchers, Mexico Tu
Casa) - If rank based on some criteria, then criteria
should me quantitative and public
34Retrospective Designs
- Hard to find good control groups
- Must live with arbitrary allocation rules
- Many times rules not transparent
- Administrative data must
- be good enough to make sure program
was implemented as described - identify beneficiaries, otherwise
surveys will be costly - Unless originally randomized, need
pre-intervention baseline survey - both controls and treatments
35Retrospective evaluation.
- Need to control for differences between control
treatment groups - Unless have baseline difficult to use
quasi-experimental methods - Sometimes can do it with baseline if
- Know why beneficiaries are beneficiaries
- Observable criteria for program rollout
36IE and Monitoring Systems
- Projects/programs regularly collect
data for management purposes - Typical content
- Lists of beneficiaries
- Distribution of benefits
- Expenditures
- Outcomes
- Ongoing process evaluation
- Key for impact evaluation
37Monitoring systems key
- Verify who is beneficiary
- When started
- What benefits were actually delivered
- Compliance with any conditionalities
- Necessary condition for program
to have an impact - benefits need to get to targeted beneficiaries
- program implemented as designed
38Use Monitoring data for IE
- Program monitoring data usually only collected in
areas where active - If start in control areas at same time as in
treatment areas have baseline for both - Add a couple of outcome indicators
- Very cost-effective as little need
for additional special surveys - Most IEs use only monitoring
39Overall Messages
- Impact evaluation useful for
- Validation program design
- Adjusting program structure
- Communicating to finance ministry
civil society - A good evaluation design requires estimating the
counterfactual - What would have happen to beneficiaries
if had not received the program - Need to know all reasons why beneficaries got
program others did not
40Design Messages
- Address policy questions
- Institutional use of results
- Stakeholder buy-in
- Easiest to use prospective designs
- Take advantage of phase rollout
- Transparency accountability use quantitative
and public criteria - Equity give eligibles equal chance of going 1st
- Good monitoring systems administrative data can
improve IE and lower costs