Using Impact Evaluation for Results Based Policy Making - PowerPoint PPT Presentation

1 / 40
About This Presentation
Title:

Using Impact Evaluation for Results Based Policy Making

Description:

Slides by Paul J. Gertler & Sebastian Martinez. Answer Three Questions ... Does replacing dirt floors with cement reduce parasites & improve child health? ... – PowerPoint PPT presentation

Number of Views:48
Avg rating:3.0/5.0
Slides: 41
Provided by: WB1673
Category:

less

Transcript and Presenter's Notes

Title: Using Impact Evaluation for Results Based Policy Making


1
Using Impact Evaluation for Results Based Policy
Making
  • Arianna Legovini
  • Impact Evaluation Cluster, AFTRL

Slides by Paul J. Gertler Sebastian Martinez
2
Answer Three Questions
  • Why is evaluation valuable?
  • What makes a good impact evaluation?
  • How to implement evaluation?

3
IE Answers How do we turn this teacher
4
into this teacher?
5
Why Evaluate?
  • Need evidence on what works
  • Limited budget forces choices
  • Bad policies could hurt
  • Improve program/policy implementation
  • Design eligibility, benefits
  • Operations efficiency targeting
  • Information key to sustainability
  • Budget negotiations
  • Informing beliefs and managing press

6
Allocate limited resources?
  • Benefit-Cost analysis
  • Comparison of choices
  • Highest return investment
  • Benefit
  • Change in outcome indicators
  • Measured through impact evaluation
  • Cost
  • Additional cost of providing benefit
  • Not accounting cost

7
Traditional M E
  • Monitoring
  • Outcome trends over time
  • e.g. poverty, school enrollment, mortality
  • Process Evaluation
  • Implementation
  • Efficiency
  • Targeting
  • Administrative Data
  • Management Information Systems

8
Impact Evaluation Answers
  • What is effect of program on outcomes?
  • How much better off are beneficiaries
    because of the intervention?
  • How would outcomes change under alternative
    program designs?
  • Does the program impact people differently (e.g.
    females, poor, minorities)
  • Is the program cost-effective?
  • Traditional ME cannot answer these

9
For Example IE Answers
  • What is the effect of Job Training on employment
    and earnings?
  • How much do cash transfers lower poverty?
  • Do scholarships increase on school attendance for
    girls more than boys?
  • Does contracting out primary health care to
    private sector lead to an increase in access?
  • Does replacing dirt floors with cement reduce
    parasites improve child health?
  • Do improved roads increase access to labor
    markets raise income for the poor?

10
Types of Impact Evaluation
  • Efficacy
  • Proof of Concept
  • Pilot under ideal conditions
  • Effectiveness
  • Normal circumstances capabilities
  • Impact will be lower
  • Impact at higher scale will be different
  • Costs will be different as there are
    economies of scale from fixed costs

11
So, Use impact evaluation to.
  • Scale up pilot-interventions/programs
  • Kill programs
  • Adjust program benefits
  • Inform (i.e. Finance Press)
  • e.g. PROGRESA/OPORTUNIDADES (Mexico)
  • Transition across presidential terms
  • Expansion to 5 million households
  • Change in benefits
  • Battle with the press

12
Next question please
  • Why is evaluation valuable?
  • What makes a good impact evaluation?
  • How to implement evaluation?

13
Assessing impact
  • examples.
  • How much does an anti-poverty program lower
    poverty?
  • What is beneficiarys income with program
    compared to without program?
  • Compare same individual with without programs
    at same point in time
  • Never observe same individual with and without
    program at same point in time

14
Solving the evaluation problem
  • Counterfactual what would have
    happened without the program
  • Need to estimate counterfactual
  • i.e. find a control or comparison group
  • Counterfactual Criteria
  • Treated counterfactual groups have identical
    characteristics on average,
  • Only reason for the difference in
    outcomes is due to the intervention

15
2 Counterfeit Counterfactuals
  • Before and after
  • Same Individual before the treatment
  • Non-Participants
  • Those who choose not to enroll in program
  • Those who were not offered the program

16
Before and After Examples
  • Agricultural assistance program
  • Financial assistance to purchase inputs
  • Compare rice yields before and after
  • Find fall in rice yield
  • Did the program fail?
  • Before is normal rainfall, but after is drought
  • Could not separate (identify) effect of financial
    assistance program from effect of rainfall
  • School scholarship program on enrollment

17
Before and After
  • Compare Y before and after intervention
  • A-B Estimated Impact
  • B counterfactual Estimate
  • Does not control for time varying factors
  • C True counterfactual
  • A-C True impact
  • A-B is under-estimate

Y
After
Before
A
B
C
t-1
t
Time
18
Non-Participants.
  • Compare non-participants to participants
  • Counterfactual non-participant outcomes
  • Problem why did they not participate?

19
Job training program example
  • Eligible group offered job training
  • Compare employment earning of those who sign up
    to those who did not
  • Who signs up?
  • Those who are most likely to benefit,
    i.e. those with more ability
  • Would have higher earnings than non-participants
    without job training
  • Poor estimate of counterfactual

20
Health Insurance Example
  • Health insurance offered
  • Compare health care utilization of those who got
    insurance to those who did not
  • Who buys health insurance?
  • Expect large medical expenditures
  • Less healthy
  • Who does not buy? The healthy!
  • Cannot separately identify impact of insurance
    from health on utilization

21
What's wrong?
  • Selection bias People choose to participate for
    specific reasons
  • Many times reasons are directly related
    to the outcome of interest
  • Job Training ability and earning
  • Health Insurance health status
    and medical expenditures
  • Cannot separately identify impact of the program
    from these other factors/reasons

22
Program placement example
  • Govt offers family planning program to villages
    with high fertility
  • Compare fertility in villages offered program to
    fertility in villages not offered
  • Program targeted based on fertility, so
  • Treatments have high fertility
  • Counterfactuals have low fertility
  • Cannot separately identify program impact from
    geographic targeting criteria

23
Need to know
  • Why some get program and others not
  • How beneficiaries get into treatment versus
    control group
  • If reasons correlated w/ outcome
  • cannot identify/separate program impact from
  • other explanations of differences in outcomes
  • The process by which data is generated

24
Possible Solutions
  • Guarantee comparability of treatment
    and control groups
  • ONLY remaining difference is intervention
  • In this seminar we will consider
  • Experimental design/randomization
  • Quasi-experiments
  • Regression Discontinuity
  • Double differences
  • Instrumental Variables

25
These solutions all involve
  • knowing how the data are generated
  • Randomization
  • Give all equal chance of being in
    control or treatment groups
  • Guarantees that all factors/characteristics will
    be on average equal btw groups
  • Only difference is the intervention
  • If not, need transparent observable criteria
    for who is offered program

26
The Last Question
  • Why is evaluation valuable?
  • What makes a good impact evaluation?
  • How to implement evaluation?

27
Implementation Issues
  • Policy relevance
  • Political Economy
  • Finding a good control group.
  • Retrospective versus prospective designs
  • Making the design compatible with operations
  • Ethical Issues
  • Relationship to results monitoring

28
The Policy Context
  • IE needs answers policy questions
  • What policy questions need to be answered?
  • What outcomes answer those questions?
  • What indicators measures outcomes?
  • How much of a change in the outcomes
    would determine success?
  • Example teacher performance-based pay
  • Scale up pilot?
  • Criteria Need at least a 10 increase in test
    scores with no change in unit costs

29
Political Economy
  • Is IE needed for some policy purpose?
  • Ex ante build into institutions of government
    decision-making
  • Stakeholders Collaboration btw country,
    stakeholders evaluation team
  • How will negative results affect program
    managers, policy makers stakeholders?
  • Job performance vs knowledge generation
  • Reward for using IE to change/close
    weak programs

30
Two paths to Control Groups
  • Retrospective (very hard)
  • Try to evaluate after program implemented
  • Statistically model how governments individuals
    made allocation choices
  • Cannot alter treatment or control group
  • Prospective
  • Can introduce some reasons for participation that
    are uncorrelated with outcomes
  • Complement operational objectives
  • Easier and more robust

31
Easier in prospective designs
  • Generate good control groups
  • Most interventions cannot immediately deliver
    benefits to all those legible
  • Budgetary limitations
  • Logistical limitations
  • Typically phased in
  • Those who go first are potential treatments
  • Those who go later are potential controls
  • Use Rollout to find control groups

32
Who goes first among equals?
  • Cost considerations
  • What is most efficient scale to deliver program
  • Operations, social and political costs
  • Individual/household or community?
  • e.g. welfare program, roads, health insurance
  • Eligibility criteria
  • Are benefits targeted?
  • How are they targeted?
  • Can we rank eligible's priority?
  • Are measures good enough for fine rankings?

33
Ethical Considerations
  • Do not delay benefits Rollout based on
    budget/administrative constraints
  • Equity equally deserving beneficiaries deserve
    an equal chance of going first
  • Transparent accountable method
  • Give everyone eligible an equal chance
    (e.g. Colombia School Vouchers, Mexico Tu
    Casa)
  • If rank based on some criteria, then criteria
    should me quantitative and public

34
Retrospective Designs
  • Hard to find good control groups
  • Must live with arbitrary allocation rules
  • Many times rules not transparent
  • Administrative data must
  • be good enough to make sure program
    was implemented as described
  • identify beneficiaries, otherwise
    surveys will be costly
  • Unless originally randomized, need
    pre-intervention baseline survey
  • both controls and treatments

35
Retrospective evaluation.
  • Need to control for differences between control
    treatment groups
  • Unless have baseline difficult to use
    quasi-experimental methods
  • Sometimes can do it with baseline if
  • Know why beneficiaries are beneficiaries
  • Observable criteria for program rollout

36
IE and Monitoring Systems
  • Projects/programs regularly collect
    data for management purposes
  • Typical content
  • Lists of beneficiaries
  • Distribution of benefits
  • Expenditures
  • Outcomes
  • Ongoing process evaluation
  • Key for impact evaluation

37
Monitoring systems key
  • Verify who is beneficiary
  • When started
  • What benefits were actually delivered
  • Compliance with any conditionalities
  • Necessary condition for program
    to have an impact
  • benefits need to get to targeted beneficiaries
  • program implemented as designed

38
Use Monitoring data for IE
  • Program monitoring data usually only collected in
    areas where active
  • If start in control areas at same time as in
    treatment areas have baseline for both
  • Add a couple of outcome indicators
  • Very cost-effective as little need
    for additional special surveys
  • Most IEs use only monitoring

39
Overall Messages
  • Impact evaluation useful for
  • Validation program design
  • Adjusting program structure
  • Communicating to finance ministry
    civil society
  • A good evaluation design requires estimating the
    counterfactual
  • What would have happen to beneficiaries
    if had not received the program
  • Need to know all reasons why beneficaries got
    program others did not

40
Design Messages
  • Address policy questions
  • Institutional use of results
  • Stakeholder buy-in
  • Easiest to use prospective designs
  • Take advantage of phase rollout
  • Transparency accountability use quantitative
    and public criteria
  • Equity give eligibles equal chance of going 1st
  • Good monitoring systems administrative data can
    improve IE and lower costs
Write a Comment
User Comments (0)
About PowerShow.com