Ex Ante Program Evaluation

1
Ex Ante Program Evaluation

• Petra E. Todd
• University of Pennsylvania
• (Based on joint work with Ken Wolpin)

2
Overview

• Ex Ante vs. Ex Post Approaches
• Examples of how behavioral models are required
  for ex ante evaluation estimators
• Functional forms not necessarily required
• Types of programs Wage subsidies, income
  subsidies, schooling subsidies
• Application study the performance of
  nonparametric ex ante evaluation estimators using
  data from the PROGRESA randomized experiment

3
Ex Post Evaluation Methods

• Evaluate program impacts after implementation
• Alternative Approaches
• Randomization
• Difference-in-Difference
• Matching (Cross-sectional and Difference-in-differ
  ence)
• Control function methods
• Regression-Discontinuity
• IV Methods, MTE, Local IV (LIV), LATE
• All methods require data on a treatment group and
  on a comparison group

4
Advances in Ex-post Evaluation

• Matching
• Does not require functional form assumption on
  the outcome equation (Rosenbaum and Rubin, 1983)
• Propensity scores can be estimated
  semi-parametrically, (Heckman, Ichimura
  and Todd, 1997, Buchinsky, 1998)
• Regression-Discontinuity (RD) method
• Requires discontinuity in the probability of
  receiving treatment (Hahn, Todd and Van der
  Klaauw, 2001)
• Does not require specifying the functional form
  of the outcome equation
• Control function methods
• Implementable without distributional assumptions
  on the error terms in the participation and
  outcome equations (e.g. Heckman, 1980, Newey
  (1988), Andrews (1991))
• Usually requires an exclusion restriction

5

• IV estimators
• Has LATE interpretation under weak assumptions
  (e.g. Imbens and Angrist, 1994)
• MTE, LIV estimators (Heckman and Vytlacil (2005))
• Require a continuous instrument
• Permit investigation of program impact
  heterogeneity
• Relax assumptions about additive separability of
  error terms

6
Goals of Ex Ante Evaluation

• Predict program impacts prior to implementation
• Needed for optimal program design and placement
• Requires simulating program effects and costs
  (take-up rates)
• Experimental approach often not feasible (high
  cost, time delay)
• Identify range of potential impacts, helpful in
  choosing sample sizes for future evaluation
• Evaluate effects of counterfactual programs
• Study how impacts change if parameters of an
  existing program are altered
• For example, changing school subsidy levels
• Evaluate effects of longer terms of exposure than
  are observed in the data

7
Using Static Models

• Forecast demand for a new good prior to its being
  introduced into the choice set
• e.g. McFadden (1977) BART subway
• Impose structure on utility function and on the
  distribution of the error terms (e.g.
  multivariate probit or logit)
• Forecast effect of changing the characteristics
  of a good
• Berry, Levensohn, Pakes (1985) changing car
  characteristics (e.g. price, fuel efficiency)

8
Using Dynamic Models

• Impose functional form assumptions on utility
  function and on the joint distribution of error
  terms
• Evaluate model performance by comparing forecast
  based on structural predictions to experimental
  results
• Wise (1985) effect of housing subsidy on
  housing demand
• Lumsdaine, Stock and Wise (1992) retirement
  bonus
• Lise, Seitz, and Smith (2003) welfare bonus
  program
• Todd and Wolpin (2006) effects of Mexican
  school subsidy program

9
Early Efforts to Relax Functional Forms for Ex
Ante Evaluation

• Marschak (1953) and Hurwicz (1962)
• Observe that it is not necessary to know the
  entire structure of the problem to answer certain
  policy questions (studied tax changes)
• Recognize that an economic model is required to
  extrapolate from historical experience

10
More recent efforts

• Ichimura and Taber (1998,2002)
• Present general set of conditions under which
  nonparametric policy evaluation is possible
• Estimate the effects of a college tuition subsidy
  using tuition variation in the data
• Heckman (2000, 2001)
• Discusses Marschaks Maxim
• Provides some new examples where nonparametric
  assessment of new policies is feasible
• Blomquist and Newey (2002)
• Nonparametric estimation of labor supply
  responses with nonlinear budget sets.
• Bourguignon, Ferreira, and Leite (2002)
• Use reduced form random utility model for
  forecast impact of school subsidy program in
  Brasil

11
Goals of this paper

• Consider nonparametric and semiparametric methods
  for evaluating the impacts of social programs
  prior to their implementation
• Illustrate use of behavioral models in evaluating
  effects of hypothetical programs
• Show that fully nonparametric strategy sometimes
  feasible
• Suggest estimation strategy based on a modified
  version of the method of matching
• Study the performance of the methods using data
  from the PROGRESA school subsidy experiment in
  Mexico
• Compare ex ante predictions to experimentally
  estimated impacts
• Evaluate the effects of counterfactual programs
• Changes to the subsidy schedule
• Unconditional income transfer

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
Combination wage subsidy and income transfer
18
Estimation
19
School attendance subsidies when child wages are
observed
20
(No Transcript)
21
Required Assumption
22
Intent-to-treat estimator
23
Coverage Rate and Treatment-on-the-Treated
Estimator
24
Extension to multiple children, fertility assumed
to be exogenous
25
Multiple children, endogenous fertility
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
Example Only accepted child wages observed,
selection on unobservables
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
Extension to Two Period Model
35
(No Transcript)
36
Description of PROGRESA, Oportunidades(Programa
de Educacion, Salud, y Alimentacion)

• Large scale anti-poverty program
• begun in 1997
• originally provided aid to about 10 million poor
  families (40 of all rural households)
• operates in 31 states with a budget ? 1 billion
  U.S. dollars
• Recent expansion into urban areas
• Provides educational grants to parents (mothers)
  to encourage childrens school attendance.
• Must attend 85 of days
• Benefit levels increase with grade level, higher
  for girls
• Subsidies amounted to about 25 percent of average
  annual income over all children that actually
  attended in the first year of the program.

37
Experimental design and Data

• Program implemented as a randomized social
  experiment
• 506 villages randomly selected from 7 states in
  Mexico (of 31 states)
• 320 randomly assigned to the treatment group and
  186 to the control group
• Controls incorporated after third year of the
  program, but not told about the program until
  incorporated
• Use Oct. 1997 Baseline and Oct. 1998 Follow-up
  Surveys
• Data elements
• school attendance and grade attainment,
  information on employment and wages (to construct
  total family income net of child income)
• Village level data on the minimum wage paid to
  daily laborers
• Subsample
• children from program eligible families, age 12
  to 15 in 1998, who are the son or daughter of the
  household head, and for whom information is
  available in the 1997 and 1998 surveys.

38
Overview of Empirical Results

• Compare the predicted ex-ante impacts to the
  actual impacts (These are ITT impacts)
• Multiple child model
• Single child model
• Implement exact matching on age and gender
• Evaluate effects of counterfactual programs
• Doubling subsidy, cutting subsidy by 25
• Unconditional income transfer of 5000 pesos per
  year (about half of family income)

39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
Conclusions and future research

• Considered nonparametric methods for evaluating
  the impacts of social programs prior to their
  implementation.
• Behavioral models required to justify particular
  estimation strategies.
• Estimators are modified versions of matching
  estimators.
• Require stronger assumptions on unobservables
  (future research)
• In some cases, can accommodate other endogenous
  choices
• Studied performance of the ex-ante prediction
  method using data from the Mexican PROGRESA
  experiment.
• The predictions are generally of the correct sign
  and usually come within 30 of the experimental
  impact.
• Predictions more accurate for girls than for boys
• Counterfactual programs
• Changes in subsidy schedule enrollment of older
  children more elastic with respect to level of
  subsidy
• Unconditional income transfers unlikely to be
  effective