Using A Regression Discontinuity Design (RDD) to Measure Educational Effectiveness:

1
Using A Regression Discontinuity Design (RDD) to
Measure Educational Effectiveness

• Howard S. Bloom
• MDRC
• 12-11-02
• Howard.bloom_at_mdrc.org

2
This Talk Will

• introduce the history and logic of RDD,
• consider conditions for its internal validity,
• considers its sample size requirements,
• consider its dependence on functional form,
• illustrate some specification tests for it,
• consider limits to its external validity,
• consider how to deal with noncompliance,
• describe an application.

3
RDD History

• In the beginning there was Thislethwaite and
  Campbell (1960)
• This was followed by a flurry of applications to
  Title I (Trochim, 1984)
• Only a few economists were involved initially
  (Goldberger, 1972)
• Then RDD went into hibernation
• It recently experienced a renaissance among
  economists (e.g. Hahn, Todd and van der Klaauw,
  2001 Jacob and Lefgren, 2002)
• Tom Cook has written about this story

4
RDD Logic

• Selection on an observable (a rating)
• A tie-breaking experiment
• Modeling close to the cut-point
• Modeling the full distribution of ratings

5
(No Transcript)
6
RDD As A Linear Regression
7
(No Transcript)
8
Conditions for Internal Validity

• The outcome-by-rating regression is a continuous
  function (absent treatment).
• The cut-point is determined independently of
  knowledge about ratings.
• Ratings are determined independently of knowledge
  about the cut-point.
• The functional form of the outcome-by-rating
  regression is specified properly.

9
RDD Statistical Model
where Yi outcome for subject i,


Ti
one for subjects in the treatment group
and zero otherwise, Ri rating for subject
i, ei random error term for subject i, which
is
independently and identically distributed
10
Variance of the Impact Estimator

• s2 variance of mean outcomes across subjects in
  the
• treatment group or comparison group
• R12 square of the correlation between outcomes
  and
• ratings within the treatment and
  comparison group
• R22 square of the correlation between treatment
  status and
• the rating
• P proportion of subjects in the treatment
  group,
• N total number of subjects

11
Sample Size Implications

• Because of the substantial multi-collinearity
  that exists between its rating variable and
  treatment indicator, an RDD requires 3 to 4 times
  as many sample members as a corresponding
  randomized experiment

12
Specification Tests

• Using the RDD to compare baseline characteristics
  of the treatment and comparison groups
• Re-estimating impacts and sequentially deleting
  subjects with the highest and lowest ratings
• Re-estimating impacts and adding
• a treatment status/rating interaction
• a quadratic rating term
• interacting the quadratic with treatment status
• Using non-parametric estimation

13
External Validity

• Estimating impacts at the cut-point
• Extrapolating impacts beyond the cut-point with a
  simple linear model
• Estimating varying impacts beyond the cut-point
  with more complex functional forms

14
Dealing With Noncompliance

• Sharp and fuzzy RDDs
• No-shows and crossovers
• The effect of intent to treat (ITT)
• The local average treatment effect (LATE)
• The effect of treatment on the treated (TOT)
• Where rT and rC the proportion of the treatment
  and control groups receiving treatment,
  respectively

15
Application of RDD To Reading First

• Reading First (RF) is a cornerstone of No Child
  Left Behind
• RF resources are allocated purposefully to
  schools that need it most and will benefit most
• Some districts allocated RF resources based on
  quantitative indicators
• We chose a sample of 251 schools near the
  cut-points for 17 such districts and 1 state

16
References

• Cook, T. D. (in press) Waiting for Life to
  Arrive A History of the Regression-discontinuity
  Design in Psychology, Statistics and Economics
  Journal of Econometrics.
• Goldberger, A. S. (1972) Selection Bias in
  Evaluating Treatment Effects Some Formal
  Illustrations (Discussion Paper 129-72, Madison
  WI University of Wisconsin, Institute for
  Research on Poverty, June).
• Hahn, H., P. Todd and W. van der Klaauw (2001)
  Identification and Estimation of Treatment
  Effects with a Regression-Discontinuity Design
  Econometrica, 69(3) 201 209.
• Jacob, B. and L. Lefgren (2004) Remedial
  Education and Student Achievement A
  Regression-Discontinuity Analysis Review of
  Economics and Statistics, LXXXVI.1 226 -244.
• Thistlethwaite, D. L. and D. T. Campbell (1960)
  Regression Discontinuity Analysis An
  Alternative to the Ex Post Facto Experiment
  Journal of Educational Psychology, 51(6) 309
  317.
• Trochim, W. M. K. (1984) Research Designs for
  Program Evaluation The Regression-Discontinuity
  Approach (Newbury Park, CA Sage Publications).