Experimental Evaluations

1
Experimental Evaluations

• Methods of Economic Investigation
• Lecture 4

2
Why are we doing this?

• Experiments are becoming more popular in
  economics
• Development Economics, Field Experiments
• Behavioral Economics, Laboratory Experiments
• Sometimes experiments dont go as planned
• Need some econometrics to prove no problems
• Need some econometrics to fix the problems
• Good baseline for understanding attribution of
  estimated differences
• Can compare other forms of evaluation to the
  experimental ideal

3
Some Basic Terminology

• Start with example where X is binary (though
  simple to generalize)
• X0 is control group
• X1 is treatment group
• Causal effect sometimes called treatment effect
• Randomization implies everyone has same
  probability of treatment
• We can change this a bit with weights

4
Why is Randomization Good?

• If X allocated at random then know that X is
  independent of all pre-treatment variables in
  whole wide world
• If you dont have a random sample, then this will
  apply to internal comparisons but will make
  external comparisons more difficult
• Implies there cannot be a problem of omitted
  variables, reverse causality etc
• On average, only reason for difference between
  treatment and control group is different receipt
  of treatment

5
Pre-treatment characteristics must be
independent of randomized treatment

• Define the joint distribution of X and W as
  f(X,W)
• Can decompose this into
• f(X,W)fXW (XW)fW(W)
• Now random assignment means
• fXW (XW)fX (X)
• This implies
• f(X,W)fX (X)fW(W)
• This implies X and W independent

6
What are we estimating

• In words We want to know if, on average, there
  is a difference in the outcome of interest
  between the control group and the treatment
  groups
• In Math we want an estimate of

7
Estimating Treatment Effects

• Take mean of outcome variable in treatment group
• Take mean of outcome variable in control group
• Take difference between the two
• This is how you learned it in theory and its
  right BUT
• Does not generalize to where X is not binary
• Does not directly compute standard errors

8
Estimating Treatment Effects A Regression
Approach

• Run regression
• yiß0ß1Xiei
• The OLS estimator of ß1 is an unbiased estimator
  of the causal effect of X on y
• To see why
• Recall that the OLS estimates E(yX)
• E(yX0) ß0 so OLS estimate of intercept is
  consistent
• estimate of E(yX0)
• E(yX1) ß0ß1 so ß1 is consistent estimate of
• E(yX1) -E(yX0)
• Hence can read off estimate of treatment effect
  from coefficient on X
• Approach easily generalizes to where X is not
  binary
• Also gives estimate of standard error

9
Computing Standard Errors

• Unless told otherwise regression package will
  compute standard errors assuming errors are
  homoskedastic i.e.
• Even if only interested in effect of treatment on
  mean X may affect other aspects of distribution
  e.g. variance
• This will cause heteroskedasticity
• This is a second order issue your coefficient
  estimates are right (consistent)
• HOWEVER you cant do any inference because your
  OLS standard errors are inconsistent

10
Robust Standard Errors

• Also called
• Huber-White standard errors
• Heteroskedastic-consistent standard errors
• Statistics Approach
• Get variance of estimate of mean of treatment and
  control group
• Sum to give estimate of variance of difference in
  means

11
A Regression-Based Approach

• Can estimate this by using sample equivalents
• Note that this is same as OLS standard errors if
  X and e are independent

12
A Regression-Based Approach

• Have to interpret residual variance differentyl
  not common to all individuals but the mean across
  individuals
• With one regressor can write robust standard
  error as
• Simple to use in practice e.g. in STATA
• . reg y x, robust

13
Summary So Far

• Econometrics very easy if all data comes from
  randomized controlled experiment and everything
  went as planned
• Just need to collect data on treatment/control
  and outcome variables
• Compare means of outcomes of treatment and
  control groups
• Everything doesnt always go as planned
• Treatment effects are small
• Randomization fails
• Non-compliance to treatment

14
How to avoid Experimental problems?

• Get info on other regressors
• Can get consistent estimate of treatment effect
  without worrying about other variables
• But there are reasons to include other
  regressors
• Improved efficiency
• Check for randomization
• Improve randomization
• Control for conditional randomization
• Heterogeneity in treatment effects

15
The Uses of Other Regressors I Improved
Efficiency

• Dont just want consistent estimate of causal
  effect also want low standard error (or high
  precision or efficiency).
• Especially important if treatment effects are
  small
• Standard formula for standard error of OLS
  estimate of ß is s2(XX)-1
• s2 comes from variance of residual in regression
  (1-R2) Var(y)

16
The asymptotic variance of ߈ is lower when W is
included

• Proof (Will only do case where X and W are
  one-dimensional)
• When W is included variance of the estimate of
  the treatment effect will by first diagonal
  element of

17
Proof (continued)

• Now
• Using trick from end of notes on causal effects
  we can write this as

18
Proof (continued)

• Inverting leads to
• By randomization X and W are independent so
• The only difference is in the error variance
  this must be smaller when W is included as R2
  rises

19
The Uses of Other Regressors II Check for
Randomization

• Randomization can go wrong
• Poor implementation of research design
• Bad luck
• If randomization done well then W should be
  independent of X this is testable
• Test for differences in W in treatment/control
  groups
• Probit model for X on W

20
The Uses of Other Regressors IIIImprove
Randomization

• Can also use W at stage of assigning treatment
• Can guarantee that in your sample X and W are
  independent instead of it being just
  probabilistic

21
The Uses of Other RegressorsAdjust for
Conditional Randomization

• Conditional randomization is where probability of
  treatment is different for people with different
  values of W, but random conditional on W
• Why have conditional randomization?
• May have no choice
• May want to do it (c.f. stratification)
• MUST include W to get consistent estimates of
  treatment effects

22
Controlling for Conditional Randomization

• What we know about our treatment
• X is by construction correlated with W
• But, conditional on W, X independent of other
  factors
• But must get functional form of relationship
  between y and W correct matching procedures
• This is not the case with (unconditional)
  randomization

23
The Uses of Other Regressors Heterogeneity in
Treatment Effects

• So far have assumed causal (treatment) effect the
  same for everyone but theres no reason to
  believe this
• May use other variables to test if treatment
  effect is different between two groups

24
Estimating Heterogeneous Treatment Effecs

• Start with case of no other regressors, suppose
  that there are different betas for everyone, so
  that
• yiß0ß1iXiei
• Random assignment implies X independent of ß1i
• Sometimes called random coefficients model

25
What treatment effect to estimate?

• Would like to estimate causal effect for everyone
  this is not possible because we only have one
  realization of the treatment effect of each
  person
• Instead, we estimate some average this is called
  the Average Treatment Effect (ATE)
• We can estimate ATE with OLS because

26
Proposition 2.5OLS estimates ATE

• Proof for single regressor

27
Observable Heterogeneity

• Full outcomes notation
• Outcome if in control group y0i?0Wiu0i
• Outcome if in treatment group y1i?1Wiu1i
• Treatment effect is (y1i-y0i) and can be written
  as
• (y1i-y0i )(?1- ?0 )Wiu1i-u0i
• Note treatment effect has observable and
  unobservable component
• Can estimate as
• Two separate equations
• One single equation

28
Combining treatment and control groups into
single regression

• We can write
• Combining outcomes equations leads to
• Regression includes W and interactions of W with
  X these are observable part of treatment effect
• Note error likely to be heteroskedastic

29
Units of Measurement

• Causal effect measured in units of experiment
  not very helpful
• Often want to convert causal effects to more
  meaningful units
• Health Example last week How much does an extra
  in income get you in better health?
• How to interpret improvements in X percent or X
  standard deviations

30
Simple estimator of this would be

• where S is the factor we change in the treatment
• Takes the treatment effect on outcome variable
  and divides by treatment effect
• Not hard to compute but how to get standard
  error?
• We can do it in a regression
• Can use your X as an instrument (more on this
  when we do IV)

31
Uses of Extra Regressors Partial Compliance

• So far
• in control group implies no treatment
• in treatment group implies get treatment
• Often things are not as clean as this
• Treatment is an opportunitynot everyone takes it
  up
• Close substitutes available to those in control
  group
• Implementation not perfect so some people get
  into or out of treatment despite RA

32
Some Terminology ITT

• Z denotes whether in control or treatment group
  Intention To Treat (ITT)
• X denotes whether actually get treatment
• With perfect compliance
• Pr(X1Z1)1
• Pr(X1Z0)0
• With imperfect compliance
• 1gtPr(X1Z1)gtPr(X1Z0)gt
• 1gtPr(X0Z0)gtPr(X0Z1)gt0

33
What Do We Want to Estimate?

• Intention-to-Treat
• ITTE(yZ1)-E(yZ0)
• This can be estimated in usual way in the
  regression
• Pros
• Dont worry about selection in compliance
• Get an average effect of your intervention
• Cons
• Dont know what the effect of the actual
  treatment iscombined effect of treatment and
  non-compliance

34
More Terminology TOT

• What if we only looked at those who took up the
  treatment then we would estimate Treatment Effect
  on Treated (TOT)
• Pros
• Looks directly at treatment
• Cons
• May be biased by the selection of individuals
  into treatment and control groups

35
Estimating TOT

• Cant use simple regression of y on Z
• But should recognize TOT as Wald estimator
• Can estimated by regressing y on X using Z as
  instrument
• Relationship between TOT and ITT

36
Next Steps

• What to do when we cant do experiments because
• They didnt work out like planned
• We couldnt do one on this particular issue
• Natural Experiments
• Use variation in the world
• Several different methods over the next few
  classes