Causality%20and%20Randomized%20Control%20Trials

1
Causality and Randomized Control Trials
2
Empirical Research

• Three broad types of Empirical papers
• Paper Type I - Descriptive
• CVD mortality over time
• Regional differences in medical care
• How are health insurance premiums changing over
  time?
• These papers generally DONT TRY AND SAY WHY the
  trend might be changing over time
• Although there is likely to be some speculation

3
Empirical Research (cont.) -

• Paper Type II Relate variable X to variable Y
• Effect of Price on the quantity of Medical Care
• Effect of race on Income/Health
• Effect of hypertension on risk of CVD
• These papers are making a causal argument
• The strength of which is up to the reader to
  evaluate

4
Empirical Research (cont.) -

• Paper Type III
• Use estimates from the first two types of papers
  to make policy recommendations
• For ex. Some studies find that insurance
  generosity affects the use of IVF services
• Because of limited opportunities, individuals
  maximize the chance of having at least one child

5
Empirical Research (cont.)-

• One unintended consequence of this is multiple
  births
• Multiple births result in higher costs and lower
  infant health
• Using estimates from the IVF papers, someone else
  might write a paper about the optimal level of
  insurance benefit

6
Policy Relevance

• We are going to focus on the Second Paper Type
• All three types of papers influence policy
• But paper type II is generally of most interest
  to policy researchers because it provides
  magnitudes for the phenomena of interest
• Magnitudes aid policy makers in the decision to
  allocate resources

7
Causation

• What do we mean by causation?
• We are asking a WHAT IF question
• What if instead of X happening, Z happened. How
  would that change the outcome of Interest?
• Thus one must always state the alternative
• The what if scenario is also called a
  COUNTERFACTUAL

8
Some Notation

• Following Folland (1986)
• Some units U-where U can be a person, city,
  school
• Assume for simplicity two treatments T and C
• T-Treatment and C-Control
• Treatment can be a variety of things Drug,
  education, income, textbooks, co-pays
• Y represents outcome from receiving treatment
• So YT(u) And YC(u)

9
Fundamental Problem of Causation

• CANNOT observe the effect of treatment and
  control for the same person
• Unless Temporal Stability AND Causal Transience
  are observed
• Temporal Stability (TS) -Effect of T on U is same
  now and the future
• Causal Transience (CT) Effect of T on U doesnt
  change once U is exposed to T
• Or Unit homogeneity is observed
• Yt(U1)Yt(U2) and Yc(U1)Yc(U2)

10
Fundamental Problem of Causation (cont.) -

• Because of Temporal Stability and Causal
  Transience we can only estimate average treatment
  effects
• Average treatment effect equals
• E(Yt(U)) E(Yc(U))
• This is simply the mean difference of the outcome
  across the treatment and control groups

11
Paper Type II-Causality

• Observational Studies Most are cross-sectional
• Some type of statistical procedure that relates
  variables X and Y
• Ordinary Least Squares, Logistic Regression,
• Propensity Scores
• Quasi Experimental/Natural Experiments
• Regression Discontinuity
• Difference in Difference
• Instrumental Variables
• Randomized Control Trial (RCT) Gold Standard

12
Observational Studies I

• Difficult to show causation purely from
  observational data, why?
• An example Researchers are interested in
  whether income is related to health
• Direct effects Can buy more medical care
• Indirect effects Able to afford health
  insurance
• Some researchers believe health insurance affects
  health

13
Observational Studies I (cont.)

• Money can affect level of education
• Education might help you get better information
• Education might help you process information
  faster

14
Observational Studies I (cont.)-

• Take data from the cross section (point in time)
• Self-reported health as the dependent variable
  and Income as the independent variable
• Also adjust for a variables such as education,
  insurance, geography, age, sex, race, income,
  family education etc. and identify an effect
• Can we say this is the true effect of income on
  health?

15
Observational Studies II -

• Magnitudes from observational studies are
  generally biased upwards - Especially from
  cross-sectional studies
• There are some examples where estimates from
  observational studies are biased downward
• These are rare cases in the universe of all
  published studies
• Can you think of an association that is biased
  downward?
• I.e. An RCT would increase the size of your
  coefficient

16
Observational Studies II (cont.) -

• In some studies the bias is hard to sign
• For example a researcher is interested in whether
  having fire insurance leads to more fire
  accidents relative to not having fire insurance.
• What is the IDEA?
• Fire insurance lowers the cost of having your
  place burn down
• Thus individuals have less of an incentive to be
  careful, which in turn increases probability of a
  fire (also called Ex-Ante Moral Hazard)

17
Observational Studies II (cont.) -

• Look at the Correlation between purchasing
  insurance and Having a fire in the next 5 years?
• In observational data-Individuals for whom fire
  insurance is more valuable (more likely to have a
  fire) will be more likely to buy fire insurance,
  How does this affect the coefficient?
• Not adjusting for this biases the coefficient
  upwards
• In observational data individuals who are more
  cautious might also be more likely to buy fire
  insurance.
• Cautious people might have fewer fires than risky
  people
• Not adjusting for this will bias the coefficient
  downward

18
Observational Studies II (cont.) -

• Conclusion A-priori impossible to tell whether
  relationship obtained from observational data is
  above or below the true effect of having fire
  insurance on having a fire.

19
Observational Studies III

• Given the above examples, Observational studies
  primarily show associations
• We will talk more about research designs with
  observational data that get us closer to
  causality
• Why is it important to show that something is
  truly causal and not just an association?

20
Randomized Control Trial

• Randomization is a process used to assign a
  treatment to either treatment or control
• Randomization guarantees independence between
  treatment and all the other variables that might
  affect outcomes of interest
• A simple procedure for randomization coin
  flipping
• If randomizations is done correctly the mean
  difference across treatment and control groups
  E(Yt(U)) E(Yc(U)) is said to be unbiased
• How can we test whether randomization worked?

21
RCT (cont.)-

• Without randomization it is very difficult to
  guarantee that it is truly the treatment that is
  responsible for the outcome
• Most non-experimental procedures are aimed at
  finding a control group that is similar to the
  treatment group

22
RCT (cont.) -

• If its such a good idea why arent there more
  RCTs?
• Ethical Problems
• Smoking is a good example
• Costs
• RCTs cost a lot of money
• The Rand HI experiment cost 280 Million 2004
  dollars
• This was to randomize 7,791 people and to follow
  them for approximately 8 years.

23
RCT (cont.) -

• Costs also impact the duration of the experiment
  Rand Health Insurance experiment only ran for 8
  years
• Attrition can be high
• This is also a problem with non-experimental
  designs
• Importantly people who drop out of the experiment
  are likely different from people who stay in the
  experiment
• Treatment effects could be different for the two
  groups

24
RCT (cont.) -

• i.e. Conjecture that treatment effect is higher
  for the group that stays in the experiment
• If you only used people who stayed in the
  experiment there would again be a upward bias to
  the measured treatment effect.
• Even though there is attrition, one strategy is
  to estimate the effect as if there was no
  attrition.

25
RCT (cont.) -

• Keep everyone in the sample even if some people
  are not longer taking the treatment
• This is called intent to treat analysis
• Intent to treat will dilute the true effect since
  not all individuals in treatment are taking the
  drug
• But this preserves the experiment and any
  estimates are still valid
• In a later lecture we will consider another
  solution to the attrition problem

26
RCT (cont.) -

• Treatment becomes Controls
• Different from Attrition
• Difficult to generalize from location to location
• Will experiment in location A reveal the same
  effect if done in location B
• Hawthorne Effects
• Observation makes people behave differently
• Thus results might not apply to non-observed
  setting

27
RCT (cont.) -

• Finally Some things are not easily Manipulated
• How does one randomize Sex?
• How about race?
• Lets come back to this

28
In Depth Example-Discrimination

• What is the effect of Sex (Race) on Income?
• Many studies show differences across the groups
  on a variety of outcomes
• For ex. Some studies report that a woman makes
  .80 cents for each dollar a man makes

29
What Does Theory Say?

• Two Theories
• Statistical Discrimination
• Employers have limited resources to get
  information about any single individual, but know
  something about group averages
• They use information on the group average to make
  an inference about a specific individual

30
What Does Theory Say (cont.) -

• Wide applicability Physician decision making,
  Product selection, Speeding tickets - This is
  Profiling
• Taste-Based Discrimination
• Employers do not like to employ individuals from
  a specific group

31
What Does Theory Say (cont.)-

• Two types of discrimination have very different
  policy implication
• In a competitive market firm will bear the cost
  of taste-based discrimination
• Statistical discrimination will likely never be
  competed away
• Why?
• Because using information about the group solves
  a problem that the profiler faces

32
Testing for Discrimination I

• How do we test whether there is discrimination
  and second if so what type of discrimination?
• One idea is to simply compare mean wages across
  different groups from real world data
• What are the problems with this method?
• Employer observes something that you as a
  researcher do not (experience, good looks)
• Cannot separate out two theories with this method

33
Testing for Discrimination I (cont.)

• Lets take a step back
• How would one design an experiment to determine
  whether there is discrimination?
• In the RCT framework this question amounts to,
  How does one randomize race?
• Seems impossible to do
• Falls into one of these characteristics that
  cannot be manipulated

34
Testing for Discrimination II

• Audit Studies Send in hispanics, african
  americans and whites for job interviews
• Two Problems
• Auditors are matched on some observables except
  race
• height, weight, age, dialect, dressing style and
  hairdo, Is that enough?
• Study is not Double blind This can effect
  treatment effects

35
Testing for Discrimination III

• Hard to manipulate race in life, but EASY to
  manipulate race on paper
• Which name doesnt belong?
• Chow Yun Phat, Pete Sampras, Srikanth Kadiyala
• Correct answer is clearly Srikanth because he is
  not rich and famous
• Racial groups can have very different sounding
  names

36
Testing for Discrimination III (cont.)

• Manipulate the resume so only difference is a
  Black sounding name vs. a White Sounding name
• Emily Walsh vs. Jamal Jones
• Greg Baker vs. Lakisha Washington
• Find some real Employers from the newspapers
• Two markets Chicago/Boston
• 1300 Ads

37
Testing for Discrimination III (cont.)

• They vary not only the name (two resumes) but
  also type of resume
• More experience and Skills vs. Less experience
  and Skills
• Typically 4 different types of resumes to each
  job advertisement
• Measure Call Back Rate
• Researchers set up fake tel. s to receive call
  backs

38
Testing for Discrimination III (cont.)-

• Results
• African Americans need to send 15 resumes to get
  1 call back
• Whites need to send 10 resumes to get 1 call back
• 50 gap in call back
• Whites with high quality resume receive nearly
  30 more callbacks vs. whites with low quality
  resumes
• Blacks with high quality resumes dont experience
  the same benefit
• Amazing fact, experience and some other skills
  not being rewarded in the marketplace for blacks

39
Separating Theories

• Does this method separate Statistical from Taste
  Based Discrimination?
• YES, Why?
• This study is superior to Audit studies, why?
• Perfect Matching on Treatment and Control
• Unlike audit studies no bias from either
  participant or researchers
• This study has quite a few positives in the Realm
  of RCTs, What are they?
• No attrition!
• No mixing of treatment and control!
• Cheap!

40
Some Common Non-Experimental Designs

• Designs without control groups
• X 01 - Observe only data from post treatment (X)
  treatment
• 01 X 02 Observe data from pre and post
  treatment period
• 01 02 X 03 Observe data from pre and post
  observe a longer pre period

41
Some Common Problems with Non-Experimental Design

• Ambiguous Temporal Precedence
• For cross-sectional data
• History- Events occurring concurrently with
  intervention affect results
• Maturation Naturally occurring changes over
  time confused with intervention
• Regression to the mean

42
Some Common Non-Experimental Designs

• Designs without control groups
• X 01
• No control group
• Causality impossible to show
• 01 X 02
• No true control group, pre-period is used as one
• History, maturation are problems
• Regression to the mean is also a problem
• Most Important thing to remember Treatment
  timing might not be random

43
Some Common Non-Experimental Designs (cont.) -

• 01 02 X 03
• No true control group,
• History, maturation are problems
• Arguments can be made against regression to the
  mean since you have longer time period
• Most important thing to remember Treatment Timing
  might not be random

44
Cites

• Free For All? Lessons from the Rand Health
  Insurance Experiment, Joe Newhouse
• Statistics and Causal Inference, Journal of
  American Statistical Association, Vol. 81, no.
  396, Dec. 1986, pp 945-960, Paul Holland
• Are Emily and Greg More Employable than Lakisha
  and Jamal? A Field Experiment on Labor Market
  Discrimination, American Economic Review, Vol.
  94, no. 4, Sept. 2004, pp. 991-1013, Marianne
  Bertrand, Sendhil Mullainathan