Alternatives to Randomized Trials for Estimating Treatment Efficacy (or Harm)

1
Alternatives to Randomized Trials for Estimating
Treatment Efficacy (or Harm)

• Thomas B. Newman, MD, MPH
• Professor of Epidemiology and Biostatistics and
  Pediatrics, UCSF

AltToRcts31Oct06
2
Lecture Outline

• Prelude
• Announcements
• Double gold standard bias
• Background
• Instrumental variables and natural experiments
• Measuring additional unrelated variables to
  estimate bias
• Propensity scores
• Illustration using phototherapy for jaundice

3
Announcements

• Handouts Chapter 11, problems, this presentation
• Exam question contest questions due Thurs,
  11/9/06 by e-mail to TN
• Real examples only, recent articles strongly
  preferred
• Include answer
• We may change it
• Take-home final will be handed out 11/16,
  discussed in class 11/30

4
Double Gold Standard Bias Revisited

• Test ultrasound
• Disease Intussusception
• 2 different gold standards.
• Air contrast enema
• Observation/follow-up
• As long is there is no spontaneous resolution the
  two gold standards give the same answer
• Spontaneous resolution gives BE and F/U

5
Suppose spontaneous resolution occurs

• In q cases with U/S
• In r cases with U/S
• BE will be for q and r
• F/U will be for q and r

6
Effect of double gold-standard
7
Alternatives to Randomized Trials for Estimating
Treatment Efficacy (or Harm)

• Thomas B. Newman, MD, MPH
• Professor of Epidemiology and Biostatistics and
  Pediatrics, UCSF

AltToRcts31Oct06
8
Background

• Why do RCTs?
• Assemble comparable groups (avoid confounding)
• Allow blinding (to avoid placebo effect,
  cointerventions, and bias in measuring outcome
  variable)
• Observational studies
• May be able to assemble comparable groups or use
  statistical adjustment
• Wont be blinded

9
Why is it hard to assemble comparable groups
without randomizing?

• People who get treated differ from those who
  dont
• Important differences are with respect risk of
  the outcome
• Treated people often at higher risk (confounding
  by indication for treatment).
• Treated people may be at lower risk (selection
  bias)

10
Pre-test

• Observational studies can never establish
  causation. Proof of causation requires
  randomized trials.
• How many have heard this?
• How many agree?

11
Do you believe there is a causal relationship
between

• Acetaminophen overdose or mushroom poisoning and
  liver failure?
• Wearing glasses for refractive errors and
  improved vision?
• Infiltrate of IV calcium infusion and skin
  sloughing?
• Receipt of fluids and recovery from dehydration?
• Land mine explosions and limb injuries?

12
Post-test

• Observational studies can never establish
  causation. Proof of causality requires
  randomized trials.
• How many agree?

13
When is causal inference from observational
studies easy?

• Outcomes
• not related to indications for treatment
• rarely if ever occurs spontaneously
• highly localized in time or space
• Treatment
• well-understood biologically
• very rapidly acting

14
When its hard

• Outcomes are related to indications or selection
  for treatment, are delayed, non specific, or not
  well understood
• Learning disabilities in children treated with
  anticonvulsants
• Suicide in users of antidepressants
• Mortality after surgery for gastroesophageal
  reflux in children

15
Natural Experiments and Instrumental Variables

• Find a time or place where receipt of treatment
  was unlikely to be related to prognosis
• E.g., time-series analyses where something
  changed (e.g. new intervention became available)
• Instrumental variables (IV) measurable factors
  that influence probability of treatment that are
  not otherwise associated with outcome

16
Use of large databases

• Allows use of (weak) surrogate measures for
  actual predictor
• Biased towards null
• Achieve statistical significance with large
  sample size
• Algebraically reverse bias towards null (with
  various assumptions)

17
Delayed Effects of the Military Draft on
Mortality

• Origin of study Agent Orange concern
• Design Randomized natural experiment using the
  draft lottery
• Data source computerized death certificate
  registries, CA and PA
• Predictor variable of interest military service

Hearst N, Newman TB, Hulley SB. NEJM 1986
314620-24
18
Why not compare outcomes according to the
predictor variable of interest?

• Biased comparison those who serve in the
  military start out healthier
• Healthy warrior effect

19
Delayed Effects of the Military Draft on Mortality

• The instrumental variable measured draft lottery
  number below cutoff (based on date of birth)
• IV associated with predictor variable of
  interest, not independently associated with
  outcome

20
BUT Having an eligible number was a poor measure
of military service
21
Results
22
RCT as an Instrumental Variable Health effects
of exclusive breast feeding

• Cant do RCT of exclusive breast-feeding
• Can do RCT of breast-feeding PROMOTION
• Assignment to BF promotion group should be
  associated with exclusive breast feeding, but not
  independently associated with outcome
• Need very large sample size
• Algebraic correction

23
Promotion of Breastfeeding Intervention Trial
(PROBIT)

• Cluster-randomized trial at 31 sites in Belarus
• Subjects 17,046 term singleton infants gt2500g
  initially breastfed
• Intervention WHO/UNICEF Baby Friendly Hospital
  Initiative
• Outcomes BF _at_ 3,6,9,12 months and allergic,
  gastrointestinal and respiratory disease
• F/U to 12 months on 16,491 (96.7)

Kramer MS, et al. JAMA 2001285413-20.
24
PROBIT, RQ 1

• Does a Baby Friendly Hospital increase
  exclusive breastfeeding?
• Predictor Group assignment
• Outcome Exclusive breast feeding
• Intention-to-treat (ITT) analysis is fine
• Exclusive BF at 3 months (rounded) 40 vs 5 P lt
  0.001

25
Probit RQ2

• Does exclusive breastfeeding reduce the risk of
  eczema in the infant?
• If the only effect of intervention related to
  eczema is increasing exclusive BF, then
• Predictor Group assignment
• Outcome Eczema
• ITT analysis biased towards null informative if
  study positive
• Eczema 3.3 vs 6.3 adjusted OR 0.54 (95 CI
  0.31-.95 based on GLIMMIX P 0.03)

26
PROBIT, RQ3

• How much does exclusive breastfeeding reduce the
  risk of eczema in the infant? (What is the
  NNEBF? )
• Predictor Group assignment
• Outcome Eczema
• ITT wont work -- too much misclassification.
  (Gives the number needed to be exposed to the
  intervention, not the NNEBF.)

Number Needed Exclusively to Breast Feed
27
Algebraic correction

• If all of the difference in eczema is due to the
  difference in exclusive breast feeding, it can be
  shown that the ARR is

28
NNEBF and caveat

• Since ARR 8.6, NNEBF to prevent 1 case of
  eczema is about 1/.086 12
• Caveats
• Results are for the effect of breastfeeding in
  response to the intervention
• Assumes the only effect of the Baby Friendly
  Hospital is via difference in exclusive
  breastfeeding
• Similarly, effects of draft lottery only apply
  to those who served as a result of the lottery.

29
Summary/other examples

• If variables known NOT to be associated with
  outcome are associated with treatment of
  interest, consider this approach.
• Generalizes to manynatural experiments.
• E.g., an intervention is intermittently
  available, or only available to certain groups.
  -- different outcome by day of the week, etc.

30
More natural experiments

• Costs of discontinuity of care increased
  laboratory test ordering in patients transferred
  to a different team the next morning
• Effect of ER Copay rate of appendicitis
  perforation unchanged after increase in co-pay.
• Aircraft cabin air recirculation and symptoms of
  the common cold no difference by type of air
  recirculation in aircraft

Lofgren, RO. J Gen Intern Med. 19905501-5
Hsu J, et al. Presented at Bay Area Clinical
Research Symposium 10/17/03 Zitter JN et al.
JAMA 2002288483-6
31
Unrelated variables to estimate bias or
confounding

• Measure an outcome that WOULD be affected by
  bias, but not by intervention (and see if it is)
• Measure a predictor that WOULD cause the same
  bias as the predictor of interest (and see if it
  does)

32
Observational study of screening sigmoidoscopy

• Possible bias patients who agree to
  sigmoidoscopy are likely to be different
• Solution measure an outcome that would be
  similarly affected by bias
• Results
• Decreased deaths from cancers within the reach of
  the sigmoidoscope (OR 0.41)
• No effect on deaths from more proximal cancers
  (OR 0.96).

Selby et al, NEJM 1992326653-7
33
Effect of British breathalyser crackdown

• Abrupt drop in accidents occurring during weekend
  nights (when pubs are open)
• Measure an outcome that would be affected by
  bias accidents during other times
• Result No change in accidents occurring during
  other hours

See Cook and Campbell Quasi-Experimentation.Bosto
nHoughton Mifflin, p. 219
34
Calcium Channel Blockers (CCB) and AMI

• Population based case-control study at Group
  Health
• Progressive increase in risk of AMI with higher
  doses of CCB (P lt0.01)
• Concern confounding by indication
• Measure a predictor that would cause same bias
  beta-blockers
• Result progressive decrease in risk associated
  with higher doses of beta-blockers (P 0.04)

Psaty et al., JAMA 1995274620-25
35
Suicide Risk in Bipolar Disorder During Treatment
With Lithium and Divalproex

• Retrospective cohort study of Kaiser Permanente
  and Group Health patients with bipolar disorder
• Compared with no treatment, patients treated with
  Valproex at 2.1 times suicide risk
• Concern confounding by indication
• Results Suicides per 1000 person/years
• 31.3 for treatment with divalproex
• 15 for no treatment (Plt0.001)
• 10.8 for Lithium (Plt0.001)
• If confounding by indication, expect same bias
  for Lithium

Goodwin et al. JAMA. 20032901467-1473
36
Initial Mood Stabilizer Prescription by Year of
Initial Diagnosis
Goodwin et al. JAMA. 20032901467-1473
37
Estimating biases Cautionary Tale

• Nurses Health Study
• Vitamin E assoc. with decreased risk of CHD (RR
  .6)
• No significant effect of multiple vitamins
• Health Professionals Study
• Vitamin E assoc. with decreased risk of CHD (RR
  .6)
• No significant effect of Vitamin C
• TN began taking Vitamin E

N Engl J Med. 19933281444-9 and 1450-6
38
Meta-analysis high-dosage vitamin E
supplementation may increase all-cause mortality
Miller ER et al. Ann Intern Med. 2005 Jan
4142(1)37-46
39
Propensity Scores -1

• Big picture want to know if association between
  treatment and outcome is CAUSAL
• Recall competing explanation confounding by
  indication for treatment
• Factor must be associated with outcome
• Factor must be associated with treatment
• Traditional approach adjust for factors
  associated with outcome

40
Propensity Scores -2

• Alternative approach Create a new variable,
  propensity to be treated with the intervention
• Then match, stratify, or include it in
  multivariable analyses
• Advantages
• Better power to control for covariables (because
  receipt of the intervention may be much more
  common than occurrence of the outcome)
• You can more easily tell when treated and
  untreated groups are not comparable

41
How Much Overlap In The Propensity Scores Do We
Want?
A
B
42
Example Aspirin use and all-cause mortality
among patients being evaluated for known or
suspected Coronary Artery Disease

• RQ Does aspirin reduce all-cause mortality in
  patients with coronary disease
• Design Cohort study
• Subjects 6174 consecutive patients getting
  stress echocardiograms
• Predictor Aspirin use
• Outcome All-cause mortality
• Crude result 4.5 mortality in each group

Gum PA et al. JAMA 2001 286 1187-94
43
Analysis using Propensity Scores

• Two multivariable analyses
• Predictors of aspirin use
• Predictors of death
• Predictors of ASA use turned into a propensity
  score
• Users and non-users of ASA matched on ASA
  propensity score
• Compare mortality in matched groups
• (Unmatched patients cannot be analyzed)

44
Survival in Propensity-Matched Patients
Recall total N6174
45
Limitations

• Can only compare subjects whose propensity scores
  overlap
• Can only generalize to subjects who could have
  received either treatment
• Limitations similar to exclusions from clinical
  trials
• Important variables may be missing from your model

46
Illustration Phototherapy for Neonatal Jaundice

• RQ How effective is phototherapy in babies with
  Total Serum Bilirubin levels of 20-22.9 mg/dL?
• Subjects Newborns at NC-KPMCP 2000 g, 34
  wks with TSB 20-22.9 mg/dL at 48 hr (N1777)
• Intervention Phototherapy within 8 hours of TSB
  20-22.9 mg/dL (N635, 36)
• Outcome TSB 25 mg/dL (N21, 1.2)

47
Logistic regression

• Phototherapy only OR0.30 (P .05)
• Phototherapy gest age OR 0.28 (P0.04)
• Phototherapy gest age rate of riseOR 0.12
  (P.002)

48
Propensity analysis

• Step 1 predictors of PT within 8 hr oif TSB
  20-22.9
• Rate of rise of TSB, gestational age, race, sex,
  maternal age, hospital of birth, etc.
• Generate new variable, propensityPT predicted
  probability of PT

49
Propensity by whether PT received
50
Logistic Regression With Propensity Score

• Phototherapy only OR0.30 (P .05)
• Phototherapy gest age OR 0.28 (P0.04)
• Phototherapy gest age rate of riseOR 0.12
  (P.002)
• Logistic Phototherapy propensity scoreOR
  0.13 (P.002)
• Mantel-Haenszel stratified analysis PT and
  propensity score in 4 strata OR 0.14, P0002

51
Efficacy of Phototherapy (PT) for Neonatal
Jaundice

• Large interfacility practice variation in use of
  phototherapy in the NC KPMCP
• Hospital of birth thus an IV for phototherapy use
• We can use individual-level data to adjust for
  other risk factors for TSB 20 mg/dL

52
Instrumental variable

• N too small for IV for TSB 25 mg/dL, so predict
  TSB 20 mg/dL
• For each hospital, calculate the proportion of
  newborns in group C (AAP consider PT group) who
  received phototherapy
• Use this proportion as a predictor of TSB 20
  mg/dL in individual level analyses

53
Group R AAP RECOMMENDS phototherapy Group C
AAP says CONSIDER phototherapy
Atkinson L, Escobar G, Takayama J, Newman TB.
Pediatrics 2003111e555-61
54
Rate of hyperbilirubinemia by PT use in 11
hospitals, 1995-6
55
IV analysis to Predict TSB 20 mg/dL
56
Summary

• RCTs are most definitive, but
• Not always feasible or necessary
• Look for opportunities to answer questions with
  observational studies
• Instrumental variables
• Natural experiments
• Propensity scores
• Estimating biases