Why be cautious about acting on evidence from nonrandomised studies

1
Why be cautious?about acting on evidence from
non-randomised studies
Barnaby REEVES Royal College of Surgeons London
School of Hygiene Tropical Medicine
2
1. The dilemma 2. Differences between RCTs and
non-randomised studies (NRS) 3. Some of the
evidence
3
The dilemma

• There are many health care interventions for
  which there is little or no evidence from
  randomised controlled trials (RCTs)
• RCTs of many of these interventions are unlikely
  ever to be carried out Black, 1996
• Health care decision-makers need to act both when
  there is and when there is not high quality
  evidence from RCTs

4
Kinds of interventions

• Aspects of organisation of care, e.g. specialist
  (high volume) vs. generalist (low volume)
• Interventions delivered to populations rather
  than individuals, e.g. health promotion,
  immunisation
• Interventions that are dependent on the skills of
  the practitioner or which are difficult to
  standardise for other reasons, e.g. surgery

5
Observational methods provide no useful means of
assessing the value of a therapy Doll R. Ann
N Y Acad Sci 1993 703 313 .... they
non-randomised designs cannot discriminate
reliably between moderate differences and
negligible differences in outcome, and the
mistaken clinical conclusions that they engender
could well result in the under-treatment,
over-treatment or other mistreatment of millions
of future patients Peto R et al. J Clin
Epidemiol 19954823-40
6
Differences between RCTs non-randomised studies

• Random allocation to alternative treatments -
  control for confounding
• Better opportunity to control for biases?
• Explanatory (efficacy) vs. pragmatic
  (effectiveness) results, e.g. intention-to-treat
  analyses
• Accounting for all eligible patients
• CONSORT statement - JAMA 1996

7
Randomised controlled trial
new treatment
group 1
Outcome
population
Outcome
group 2
control treatment
8
Confounding
Process / Measures health care
of factors outcome Age (sex) Disease
severity Comorbidity
9
Confounding in poor quality RCTs

• Moher and Jadad (Lancet 1999)
• RCTs with inadequate concealment of allocation
  led to 37 increased estimate of benefit cf. well
  concealed RCTs
• Schulz et al. (JAMA 1995)
• RCTs with inadequate or unclear concealment of
  allocation led to 41 and 33 increased estimate
  of benefit respectively cf. well concealed RCTs

10
TURP vs open prostatectomy
TURP vs open relative risk Study
n Unadjusted Adjusted (95 CI) Manitoba (1)
1650 1.45 1.45 (1.15-1.84) Denmark
38067 1.24 1.19 (1.15-1.24) Manitoba
(2) 485 1.58 1.59 (1.06-2.37)
Concato et al. JAMA 1992
11
TURP vs open prostatectomy
Comorbidity Mortality TURP vs
open measure rate Adjusted
RR (95 CI) Discharge 0 14 1.20 (0.63 -
2.26) diagnosis gt1 27 Case record lt2 12 1.12
(0.57 - 2.10) review gt2 34 Case record 0 11
0.97 (0.51 - 1.86) review 1 16 2 29
3 45
Concato et al. JAMA 1992
12
Non-randomised studies

• Susceptibility to confounding
• Time series
• Controlled before-after studies
• Prospective cohort studies
• Retrospective cohort studies
• Historically controlled cohort studies
• Case control studies

13
Limitations of NRS

• Residual confounding - not a question of whether
  it is present but to what extent
• Need to apply the same principles to NRS as RCTs
  when carrying out studies, e.g.
  intention-to-treat, timing of invervention

14
Obtaining evidence that NRS are valid

• Comparisons between results of RCTs and NRS
• But Catch 22? Sometimes expect NRS to give a
  different result because of limitations of RCTs,
  e.g. lack of applicability
• Beware meta-confounding .

15
Meta-confounding
Study design, Size of i.e. RCT /
NRS effect Date of study Study
populations Publication bias
16
Comparisons of RCTs non-randomised studies
Review of reviews NRS gave No difference RCT
gave larger effect larger effect 5
1 2 Kunz and Oxman. BMJ 1998
17
Comparisons of RCTs non-randomised studies
Review of reviews and paired studies NRS
gave No difference RCT gave larger
effect larger effect (1) 8 7
2 (2) 15 1
19 (1) Britton et al. BMJ 1998 (2) Reeves
et al. 1998
18
Comparisons of RCTs non-randomised studies

• Case studies
• ? carotene and cardiovascular mortality. Egger et
  al. BMJ 1998
• HRT and cardiac disease. Colditz Stampfer
  1991. Hulley et al. JAMA 1998.
• Mammographic screening and mortality from breast
  cancer. Reeves et al. 1998
• Folic acid supplementation and mortality from
  breast cancer. Reeves et al. 1998

19
Validity of comparisons in different
circumstances?

• RCTs feasible
• RCTs difficult
• RCTs impossible

Same research question but NRS low
quality? Different research questions? Circumsta
nces especially prone to bias / confounding?
20
Conclusions
1. If NRS are included in a review, do NOT pool
RCT and NRS estimates of effect. 2. If pooled
estimates of effect from NRS and RCTs are similar
..? 3. If pooled estimates of effect from NRS
and RCTs are different ..? 4. If only NRS are
available, apply the same principles as when
appraising primary NRS consider the size of the
effect, NOT its precision.