Professor Peter Sandercock,

1
Part 3. Subgroup analysis pitfalls and perils

• Professor Peter Sandercock,
• University of Edinburgh
• Co-ordinating Editor Cochrane Stroke Group

Evidence-based Neurology in Europe. Perugia
December 2006
2
Why do subgroup analysis?
To explore differences in treatment effect
between various groups of patients who might be
expected to respond to treatment differently
3
Subgroups warnings

• Subgroup analyses can pose serious multiplicity
  concerns. By testing enough subgroups, a
  false-positive result will probably emerge by
  chance alone.
• Investigators might undertake many analyses but
  only report the significant effects, distorting
  the medical literature.
• In general, we discourage subgroup analyses.
  However, if they are necessary, researchers
  should do statistical tests of interaction,
  rather than analyse every separate subgroup.

Schulz, Lancet 2005 365 165761
4
Inappropriate subgroup analysis can kill
5
Canadian Co-op Study Group 1978relative odds of
stroke or death in 585 TIA/stroke patients
treated longterm with aspirin vs no aspirin
Males Females Both
Odds ratio 0.7 (95 CI 0.5 to 1.0)
0
0.5
1
2
3
Aspirin better Aspirin worse
Gent. Circulation 1980
6
Impact of this result

• FDA did not licence aspirin for stroke prevention
  in women
• Millions of women were denied effective therapy
• Many avoidable strokes and deaths from vascular
  disease occurred

7
the question is NOT Is the treatment effect in
this subgroup statistically significantly
different from zero?BUTare there any
differences in the treatment effect between the
various subgroups? The correct statistical
procedures are either a test of heterogeneity or
a test for interaction
8
Canadian Co-op Study Group 1978relative odds of
stroke or death in 585 TIA/stroke patients
treated longterm with aspirin vs no aspirin
Males Females Both
Odds ratio 0.7 (95 CI 0.5 to 1.0)
0
0.5
1
2
3
Aspirin better Aspirin worse
9
The Antiplatelet Trialists Collaboration
analysis of individual patient data to examine
the effects of antiplatelet drugs vs control on
vascular events in subgroups of high vascular
risk patients,
10
Conclusion

• Should have concluded no evidence of a
  difference in effect between males and females
• Systematic review of all of the evidence
  confirmed no evidence of a difference in aspirin
  effect between men and women

11
How many subgroups should you look at?

• As few as possible
• The selected few you specified before you
  analysed the data (pre-specified)

12
ISIS-2 aspirin vs control - effects on vascular
death in 17,187 patients with acute myocardial
infarction (MI)
Relative risk Aspirin
Control reduction Overall trial result
9.4 11.8 20 P 13
When this paper was submitted to the Lancet, the
editors urged the researchers to include nearly
40 subgroup analyses. The investigators
reluctantly agreed, under the condition that they
could provide a subgroup analysis of their own to
illustrate their unreliability.
14
(No Transcript)
15
(No Transcript)
16
Authors conclusions
Apparent harm in patients born under star sign of
libra or gemini, with prior MI and diabetics, all
most likely due to the play of chance All these
subgroup analyses should, perhaps, be taken less
as evidence about who benefits, than as evidence
that such analyses are potentially misleading.
17
Subgroup analysis in trials with no evidence of
an overall effect

• Be even more careful!

18
Trial of new versus standard antibiotic on
febrile morbidity
19
Most often the best estimate of the treatment
effect in a subgroup of patients is not the
actual effect in that subgroup but the overall
effect in the whole trial. The treatment effect
in a subgroup is rather likely to be imprecise
and could even be completely wrong.
20
Examples from stroke where this advice has not
been followed

• Nimodipine. Meta-analysis. Overall no effect,
  but benefit 12hrs. Further
  trial of nimodipine this.
• Clomethizole in acute stroke overall no benefit,
  but apparent benefit in large artery stroke -
  CLASS2 trial. CLASS2 Did not confirm this

21

• The answer to a randomized controlled trial
  that does not confirm ones beliefs is NOT the
  conduct of several sub-analyses until one can see
  what one believes. Rather, the answer is to
  re-examine ones beliefs carefully.

Oie, SG. BMJ 1999 318 100809.
22
Subgroups in systematic reviews

• It is important to put the results of single
  trials in the context of a systematic review
• A single trial is a subgroup of a meta-analysis

23
Dont believe the results of single small
positive trials the lesson from the 312 patient
Fraxiparine in Ischaemic Stroke Study (FISS)

• FISS 48 reduction in odds of death or
  dependency with fraxiparine
• not confirmed by the 760 patient FISS-bis trial
  (non significant 6 increase in odds)

24
Which is wrong? FISS or FISS-bis? It helps to
see the BIG picture

• Cochrane systematic review of ALL
• relevant trials
• Data on death or dependency available
• from 5 completed trials of anticoagulant vs
• control in acute stroke
• 21,800 patients randomised

25
No effect of heparin overall or in subgroups
unfractionated, LMW heparin or heparinoid
FISS result therefore probably just due to the
play of chance
26
Subgroup analysis is.

• highly unreliable if
• no overall treatment effect
• only one amongst many subgroups appears different
  from the rest
• the overall sample size is small
• more reliable if
• there is an overall treatment effect which is
  several standard deviations from the null point
  i.e. highly statistically significant
• the subgroup result is confirmed in a completely
  different trial

27
(No Transcript)
28
Reporting of subgroups in journals review of 50
trial reports from general medical journals
Of those in which the number of analyses could be
established, almost 40 did at least six subgroup
analyses - one reported 24. Fewer than half
used statistical tests of interaction. No
information given on whether the subgroup
analyses were predefined or post hoc. Some
investigators selectively report only the more
interesting subgroup analyses, thereby leaving
the reader (and us) unaware of how many less
exciting subgroup analyses were looked at and not
mentioned.
Assmann SF, Lancet 2000 355106469.
29
Indications for early aspirin use in acute
ischaemic stroke a combined analysis of over
40,000 randomised patients from CAST and IST

• Chen ZM, Sandercock PAG, on behalf of IST and
  CAST collaborative groups
• Stroke 2000 31 1240-1249.

30
Aspirin benefits similar in different categories
of patients with acute ischaemic stroke. No
significant heterogeneity with

• age, sex
• hours delay since onset
• conscious level
• prior CT / infarct visible on CT
• AF status
• stroke syndrome
• baseline prognosis

31
Effects of aspirin on early death and non-fatal
recurrent stroke in subgroups IST and CAST
combined (Chen et al 2000)
32
Effect of aspirin on recurrent ischaemic stroke
in AF and SR
33
Conclusions

• No heterogeneity across different subgroups
• No evidence of an interaction in patients with
  AF
• Should NOT conclude aspirin ineffective in AF