Statistical issues

1
Statistical issues

• Cochrane Stroke Group Editorial Training Workshop
• Valencia, Spain, 20 May 2003

by Steff Lewis (with some slides stolen from Jon
Deeks and Julian Higgins)
2
Which binary effect measure?

• Revman calculates
• Odds Ratio (OR)
• Relative Risk (RR)
• Risk Difference (RD)

3
Choice of OR or RR or RD sometimes matters...

• But it mostly doesnt matter at all

4
(No Transcript)
5
(No Transcript)
6
Which binary effect measure?

• There is no simple answer
• As long as people do something sensible, its OK.

7
How to calculate Number Needed to Treat (NNT)

• Dont use subtotals (ignores randomisation in the
  studies)
• If the study is based on Odds ratios, dont use
  the Risk difference method in Revman (assigns
  different weighting to studies).
• An appropriate method is now on our website

8
Number Needed to Treat (NNT)

• When using Odds ratios, NNT varies with baseline
  event rate
• So you should present NNT across a range of
  baseline event rate values

9
Control event rates vary from 119/270 in
Atlantis B (44) to 10/12 in Mori (83)
10
NNT varies from 21 to 34
11
Choosing an effect measure - why death matters

• You need to include all deaths in primary outcome
  measure as it is more robust

12
(No Transcript)
13
Heterogeneity
14
Testing for heterogeneity
15
Power

• The test typically has low power
• because most meta-analyses have very few studies
• If there are a lot of studies the test has high
  power
• will detect unimportant differences between
  studies
• Properties of the test (and hence interpretation)
  depend on how many studies there are

16
Trials from Cochrane logo Corticosteroids for
preterm birth (neonatal death)
Heterogeneity test Q 11.2 (6 d.f.) p 0.08
17
Corticosteroids for preterm birth (neonatal
death)
Heterogeneity test Q 11.2 (6 d.f.) p 0.08
Heterogeneity test Q 44.7 (27 d.f.) p 0.02
18

• Clinical and methodological diversity is
  inevitable in a systematic review
• Statistical heterogeneity (variation in the true
  effects underlying the studies) is therefore
  inevitable
• If heterogeneity exists, why test for its
  presence?
• The important questions are,
• How large is the heterogeneity?
• How much does it impact on the findings of the
  review?

19
New measure to quantify heterogeneity (now in
Revman)

• where Q heterogeneity c2 statistic
• I2 can be interpreted as the proportion of total
  variability explained by heterogeneity

20
Corticosteroids

• Cochrane logo trials
• Q 11.2, df 6, p 0.08
• Cochrane logo trials, four times
• Q 44.7, d.f. 27, p 0.02

I2 46 (0 to 77) I2 40 (5 to 62)
21
Heterogeneity - how do we investigate its causes?

• Subgroup analysis
• Sensitivity analysis

22
Sensitivity analysis

• 'Are my results robust to...decision?'.
• E.g
• fixed effects vs random effects
• different ways of handling missing data
• effects of low quality studies
• Often, various versions of the analysis are
  compared to the 'main' analysis to look for
  differences.

23
Subgroup analysis

• Relate to the underlying clinical question.
• Groups of studies or groups of patients within
  studies are compared to each other (not to the
  main analysis).
• There should be pre-specified hypotheses that
  different sets of patients will respond in
  different ways.

24
Subgroup vs sensitivity

• subgroup and sensitivity analyses overlap.
• subgroup analysis can be used to answer 'Are my
  results robust to...decision?', thus be
  sensitivity analysis.

25
Subgroup vs sensitivity

• Too many subgroups definitely bad
• (multiple testing).
• Too many sensitivity analyses probably bad
• (long and unreadable reviews)
• Most of our reviews have very few studies in
  them...

26
Subgroup analysis

• How do you do a subgroup analysis properly?
• What things do people do wrong?
• Some studies with statistically significant
  heterogeneity...

27
(No Transcript)
28
Subgroup analysis

• Pre-planned subgroup analysis of method of
  administering the drug
• Dont just compare p-values
• They depend on the amount of data in each subgroup

29
(No Transcript)
30
(No Transcript)
31
Subgroup analysis

• Dont just compare fixed effect heterogeneity
  p-value to random effects heterogeneity p-value
• They are identical

32
(No Transcript)
33
(No Transcript)
34
Subgroup analysis

• Do use Jon Deeks method
• or Doug Altmans method

35
(No Transcript)
36
Jon Deeks method (the easiest from Revman)

• Q 114.54-(8.580.81)
• 105.15
• 2 subgroups, so 2-1 1 degree of freedom
• Chi-squared value of 105.15 with 1 degree of
  freedom plt0.0001

37
Doug Altmans method

• Take the difference in ln(OR) between subgroups.
• Calculate an overall s.e. from the subgroup
  s.e.s (can use reciprocal of the variance of O-E
  as subgroup variance)
• See if difference in ln(OR) is sig different
  from zero using a Normal approximation.

38
Subgroup analysis

• Dont use Deeks or Altmans methods to compare a
  subgroup to the results of the whole set of
  studies.

39
(No Transcript)
40
Statistics and the Cochrane Stroke Group editors

• Probably best to leave stats comments up to me
  (we dont want to confuse reviewers with 6
  different suggestions of how to do their stats)
• When you are lead editor, if I complain, and you
  dont understand why, ask!!
• (Multiple statisticians - is just me enough)