Issues in Data Synthesis in Systematic Reviews'

1
Issues in Data Synthesis in Systematic Reviews.

• PEP Systematic Review Session
• November 18, 2005
• Ben Vandermeer, MSc
• EPC/ARCHE Statistician

2
Data Synthesis

• Which summary measure
• Dichotomous
• Continuous.
• Combining results
• Methods
• Assessing heterogeneity
• Subgroups/sensitivity analyses
• Random effects vs fixed effects models
• Publication bias assessment.

3
Which summary measure?

• Possible types of data
• Dichotomous (alive/dead)
• Continuous (Normal) discrete, long ordinal,
  counts of common events (severity ratings)
• Short ordinal (high-moderate-low)
• Time-to-event (survival)
• Rate data number of events in person-time-units
  of follow-up, Poisson
• Diagnostic measures.

4
Dichotomous--2x2 Table
5
Dichotomous RD

• Risk difference absolute or additive measure
• Risk describes the probability of an event
  occurring
• Risk range 0,1 or 0,100
• RD describes the risk of an event in the
  treatment group minus risk of an event in the
  control group

6
Example 2x2 Table
7
Risk Difference

• Easy to interpret
• RD is -0.09
• The treatment group is 9 less at risk than the
  control group, or the control group is 9 more at
  risk then the treatment group
• RDs range -1,1 or -100,100

8
Dichotomous RR

• Relative risk or risk ratio relative or
  multiplicative measure.
• RR describes the risk of an event in the
  treatment group relative to the risk of an event
  in the control group

9
Relative risk

• Easy to interpret
• If RR is 0.10
• The treatment group is 0.10 times the risk of
  the control, or the control group is 10 (1/0.10)
  times more at risk than the treatment group
• Relative risk reduction 100 x (1 RR)
• The treatment decreases the event rate by 90
• Only used for prevention, not benefit

10
Relative risk

• RRs range 0,1/riskctr
• Large RRs are impossible with common events
• Is actually two measures benefit or harm
• An event occurring vs an event not occurring
• One is not the inverse of the other
• Can be different with respect to statistical
  significance as well as (a little) in magnitude

11
Relative Risk

12
Dichotomous OR

• Odds ratio relative or multiplicative measure
• Odds describes the probability that an event will
  occur divided by the probability that an event
  will not occur not the same as risk!
• Odds range 0, n-1
• OR describes the odds of an event in the
  treatment group relative to the odds of an event
  in the control group

13
Odds Ratio

• Not easy to interpret
• OR is 0.09 (or 1/11)
• The treatment group is 0.09 times less the odds
  as compared to the control group, or the control
  group is 11 (1/0.09) times more the odds as
  compared to the treatment group
• ORs range 0,infinity)

14
Odds Ratio

• Only one measure
• Benefit is the inverse of harm

15
Odds Ratio

• Approximates RR only when events are rare
• Otherwise ORltRR when RRlt1 and ORgtRR when RRgt1
• More sensitive a measure than RR
• OR can be converted into a RR for
  interpretability in the discussion

16
Dichotomous NNT

• Number needed to treat (NNT) or number needed to
  harm (NNH) arises from side effects caused by
  the treatment
• NNT describes the number needed to be treated in
  order to prevent one failure
• Easiest to calculate from an RD, NNT is 11
  (1/-0.09)
• You need to treat 11 patients with the treatment
  in order to prevent one failure

17
Dichotomous - NNT

• Relative measures can also be converted to a NNT

18
Dichotomous

• No single best choice
• Consistency
• Understood and interpretable
• Mathematical properties

19
Dichotomous Summary

• Consistent RR and OR
• Defined on every numerical result RD
• Easy to interpret RD and RR
• Consider NNT

20
Continuous Data
21
Continuous

• Assumes that the outcome has a normal
  distribution
• Skewed results should be analyzed differently
• Continuous, here, will also include discrete
  measures (e.g., heart rate), long ordinal scales
  and counts of common events
• Counts of rare events should be treated
  differently, use rates (per persons-time-units of
  follow-up), or dichotomize how many patients
  had at least one event

22
Mean Difference

• Measured in natural units, preferable
• Treatment mean minus the control mean
• Used to calculate the weighted mean difference
  a method of data synthesis

23
Standardized Mean Difference

• Often called effect size, although the term
  effect size can be used to generally refer to any
  measure of effect
• Measured in units of standard deviation
• Used when outcomes are conceptually the same but
  measured in different ways
• e.g., scales
• Hard to interpret, can translate SMDs into WMDs
  using the most popular scale or best validated
  scale for purposes of discussion (via a pooled SD
  for that particular scale)

24
Data Synthesis

• Methods
• depends on summary measure
• Fixed effects vs random effects
• Heterogeneity class

25
Methods

• Inverse-Variance (1930s)
• Used with almost any measure with a standard
  error
• Mantel-Haenszel (1959)
• Dichotomous measures only
• Peto (1977)
• Odds ratio only
• Advanced Methods
• Maximum likelihood theory, Bayesian theory, Exact
  methods

26
Inverse-variance

• Individual measures are weighted according to the
  inverse of their variances (square of the SE)
• Birge RT 1932 in a physics journal, Cochran WG
  1937 in a stats journal

27
Method Performance

• Inverse-Variance
• For general use
• Mantel-Haenszel
• Performs better than inverse-variance with sparse
  data
• Peto
• Performs well with large balanced trials and low
  event risks

28
Confidence Intervals

• Use them!
• For every 100 repeated experiments, 95 out of 100
  of these point estimates will be contained within
  the original 95 confidence interval
• Range of plausible values, are any of them
  clinically meaningful?
• Goodman SN, Berlin JA. The use of predicted
  confidence intervals when planning experiments
  and the misuse of power when interpreting
  results. Annals of Internal Medicine. 1994
  121(3)200-206.
• The test statistic (or p-value) is less
  meaningful, generally not reported in Cochrane
  reviews

29
Data Synthesis Review

• Dichotomous summaries
• RRs are interpretable and consistent
• RDs are defined for rare events (eg, AEs)
• NNTs help clinical interpretation, use the
  primary summary statistic in its calculation
• Mantel Haenszel method usually performs the best

30
Data Synthesis Review

• Continuous summaries
• MD is in its natural unit, preferable
• SMD only if units are different (eg, diff.
  scales)
• Inverse-variance is the only possible method
• Confidence intervals
• Contain the only plausible values, check this
  against your MCID (forget about power)

31
Heterogeneity

• What is heterogeneity?
• How to quantify/test for heterogeneity?
• How to model heterogeneity?
• How to explore/investigate heterogeneity?

32
Heterogeneity

• State of being dissimilar
• Different kinds of heterogeneity clinical (or
  whatever science/content area), methodological,
  and statistical
• Clinical heterogeneity describes differences
  across participants, interventions and outcomes
  studied
• Methodological heterogeneity describes
  differences between trial designs and trial
  quality

33
Homogeneity
34
Heterogeneity
35
Heterogeneity

• Statistical heterogeneity indicates that the true
  underlying treatment effects in the trials are
  not identical, observed treatment effects are
  more different than one should expect due to
  random error (chance) alone
• Some meta-analyses focus on exploration of
  heterogeneity rather than a single combined
  estimate of effect
• An investigation of replication

36
Statistical Heterogeneity

• Two levels of variance
• Within versus between study variance
• Within or s2 random error occurring within
  studies, calculated from the variation amongst
  the individual subject data within a trial
• Between or t2 error occurring between studies,
  calculated from the variation between the
  individual study estimates

37
Assessing S. Heterogeneity

• Simply, look at the metagraph
• Testing for statistical heterogeneity (Old
  School)
• Given many studies, high power, frequent type I
  errors, false positives
• In general, there are too few studies, low power,
  frequent type II errors, false negatives
• Use P0.10 instead of P0.05
• Chi-square test for heterogeneity, has been
  referred to as the Q statistic, as well as, the
  DerSimonian and Laird test

38
Computer-Based Delivery of Health Evidence
Process of Care
39
(No Transcript)
40
Quantifying Heterogeneity

• We have moved away from testing for heterogeneity
  to quantifying heterogeneity (New School)
• because IT IS ALWAYS THERE even when our test is
  insignificant
• Higgins JPT Thompson SG Deeks JJ, and Altman
  DG. Measuring inconsistency in meta-analyses.
  BMJ. 2003 327557-560.

41
I2, the Higgins?

• Interpretation percent variability due to
  between (or inter-) study variability
• as opposed to within (or intra-) study variability

• Earlier versions of RevMan

42
Example 1
43
Example 2
44
Example 3
45
Rough Guidelines

• Small to none 0 to 20
• Moderate 20 to 50
• Large 50 to 100
• Balance I2 against total variability in terms of
  clinical importance (ie, confidence intervals)
• When large I2 can not be explained by sensitivity
  analyses, results should be interpreted
  cautiously
• A qualitative look at individual studies is
  necessary
• Consistency of direction is a gain

46
Modeling Heterogeneity

• Fixed Effects
• Assumes no between study variance
• Overestimates precision
• Fits with inverse-variance method view
• Random effects
• Incorporates both forms of variance
• Wider confidence intervals
• Intended for heterogeneity between study
  estimates that we cannot explain
• Should not be used to explain away heterogeneity

47
Random Effects vs Fixed Effects

• Some schools of thought
• Always FE, never should combine if you suspect
  heterogeneity (Egger)
• Always RE, there is always heterogeneity
  (Higgins)
• Use RE if Plt 0.10, otherwise FE (Old School)
• Not endorsed by the Cochrane handbook anymore

48
Subgroup vs Sensitivity

• Subgroups
• Investigating heterogeneous results and more
  specific questions relevant to particular
  clinical patient groups
• How about particular study groups?
• What about non-categorical covariates?
• Subgroup analyses might be part of a sensitivity
  analysis
• Sensitivity
• Testing how robust the results of the review are
  relative to key decisions and assumptions that
  were made in the process of conducting the review

49
Publication Bias

• Funnel plot
• Precision (ie, 1/SE), N, log N, SE (funnel is
  flipped or the axis is flipped) versus effect
  size
• Bias asymmetrical funnel
• Small sample bias
• Quantitative methods
• Weighted regression
• Rank correlation test
• Trim and fill

50
Symmetric Funnel Plot
51
Asymmetric Funnel Plot
52
Thanks for the invitation

• Questions?