EPI-820 Evidence-Based Medicine

1
EPI-820 Evidence-Based Medicine

• LECTURE 10 Meta-Analysis II
• Mat Reeves BVSc, PhD

2
4. Analysis

• Several considerations
• Primary focus/motivation of study?
• Summary effect or explore heterogeneity?
• What kind of data are you combining?
• Dichotomous (categorical) (OR/RR, RD )
• Continuous (effect size diff/SD)
• Diagnostic data (sensitivity and specificity)
• Fixed vs random effects model?

3
Analysis Primary Goal

• What should be the primary goal?
• To provide a summary estimate or explore of
  presence and sources of heterogeneity? It
  depends.
• If studies are homogeneous then generate a
  summary estimate with 95 CI
• Much more likely to happen in RCTs where
  randomization has helped control bias and
  confounding
• If studies are heterogeneous then focus of study
  should be to investigate the sources of this
  variability
• More likely to happen in observational studies
  where differences in populations, methods and
  uncontrolled bias and confounding are rampant.
  But also occurs in RCTs.

4
Exploring heterogeneity

• Heterogeneity is the norm rather than the
  exception
• Heterogeneity can result from
• Methodological differences
• Biological differences
• Heterogeneity Statistics (Q)
• Power generally low (because study Ns are
  typically small)
• Power also affected by size of deviations between
  studies
• Statistical vs clinical heterogeneity
• What is the size of the statistical
  heterogeneity? Does it make sense? Could it have
  arisen due to random error? (chance)

5
Sources of clinical heterogeneity

• Clinical heterogeneity can be due to differences
  in
• study design or characteristics
• hospital vs population-based observational
  designs
• DBPC vs open trials
• Study population (sources), study quality
• Selection criteria for subjects, treatments or
  follow-up
• Sub-group responses (biological interaction)
  esp. in RCTs
• Bias or confounding esp. in observation studies
• Explored using stratification/sub-group analyses
• See Bernal, 1998

6
RR of Vasectomy on Prostate CA Risk Effect of
study characteristics/quality (Bernal, 1998)
Study Characteristic Summary RR 95 CI
Design
Cohort 1.1 0.8 - 1.5
CCS 1.4 1.0 - 1.8
Setting
Population 1.1 1.0 - 1.3
Hospital 2.0 1.4 - 2.9
Rating of control selection
Adequate 1.1 0.9 - 1.3
Inadequate 2.2 1.4 - 3.5
Rating of presence detection bias
Adequate 1.1 1.0 - 1.3
Inadequate 1.9 1.4 - 2.6
7
Fixed and random effects models

• Homogeneity and heterogeneity
• Heterogeneity depends on the degree of
  between-study variability in a group of studies.
• Fixed effects models
• consider only within-study variability.
• assumption is that studies use identical methods,
  patients, and measurements that they should
  produce identical results - any differences are
  only due to within-study variation only.
• Answer the question
• Did the treatment produce benefit on average in
  the studies at hand?

8
Random Effects models

• consider both between-study and within-study
  variability.
• assumption is that studies are a random sample
  from the universe of all possible studies.
• Answer the question
• Will the treatment produce a benefit on
  average?
• Note that random effects models do not adjust
  for, account for, or explain heterogeneity
• A random effects model does not therefore solve
  the problem of heterogeneity!

9
Fixed and random effects models

• Can give very different answers, and you can
  create examples where either model gives
  counterintuitive results (see Petitti, page 96)
• Random effects gives non-significant summary
  statistic for two studies that are each
  significant
• Fixed effects model gives the same confidence
  interval when you would expect a broader and
  narrower CI
• Usually, though, answers are similar.
• Example Comparison of 22 meta-analyses, fixed
  and random effects models gave the same answer in
  19/22. In 3 cases, fixed effects models were
  significant while random effects models were not
  (Berlin, 1989).

10
Fixed and random effects models

• Differences only arise when studies are not
  homogenous.
• When there is significant heterogeneity, the
  between-study variance becomes much larger than
  the within, and studies of different sample size
  receive relatively similar weight.
• When there is homogeneity, sample size dominates,
  and both models give similar results.
• Random effects models are more conservative and
  generate a wider confidence interval (because
  they add in the between-study variance).
• Random effects models also tend to give greater
  weight to small studies (which maybe more biased?)

11
What to do?, what to do?

• If homogenous, use fixed effects model
• random will give same results
• fixed is computationally simpler
• If heterogeneousthen first ask why?!
• In the face of heterogeneity, focus of analysis
  should be to describe possible sources of
  variability - attempt to identify sources of
  important subgroup differences
• Example studies using one dose showed
  significant effect, while lower dose did not.
  Then do fixed effects analysis of each sub-group
  and report all results.

12
Use of the Random Effects Model?.

• Many observers dispute the rationale for
  random-effect based analyses. For example
• Petitti (2000) . in the very situations where
  application of the method matters (
  heterogeneity), a single summary estimate of
  effect is inappropriate
• Greenland (1994) the random effects model is
  the model or summary of last resort

13
Statistical Tests of Homogeneity (heterogeneity)

• Homogeneity calculations
• Ho studies are homogeneous
• Based on testing the sum of weighted differences
  between the summary effect and individual effects
• Calculate Mantel Haenszel Q, where
• Q ?weighti x (lnORmh - lnORi)2
• To interpret, use the chi-square distribution
  where the degrees of freedom S - 1 (where S is
  the number of studies). If p lt 0.05, then there
  is significant heterogeneity.
• Power of such statistical tests is low (a
  non-significant test does not rule out clinically
  important heterogeneity)

14
Specific methods for dichotomous data

• Mantel-Haenszel method (fixed effects)
• originally developed to handle analysis of data
  in multiple strata. If you think of each study as
  a stratum, you can do a meta-analysis!
• data must be in form of 2 x 2 table for
  Mantel-Haenszel
• odds ratio, rate ratio, risk ratio
• Most commonly used method for meta-analysis (has
  optimal statistical properties)
• Only accounts for confounding if it is
  incorporated into the study design (matching or
  randomization)
• therefore, cant use multivariable adjusted data.

15
Mantel-Haenszel Method
Exposed Unexposed Total
Diseased ai bi gi
Non-diseased ci di hi
Total ei fi ni
16
Mantel-Haenszel Method

• ORmh ? (weighti x ORi) / ? weighti
• ORi (ai x di) / (bi x ci)
• weighti 1 / variancei
• variancei ni / (bi x ci)
• 95 CI e ln(ORmh) /- 1.96 x sqrt(var ORmh)
• var ORmh
• (?F / 2 x ?R2) ?G / (2 x ?R x ?S) (?H/(2 x
  ?S2)
• where
• F ai x di x (ai di)/ni2
• G ai x di x (bici) (bi x ci x (ai di))
  / ni2
• H (bi x ci x (bici)) / ni2
• R (ai x di) / ni
• S (bi x ci) / ni

17
(Sir Richard) Peto Method

• Fixed effects
• very similar to Mantel-Haenszel method (same 2x2
  requirement)
• see Pettiti pages 104-107 or Hasselblad article
  for formulae
• computationally somewhat simpler, especially to
  calculate the confidence interval
• may provide biased results under some
  circumstances in which Mantel-Haenszel would not
• Best applied to RCTs and not observational
  studies

18
General Variance Methods

• Used to summarize rate/risk differences (RD)
• Fixed effects method
• RDs ?(wi x RDi) / ?wi
• wi 1 / variancei
• 95CI RDs /- 1.96 (variances)0.5
• Variances 1/?wi
• see text page 107 for more details
• formulas differ if analyzing rate ratio data
  (incidence-density) or risk ratio data
  (cumulative risk)
• General variance-based methods also used for
  observational studies when study results are
  presented as RR with 95 CI

19
Random effects models

• DerSimonian and Laird statistic
• Uses odds ratios only!
• lnORdl ?(wi x lnORi) / ?wi
• wi 1 / D (1/wi)
• wi 1 / variancei
• D (Q - (S - 1) x ?wi ) / (?wi)2 - ?wi2
• Q ?wi x (lnORi - lnORmh)2
• CI exp(lnORdl 1.96 x (variances)0.5
• variances ?weighti

20
Continuous outcomes

• Two approaches
• 1. Each study used the same scale or variable
  (i.e. all measured SBP, serum creatinine or
  Mini-Mental State score). Based on ANOVA model
  where studies are groups.
• meansummary ?(weighti x meani) / ?weighti
• meani meantx - meancontrol
• weighti 1 / variancei 1 / SDi2
• (use pooled variance)
• 95 CI means /- (1.96 x (variances)0.5)
• variances 1 / ?weighti
• Test of homogeneity Q ?weighti x (means -
  meani)2

21
Continuous outcomes

• 2. Each study used a similar but different scale
  (e.g., CAGE and MAST for diagnosis of
  alcoholism, pulmonary function tests PEFR,
  FEV1)
• dsummary ?(weighti x di) / ?weighti
• dsummary summary estimate of the difference in
  effect sizes
• di effect size (meantx - meancontrol) /
  SDpooled
• weighti 1 / variancei (2 x Ni) / (8 di2)
• (use pooled variance)
• 95 CI ds /- (1.96 x (variances)0.5)
• variances 1 / ?weighti
• Test of homogeneity Q ?weighti x (ds - di)2

22
Other Issues in Meta-Analysis

• Cumulative M-A
• See article by Antman for example
• Pooling Studies
• See article by Blettner (Type III study)
• M-A of observational studies
• M-A of diagnostic tests
• Meta-regression

23
M-A of Observation Studies

• Very controversial application with some authors
  rejecting the approach outright (Shapiro, 1994)
• Often applied to controversial topics where
  previous studies are inconclusive (due to small
  risks and/or small studies)
• Exam Chlorination and CA risk, EMF and CA risk.
• But can never exclude bias.
• Important to regard process as a study of
  studies and not a means of providing a summary
  estimate
• Very valuable process at identifying deficiencies
  in published literature
• See Stroup et al (JAMA 2000) proposal for
  reporting

24
Meta-analysis of diagnostic tests

• See Irwig article (bibliography) for an excellent
  overview.
• Simply averaging sensitivity and specificity is
  not useful
• Se Sp
• Study 1 0 100
• Study 2 99 99
• Study 3 100 0
• Mean 67 67

25
What to do?

• Can calculate a summary ROC curve, by plotting
  the sensitivity and specificity for each study of
  a diagnostic test.
• Especially useful for comparing tests
• e.g. stress thallium vs stress echocardiogram for
  heart disease.
• See Irwig article for details of calculations.

26
Plotting an ROC curve
O
O
O
Se
O
Each circle represents an individual study
O
1 - Sp
27
Figure 3. Summary receiver-operating
characteristic (SROC) curve analysis of ELISA
D-dimer in the diagnosis of PE. Plotted in each
of the SROC graphs are individual studies
depicted as ellipses. The x- and y-dimensions of
the ellipses are proportional to the square root
of the number of patients available to study the
sensitivity and specificity, respectively, within
the analysis. Also shown is the unweighted SROC
curve limited to the range where data are
available. The cross (x) represents the
independent random-effects pooling of sensitivity
and specificity values of the studies.
28
Meta-regression

• Multivariate approach
• Use the study characteristics as independent
  variables
• Design, age, population source, quality score etc
  etc
• Use effect size or other outcome as the dependent
  variable
• Identify significant study characteristics
• Unit of observation study
• Can be useful to identify sources of
  heterogeneity, clarify importance of quality
  scores
• Exploratory only

29
Meta-regression Example(Phillips 1991 26 HIV
studies, Dependent var Specificity)
Variable Regression Co-efficient T P value
Year of pub. -0.023 -0.90 gt 0.05
Low HIV Prev 0.114 -2.54 lt 0.05
High vs Med Quality -0.014 -0.20 lt 0.05
Low vs Med Quality -0.087 -1.38 lt0.05
30
Final comments

• Remember the art of meta-analysis knowing
  when to use which technique, rather than
  mindlessly applying formulae to studies.
• Understanding the underlying clinical rationale
  for treatment, differences in populations, and
  differences in outcomes is critical.
• An important contribution of MA is to highlight
  the variability in the design, conduct, analysis
  and findings of a particular body of literature.

31
Bibliography

• Highly recommended reading
• Hasselblad V, McCrory DC. Meta-analytic tools
  for medical decision-making a practical guide.
  Med Decis Mak 1997 15 81-96.
• Irwig L, Tosteson AN, Gatsonis C, et al.
  Guidelines for meta-analyses evaluating
  diagnostic tests. Ann Intern Med 1994 120
  667-76.

32
Other recommended reading

• Cook DJ, Guyatt GH, Ryan G, et al. Should
  unpublished data be included in meta-analyses?
  JAMA 1993p 269 2749-53.
• Greenland S. A critical look at some popular
  meta-analytic methods. Am J Epid 1994 140
  290-6.
• LAbbe K, Detsky ASlt ORourke K. Meta-analysis
  in clinical research. Ann Intern Med 1987 107
  224-33.
• LeLorier J, Gregoire G, Benhaddad A, et al.
  Discrepancies between meta-analyses and
  subsequent large randomized, controlled trials.
  N Engl J Med 1997 337 536-42.
• Eddy DM, Hasselblad V, and Schachter. An
  introduction to a Bayesian method for
  meta-analysis. Med Decis Mak 1990 10 15-23.
  (REQUIRES SPECIAL SOFTWARE)
• Sacks HS, Berrier J, Reitman D, et al.
  Meta-analyses of randomized controlled trials. N
  Engl J Med 1987 316 450-5.

33
Other recommended reading

• Cook DJ, Sackett DL, Spitzer WO. Methodologic
  guidelines for systematic reviews of randomized
  control trials in health care from the Potsdam
  Consultation on Meta-Analysis. J Clin Epidemiol
  1995 48 167-71.
• Chalmers TC, Smith H, Blackburn B, et al. A
  method for assessing the quality of a randomized
  control trial. Control Clin Trials 1981 2
  31-49.
• Sackett DL. Applying overviews and meta-analyses
  at the bedside. J Clin Epidemiol 1995 48
  61-6.
• Olkin I. Statistical and theoretical
  considerations in meta-analysis. J Clin
  Epidemiol 1995 48 133-46.

34
Sample meta-analyses

• Clark P, Tugwell P, Bennett K, Bombardier C.
  Meta-analysis of injectable gold in rheumatoid
  arthritis. J Rheumatol 1989 16 442-7.
• Rowe BH, Keller JL, Oxman AD. Effectiveness of
  steroid therapy in acute exacerbations of asthma
  a meta-analysis. Am J Emerg Med 1992 10
  301-10.
• Cummings P. Antibiotics to prevent infection in
  patients with dog bite wounds a meta-analysis
  of randomized trials. Ann Emerg Med 1994 23
  535-40.
• Callahan CM, Drake BG, Heck DA, Dittus RS.
  Patient outcomes following tricompartmental total
  knee replacement a meta-analysis. JAMA 1994
  271 1349-57.
• Phillips KA. The use of meta-analysis in
  technology assessment a meta-analysis of the
  enzyme immunosorbent assay HIV antibody tests. J
  Clin Epidemiol 1991 44 925-31.