Meta-analysis Principles and practice in cardiovascular research

1
Meta-analysisPrinciples and practice in
cardiovascular research

• Giuseppe Biondi Zoccai
• Istituto di Cardiologia, Università di Torino

2
Disclosure

• No funding or conflict of interest to declare

3
Index

• Introduction definitions
• Scientific hierarchy
• The Cochrane Collaboration
• Structured approach to systematic reviews
• Additional topics
• Statistical packages
• Further examples

4
Exponential increase in PubMed
citations
PubMed search strategy ("2001"PDAT
"2005"PDAT) AND (("systematic"title/abstract
AND "review"title/abstract) OR
("systematic"title/abstract AND
"overview"title/abstract) OR ("meta-analysis"ti
tle/abstract OR "meta-analyses"title/abstract))
5
Famous quotes

• If I have seen further it is by standing on the
  shoulders of giants
• Isaac Newton
• The great advances in science usually result
  from new tools rather than from new doctrines
• Freeman Dyson

6
Famous quotes

• I like to think of the meta-analytic process as
  similar to being in a helicopter. On the ground
  individual trees are visible with high
  resolution. This resolution diminishes as the
  helicopter rises, and in its place we begin to
  see patterns not visible from the ground
• Ingram Olkin

7
Baby steps of meta-analysis

• 1904 - Karl Pearson (UK) correlation between
  inoculation of vaccine for typhoid fever and
  mortality across apparently conflicting studies
• 1931 Leonard Tippet (UK) comparison of
  differences between and within farming techniques
  on agricultural yield adjusting for sample size
  across several studies
• 1937 William Cochran (UK) combination of
  effect sizes across different studies of medical
  treatments
• 1970s Robert Rosenthal and Gene Glass (USA),
  Archie Cochrane (UK) combination of effect sizes
  across different studies of, respectively,
  educational and psychological treatments
• 1980s exponential development/use of
  meta-analytic methods

8
Minimal glossary

• Review viewpoint on a subject quoting different
  primary authors
• Overview as above
• Qualitative review deliberately avoids a
  systematic approach
• Systematic review deliberately uses a systematic
  approach to study search, selection, abstraction,
  appraisal and pooling
• Quantitative review uses quantitative methods to
  appraise or synthesize data
• Meta-analysis uses specific statistical methods
  for data pooling and/or exploratory analysis
• Individual patient data meta-analysis uses
  specific stastistical methods for data pooling or
  exploration exploiting individual patient data
• ? Our goal systematic review ( meta-analysis)

9
Qualitative review
10
Systematic review and meta-analyses

• What is a systematic review?
• A systematic appraisal of the methodological
  quality, clinical relevance and consistency of
  published evidence on a specific clinical topic
  in order to provide clear suggestions for a
  specific healthcare problem
• What is a meta-analysis?
• A quantitative synthesis that, preserving the
  identity of individual studies, tries to provide
  an estimate of the overall effect of an
  intervention, exposure, or diagnostic strategy

11
Systematic review
12
Systematic review and meta-analyses
13
Individual patient data meta-analysis

• Ideally should be a systematic review and
  meta-analysis based on individual patient data
• Major pros
• a unique database containing primary studies is
  created and used (consistency checks and
  homogenous variables are created)
• the same analytical tools can be used across
  studies
• subgroup analyses can be performed even for
  groups that were not reported in the original
  publications
• Major cons
• some studies may have to be excluded (publication
  bias) because original authors may not provide
  source data
• poses major logistical and financial challenges

14
Individual patient data meta-analysis
BMJ 2002
15
Systematic review and meta-regression

• A meta-regression employs meta-analytic methods
  to explore the impact of covariates or moderators
  on the main effect measure or on other
• All the limitations of non-RCT studies applies,
  and thus they should mainly be regarded as
  hypothesis generating

16
Meta-regression
17
Cumulative and prospective meta-analyses

• A cumulative meta-analysis recomputes and plots
  the pooled effect estimate every time a new study
  is added
• A prospective meta-analysis is based on a
  specific a priori protocol for its conduct,
  analysis, and reporting, and may use also a
  cumulative design

18
Cumulative meta-analysis
Antman et al, JAMA 1992
19
Cumulative meta-analysis
20
Pros

• Systematic searches for clinical evidence
• Explicit and standardized methods for search and
  selection of evidence sources
• Thorough appraisal of the internal validity of
  primary studies
• Quantitative synthesis with increased statistical
  power
• Increased external validity by appraising the
  effect of an intervention (exposure) across
  different settings
• Test subgroup hypotheses
• Explore clinical and statistical heterogeneit

Lau et al, Lancet 1998
21
Cons

• Exercise in mega-silliness
• Mixing apples with oranges
• Not original research
• Big RCTs definitely better
• Pertinent studies might not be found, or may be
  of low quality or internal validity
• Publication and small study bias
• Average effect largely unapplicable to individuals

Lau et al, Lancet 1998
22
Cons
Smith et al, BMJ 2003
23
Biondi-Zoccai et al, BMJ 2006
24
Introduction

• Introduction definitions
• Scientific hierarchy
• The Cochrane Collaboration
• Structured approach to systematic reviews
• Additional topics
• Statistical packages
• Further examples

25
EBM hierarchy of evidence

1. N of 1 randomized controlled trial
2. Systematic reviews of homogeneous randomized
   trials
3. Single (large) randomized trial
4. Systematic review of homogeneous observational
   studies addressing patient-important outcomes
5. Single observational study addressing
   patient-important outcomes
6. Physiologic studies (eg blood pressure, cardiac
   output, exercise capacity, bone density, and so
   forth)
7. Unsystematic clinical observations

Guyatt and Rennie, Users guide to the medical
literature, 2002
26
Parallel hierarchy of scientific studies in
cardiovascular medicine
Qualitative reviews
Case reports and series
Observational studies
Systematic reviews
Observational controlled studies
Meta-analyses from individual studies
Randomized controlled trials
Meta-analyses from individual
patient data
Multicenter randomized controlled trials
Biondi-Zoccai, Ital Heart J 2003
27
(No Transcript)
28
Benson et al, NEJM 2000
29
Introduction

• Introduction definitions
• Scientific hierarchy
• The Cochrane Collaboration
• Structured approach to systematic reviews
• Additional topics
• Statistical packages
• Further examples

30
(No Transcript)
31
The Cochrane Collaboration

• Mission Statement
• The Cochrane Collaboration is an world-wide
  organisation that aims to help people make
  wellinformed decisions about healthcare by
  preparing, maintaining and promoting the
  accessibility of systematic reviews of the
  effects of healthcare interventions

32
The Cochrane Collaboration

• About 6000 contributors
• 50 Collaborative Review Groups (CRGs)
• 12 Centres throughout the world
• 9 Fields
• 11 Methods Groups
• 1 Consumer Network
• Campbell Collaboration

33
The Cochrane Collaboration

• Objectives
• Collaboration
• Building on the enthusiasm of individuals
• Avoiding duplication
• Minimising bias
• Keeping up to date
• Striving for relevance
• Promoting access
• Ensuring quality
• Continuity

34
The Cochrane Collaboration

• Cochrane Database of Systematic Reviews (CDSR)
  contains Cochrane systematic reviews
• Database of Abstracts of Reviews of Effectiveness
  (DARE) contains abstracts of non-Cochrane
  reviews
• Cochrane Central Controlled Trials Register
  (CENTRAL) contains titles or abstracts of RCTs
  from multiple sources
• Cochrane Database of Methodology Reviews
  contains Cochrane reviews of methods papers
• Cochrane Methodology Register (CMR) contains
  abstracts of non-Cochrane methods papers
• Health Technology Assessment Database (HTA)
  contains abstracts of HTA papers
• NHS Economic Evaluation Database (NHS EED)
  contains abstracts of economic analysis papers

35
Introduction

• Introduction definitions
• Scientific hierarchy
• The Cochrane Collaboration
• Structured approach to systematic reviews
• Additional topics
• Statistical packages
• Further examples

36
Algorithm for systematic reviews

• Definition of question and hypothetical solution
• Prospective design of the systematic review
• Problem formulation (population, intervention or
  exposure, comparison, outcome PICO)
• Data search
• Data abstraction and appraisal
• Data analysis quantitative synthesis
• Result interpretation and dissemination

FEED-BACK ON HYPOTHESIS
Biondi-Zoccai et al, Ital Heart J 2004
37
Definition of question and prospective design

• The clinical question should be clearly stated,
  being as much explicit as possible
• The review should be designed in as much details
  as possible, and yet with a limited a priori
  knowledge of the subject

Biondi-Zoccai et al, Ital Heart J 2004
38
Problem formulation according to the PICO approach

• Population of interest - eg elderly male gt2 weeks
  after myocardial infarction)
• Intervention (or exposure) eg intracoronary
  infusion of progenitor blood cells
• Comparison eg patients treated with progenitor
  cells vs standard therapy
• Outcome(s) eg change in echocardiographic left
  ventricular ejection fraction from discharge to
  6-month control

Biondi-Zoccai et al, Ital Heart J 2004
39
Data search

• After definition of question according to
  PICO approach, the appropriate key-words
  are used to search several
  databases
• Useful resources BioMedCentral, CENTRAL,
  clinicaltrials.gov, EMBASE, LILACS, and PubMed
• Conference proceedings
• Cross-referencing (snowballing)
• Contact with experts

40
Study search
A simple PubMed strategy for clinical studies on
LM PCI left AND main AND coronary AND stent NOT
case reports pt NOT review pt NOT editorial
pt A complex PubMed strategy for randomized
clinical trials on invasive vs conservative
strategies in ACS (randomized controlled
trialpt OR controlled clinical trialpt OR
randomized controlled trialsmh OR random
allocationmh OR double-blind methodmh OR
single-blind methodmh OR clinical trialpt OR
clinical trialsmh OR (clinical trialtw OR
((singltw OR doubltw OR trebltw OR
tripltw) AND (masktw OR blindtw)) OR
(latin squaretw) OR placebosmh OR
placebotw OR randomtw OR research
designmhnoexp OR comparative studymh OR
evaluation studiesmh OR follow-up studiesmh
OR prospective studiesmh OR cross-over
studiesmh OR controltw OR prospectivtw OR
volunteertw) NOT (animalmh NOT humanmh)
NOT (commentpt OR editorialpt OR
meta-analysispt OR practice-guidelinept OR
reviewpt)) AND ((invasive OR conservative AND
(coronary OR unstable angina OR acute coronary
syndrome OR unstable coronary syndrome OR
myocardial infarction)))
Biondi-Zoccai et al, Int J Epidemiol 2005
41
Study search
Reveiz et al, J Clin Epidemiol 2006
42
Study selection

• 1st - screening of titles and abstracts
• 2nd potentially pertinent citations are then
  retrieved as full reports and appraised according
  to prespecified and explicit inclusion/exclusion
  criteria
• 3rd studies fullfilling both inclusion and
  exclusion criteria, are then included in the
  systematic review

43
Andreotti et al, Eur Heart J 2005
44
Data abstraction and appraisal

• Abstraction of outcomes and moderator variables,
  possibly on prespecified data form
• Appraisal of the internal validity of primary
  studies (eg the risk of selection, performance,
  adjudication and attrition bias)
• Performed by single vs multiple reviewers, with
  divergences resolved by consensus (possibly after
  formal tests for agreement)

45
Data extraction
Buscemi et al, J Clin Epidemiol 2006
46
Internal validity of primary studies

• Many scales for the quality of included studies
  have been reported, but none is reliable or
  robust
• The recommended approach is to individually
  appraise the potential risk of the 4 biases (eg
  A-low, B-moderate, C-high, D-unclear from
  reported data)
• Selection bias (one group is different than the
  other)
• Performance bias (treatment is systematically
  different)
• Adjudication bias (outcome adjudication is
  selectively different)
• Attrition bias (follow-up duration or
  completeness is different)

47
Quality scales are unreliable
48
Quality scales are unreliable
49
Internal validity of primary studies
Hill et al, Eur Heart J 2004
50
Data synthesis

• Quantitative data synthesis is central to the
  practice of meta-analysis, and is based on a
  major assumptio
• individual studies that are going to be pooled
  are relatively homogeneous, both clinically and
  statistically, to provide a meaningful central
  tendency effect estimate

51
Effect sizes and p values

• Forms of research findings suitable to
  meta-analysis
• Central tendency research
• incidence or prevalence rates
• mean (standard error)
• Pre-post contrasts
• changes in continuous or categorical variables
• Group contrasts
• experimentally created groups
• comparison of outcomes between experimental and
  control groups
• naturally or non-experimentally occurring groups
• treatment, prognostic or diagnostic features
• Association between variables
• correlation coefficients
• regression coefficients

52
Effect sizes and p values

• The effect size makes meta-analysis possible
• it is the dependent variable
• it standardizes findings across studies such that
  they can be directly compared
• Any standardized index can be an effect size as
  long as it meets the following
• is comparable across studies (generally requires
  standardization)
• represents the magnitude and direction of the
  relationship of interest
• is independent of sample size
• We identify as p values (for effect) the measures
  of alpha error for hypothesis testing

53
Relative risks

• Relative risks (RR) are defined as the ratio of
  incidence rates, and are thus used for dichotomic
  variables)
• What is the meaning of RR
• RR1 means no difference in risk
• RRlt1 means reduced risk in group 1 vs 2
• RRgt1 means increased risk in group 1 vs 2
• RRs are easier to interpret but are less
  userfriendly from a statistical point of view
  (RRAvsB?1/RRBvsA) and may appear over-optimistic

54
Odds ratios

• Odds ratios (OR) are defined as the
  ratio of the odds (P/1-P) and also
  used for dichotomic variables
• When prevalences are low, they are a
  good approximation of RR
• They behave similarly to RR (OR1
  means no difference in risk, )
• ORs are less easy to interpret but more
  userfriendly from a statistical point of view
  (ORAvsB1/ORBvsA), yet also overoptimistic

55
Risk differences and number needed to treat/harm

• The risk difference (RD), ie absolute risk
  difference, is the difference between the
  incidence of events in the experimental vs
  control groups
• The RD is theoretically the most clinically
  relevant statistics, but changes too much with
  disease prevalence
• The number to treat (NNT), defined as 1/RD,
  identifies the number of patients that we need to
  treat with the experimental therapy to avoid one
  event
• The NNT is the most clinically meaningful
  parameter to express the impact of a treatment on
  a dichotomic outcome (eg death), but has the same
  limits of RD

Numbers needed to harm (NNH) similarly express
the number of patients that we
have to treat
with the experimental therapy to cause one
adverse event
56
RR, OR or RD/NNT?

• OR RR RD/NNT
• Communication -
• Consistency -
• Mathematics - -

57
Fixed vs random effects

• Statistical pooling may be based on
• Fixed effect methods (eg Mantel-Haenszel or
  Peto), if we can hypothesize studies were are
  estimating the same population risk estimate (eg
  with RR, OR, or RD)
• Random effect methods (eg DerSimonian-Laird),
  hypothesize individual studies are estimating
  different treatment effects (eg with RR, OR, or
  RD)
• Additionally, inverse variance weighting (either
  based on random or fixed effects) may be used to
  pool individual point estimates with pertinent
  standard errors (even with HR)

58
Our advice

• Both RR and OR can be your first choice
  statistics for uncommon events
• For common events, the OR is clearly less
  informative than the RR for the busy reader
• Complete your analyses by reporting RD and/or NNT
  for the sake of clarity
• Fixed effect methods are quite fine for
  homogeneous/ consistent data
• Random effect methods may be more appropriate for
  heterogeneous/inconsistent data, but often
  meta-regression (or even refraining from
  meta-analysis at all) might be the best option

59
Continous variables

• Continous variables can be pooled with
• Weighted mean differences (WMD), if the same
  variable is used across studies
• Standardized mean differences (SMD), if similar
  but not identical variables are used
• Inverse variance weighting, if only point
  estimates and standard errors are available

60
Small study bias

• Publication bias (eg the lower likelihood of
  being published for studies with negative
  findings, or those originating in non-English
  speaking countries) may bias the results of a
  meta-analysis
• Other types of small study bias may undermine the
  validity of a meta-analysis
• A number of tests, analogical (eg the funnel
  plot) or analytical (eg Eggers or Peters) have
  been proposed to appraise the likelihood of such
  small study bias

Peters et al, JAMA 2006
61
Statistical heterogeneity

• Statistical heterogeneity may be suspected by
  inspecting tables (summary estimates/SE) and
  forest plots, or analytically
• Chi-square, Breslow, or Cochran tests are most
  commonly used
• While a 2-tailed p0.05 is used for cut-off for
  hypothesis testing of effect, a 2-tailed p0.10
  is conventionally chosen for heterogeneity

62
Statistical inconsistency

• Statistical inconsistency (I2) has been recently
  introduced to overcome the risk of alpha and beta
  error of standard tests for statistical
  heterogeneity
• It is computed as (Q df)/Q x 100, where Q is
  the chi-squared statistic and df is its degrees
  of freedom
• I2 values of 25 suggest low inconsistency, 50
  moderate inconsistency, and 75 severe
  inconsistency

Higgins et al, BMJ 2003
63
Index

• Introduction definitions
• Scientific hierarchy
• The Cochrane Collaboration
• Structured approach to systematic reviews
• Additional topics
• Statistical packages
• Further examples

64
QUOROM

• xxx

65
(No Transcript)
66
QUADAS
Whithing et al, BMC Med Res Method 2003
67
Oxman and Guyatt index

• Evaluates the internal validity of a review on 9
  separate questions for which 3 distinct anwers
  are eligible (yes, partially/cant tell,
  no)
• 1. Where the search methods used to find evidence
  stated?
• 2. Was the search for evidence reasonably
  comprehensive?
• 3. Were the criteria for deciding which studies
  to include in the overview reported
• 4. Was bias in the selection of studies avoided
• 5. Were the criteria used for assessing the
  validity of the included studies reported?
• 6. Was the validity of all studies referred to in
  the text assessed using appropriate criteria
• 7. Were the methods used to combine the findings
  of the relevant studies reported?
• 8. Were the findings of the relevant studies
  combined appropriately relative to the primary
  question the overview addresses?
• 9. Were the conclusions made by the author(s)
  supported by the data and/or analysis reported in
  the overview?
• Question 10 summarizes the previous ones and,
  specifically, asks to rate the scientific quality
  of the review from 1 (being extensively flawed)
  to 3 (carrying major flaws) to 5 (carrying minor
  flaws) to 7 (minimally flawed). The developers of
  the index specify that if the partially/cant
  tell answer is used one or more times in
  questions 2, 4, 6, or 8, a review is likely to
  have minor flaws at best and is difficult to rule
  out major flaws (ie a score4). If the no
  option is used on question 2, 4, 6 or 8, the
  review is likely to have major flaws (ie a
  score3).

Oxman et al, J Clin Epidemiol 1991
68
Index

• Introduction definitions
• Scientific hierarchy
• The Cochrane Collaboration
• Structured approach to systematic reviews
• Additional topics
• Statistical packages
• Further examples

69
Statistical packages

• EasyMA (http//www.spc.univ-lyon1.fr/easyma.net/)
• RevMan (http//www.cochrane.org)
• For meta-analyses of medical interventions
• Meta-Test (jlau1_at_tufts.edu)
• Meta-DiSc (http//www.hrc.es/investigacion/metadis
  c.html)
• For meta-analyses of diagnostic tests
• FastPro
• NCSS
• SAS
• SPSS
• Stata
• WEasyMA

70
Revman
71
Revman
72
Funnel plot
73
Index

• Introduction definitions
• Scientific hierarchy
• The Cochrane Collaboration
• Structured approach to systematic reviews
• Additional topics
• Statistical packages
• Further examples

74
Meta-analysis of intervention studies
75
Meta-analysis of intervention studies
76
Meta-analysis of intervention studies
77
Meta-analysis of intervention studies
78
Meta-analysis of intervention studies
79
Meta-analysis of prognostic studies
Troponin
80
Meta-analysis of diagnostic studies
Louvard et al, JACC 2006
81
Take home message

• The validity of a meta-analysis refers to the
  soundness of the original studies and the
  procedures used to combine them
• Several dozens of potential validity leaks have
  been identified in these procedures
• Given possible weaknesses, the enterprise may
  seem hopeless, yet its no worse than that of a
  pilot reading a preflight checklist and testing
  against possibly disastrous conditions

82
Take home message

• However, the checking and testing makes it
  possible for airplanes to fly even long distances
  with only minimal risk of equipment failure
• Systematic reviews and meta-analyses, similarly,
  succeed when researchers enforce sound validity
  checklists

83
A few references

• Biondi-Zoccai GGL et al. Parallel hierarchy of
  scientific studies in cardiovascular medicine.
  Ital Heart J 2003 4 819-20
• Biondi-Zoccai GGL et al. Compliance with QUOROM
  and quality of reporting of overlapping
  meta-analyses on the role of acetylcysteine in
  the prevention of contrast associated
  nephropathy case study. BMJ 2006332202-209
• Biondi-Zoccai GGL et al. A practical algorithm
  for systematic reviews in cardiovascular
  medicine. Ital Heart J 20045486 -7
• Bucher HC et al. The results of direct and
  indirect treatment comparisons in meta-analysis
  of randomized controlled trials. J Clin Epidemiol
  199750683 9
• Cappelleri JC et al. Large trials vs
  meta-analysis of smaller trials how do their
  results compare? JAMA 1996 276 1332-8
• Clarke M et al, eds. Cochrane reviewers handbook
  4.2.0. (www.cochrane.org/resources/handbook/handbo
  ok.pdf)
• Cooper H et al, eds. The handbook of research
  synthesis. New York, NY Russell Sage Foundation,
  1994
• Cucherat M et al. EasyMA a program for the
  meta-analysis of clinical trials. Comput Methods
  Programs Biomed 199753187- 90
• Egger M et al, eds. Systematic reviews in health
  care meta-analysis in context. 2nd ed. London
  BMJ Publishing Group, 2001
• Glass G. Primary, secondary and meta-analysis of
  research. Educ Res 197653-8
• Glasziou P et al. Systematic reviews in health
  care. A practical guide. Cambridge Cambridge
  University Press, 2001
• Guyatt G et al, eds. Users guides to the medical
  literature. A manual for evidence-based clinical
  practice. Chicago, IL AMA Press, 2002
• Higgins JPT et al. Measuring inconsistency in
  meta-analyses. BMJ 2003327557 60
• Lau J et al. Summing up evidence one answer is
  not always enough. Lancet 1998351123 -7
• Moher D et al. Improving the quality of reports
  of meta-analyses of randomised controlled trials
  the QUORUM statement. Lancet 1999 354 1896-900
• Petitti DB. Meta-analysis, decision analysis, and
  cost-effectiveness analysis methods for
  quantitative synthesis in medicine. New York, NY
  Oxford University Press, 2000
• Song F et al. Validity of indirect comparison for
  estimating efficacy of competing interventions
• empirical evidence from published
  meta-analysis. BMJ 2003326472
• Thompson SG et al. How should meta-regression
  analyses undertaken and interpreted? Stat Med
  2002211559-73

84
For further slides on these topics please feel
free to visit the metcardio.org
websitehttp//www.metcardio.org/slides.html