Large Trials vs. MetaAnalysis of Smaller Trials: Implications to EvidenceBased Medicine

1
Large Trials vs. Meta-Analysis of Smaller
TrialsImplications to Evidence-Based Medicine

• Joseph C. Cappelleri, Ph.D., M.P.H
• Pfizer Inc
• Pfizer Global Research Development -
  Biostatistics
• Global Outcomes Research
• Groton, CT

2
Outline

• Objective
• To evaluate results of large clinical trials
  versus pooled results of smaller trials
• Prologue and Introduction
• Methods
• Results
• Lessons Learned
• Exploring Heterogeneity
• Planning a Large Study or Mega Trial
• Conclusions and Epilogue

3
Prologue

• Evidence-based medicine is the conscientious,
  explicit, and judicious use of current best
  evidence in making decisions about the care of
  individual patients
• Sackett et al. BMJ 1996 31271-72
• The practice of evidence-based medicine means
  integrating individual clinical expertise with
  the best available external clinical evidence
  from systematic research

4
IntroductionMotivation

• Mega trials for acute myocardial infarction (AMI)
• ISIS-4 (n58,050)
• Nitrates reduced mortality by 3 (not
  significant)
• Magnesium increased mortality by 5 (not
  significant)
• GISSI-3 (n18,895)
• Nitrates reduced mortality by 6 (not
  significant)
• Meta-analysis of smaller trials
• Nitrates significantly reduced mortality by 32
• Magnesium significantly reduced mortality by 48
• LIMIT-2 (n2,316) gave significant mortality
  reduction of 25

5
IntroductionQuestions

• How well do large and smaller studies agree?
• How frequent are the statistically significant
  disagreements?
• Why do these disagreements occur?
• Are the disagreements clinically important?
• When should a study be declared large?

6
MethodsSelection of Meta-analyses

• Meta-analyses of randomized controlled trials
• Human subjects
• Binary outcomes
• At least one large study and two smaller
  studies
• MEDLINE (1966-1995)
• English literature
• Comprehensive search
• Cochrane Pregnancy and Childbirth Database (1994)
• Data extraction
• Most important primary outcome as stated in the
  article
• If not stated, most important outcome by blinding
  consensus

7
MethodsDefinition of a Large Study -- Sample
Size Definition

• At least 1000 subjects
• For example,
• Control rate 0.10
• Experimental rate 0.05
• Risk ratio 0.5
• 0.05 level of significance
• 80 power

8
MethodsDefinition of a Large Study --
Statistical Power Definition

• 4-Step algorithm
• (1) Pooled relative risk reduction and weighted
  control rate
• All studies except largest by size (N)
• (2) Minimum total sample size (n)
• 80 statistical power and 5 level of
  significance
• (3) Determine whether study is large
• If N
• If N ? n, study large
• (4) If study large, repeat procedure sequentially
• With next study largest by size

9
MethodsStatistical Analysis -- Models

• Risk Ratio (RR)
• DerSimonian and Laird (random-effect) model
• Main model
• Incorporates potential diversity among trials
• Mantel-Haenszel (fixed-effect) model
• Assumes a common effect for each study
• Does not consider inter-study variability

10
MethodsStatistical Analysis -- Quantifying
Concordance (Agreement)
ln(RR in large study or studies) - ln(RR
in smaller studies)

• Z

square root of variance of ln(RR in
large) variance of ln(RR in smaller)

• Agreement -1.96

• Disagreement Otherwise

• Probed disagreements (from random effects model)

11
MethodsProbing Disagreements -- Control Rate
Meta-Regression

• Schmid et al. Statistics in Medicine 1998
  171923-1942
• Control rate (CR) is a summary measure of patient
  or study differences
• Regressed ln(RR) on CR in each study
• Fitted hierarchical model by EM algorithm
• Tested for statistical difference
• CR of largest study vs. pooled CR of smaller
  studies
• Pooled CR of large studies vs. pooled CR of
  smaller studies
• CR pooled with random-effect logistic
  transformation

12
MethodsProbing Disagreements -- Publication Bias

• Studies with non-significant results less likely
  to be published
• Mostly studies with small sample sizes
• Begg, Mazumdar. Biometrics 1994 501088-110
• Treatment effects associated with variance (or
  sample size)
• Rank correlation test
• Publication bias suggested when p-value ? 0.10

13
MethodsProbing Disagreements -- Specified and
Unspecified Reasons

• Specified protocol differences by authors of
  meta-analysis
• If none identified, difference were examined for
• Clinical relevance
• Previously unspecified (by others) protocol or
  methodological differences

14
ResultsSelection of Studies

• Initially screened
• 2100 MEDLINE citations
• 500 Cochrane systematic reviews
• Large sample size (? 1000 subjects)
• 79 meta-analyses
• Sufficient power (80 power)
• 61 meta-analyses

15
ResultsConcordance of Large Studies with
Smaller Studies

• Random-effect model
• Large sample size
• 71 of 79 meta-analysis agreed (89.9)
• Sufficient power
• 50 of 61 agreed (82)
• Correlation0.75

• Fixed-effect model
• Large sample size
• 65 of 79 agreed (82.3)
• Sufficient power
• 40 of 61 agreed (65.6)
• Correlation 0.75

• More agreement with random-effect model
• More agreement with sample size approach

16
ResultsAccounting for Disagreements

• Random-effect model gave 15 disagreements
• Control rate
• RR significantly related to CR in 5 of 15
  disagreements
• Pooled CR of smaller trials pooled CR of large
  trials
• Nitrates for AMI
• Pooled CR of smaller trials CR of largest trial
  (ISIS-4)
• Magnesium for acute MI
• Nitrates for AMI

17
ResultsAccounting for Disagreements

• Publication bias 1 meta-analysis
• Not clinically important 2 meta-analyses
• Specific protocol differences (cited by others)
• 4 meta-analyses
• Tentative reasons differences (we observed)
• 4 meta-analyses
• No specific reasons 1 meta-analysis
• Studies had similar sample sizes

18
Lessons Learned

• Study differences
• Treatment effect may depend on dose, timing,
  compliance, length of follow-up, concomitant
  interventions, inclusion/exclusion criteria, type
  of patient subgroup
• ISIS-4 late administration of magnesium, high
  rate of use of other mortality-reducing therapy
  (thromboloytics and aspirin)
• Baseline risk of patients
• Treatment effect may depend on risk level of
  cohort
• Magnesium may benefit mostly those at high risk
  of mortality
• ISIS-4 control rate only 7.2
• Pooled control rate of other studies 10.1
• Spawned new mega trialMagnesium in
  Coronaries(MAGIC)
• Short-term mortality in 10,000 high-risk AMI
  patients
• Magnesium administered early

19
Exploring Heterogeneity

• Meta-analysis can be used to capitalize on
  heterogeneous populations and studies
• Pooled result may not tell whole story for all
  treatments diseases
• Meta-analysis can uncover unknown effect
  modifiers
• Meta-analysis can generate hypotheses to test in
  a large study or mega trial

20
Planning a Large Study or Mega Trial

• Include subgroups or individuals with diverse
  treatment responses previously judged by
  meta-analysis
• Determine benefit-risk overall and for subgroups
• Or include individuals (or subgroups) likely to
  benefit
• Saves cost of a large study
• Reduces chance of a negative finding
• Detect modest effect and increase precision of
  effect
• Especially in a non-placebo large study
• Helps physician to treat individual patient
• Identify characteristics of treatment and study
  cohort or subgroup
• Decide whether that result applies to a
  particular patient

21
Conclusions

• Results between large and small studies usually
  agree
• But discrepancies not uncommon
• Learnings from heterogeneity of study results
  should be embraced and used to help plan large
  studies or mega trials
• Meta-analyses and small large trials offer
  continuum of complementary evidence in forming
  part of evidence-based medicine
• References
• Cappelleri et al. JAMA. 19962761332-1338
• Antman et al. Cardiovascular Drugs and Therapy.
  199610297-301

22
EpilogueThree Major Investigations

• Cappelleri et al. JAMA 1996 2761332-1338
• Villar et al. Lancet 1995 345772-776
• Moderate agreement between meta-analyses and
  large trials
• LeLorier et al. NEJM 1997 337536-543
• 35 of the time meta-analysis failed to
  accurately predict results of large trial
• Seemingly discordant conclusions led to
  controversy
• NEJM 1997 33859-62

23
Epilogue

• Meta-Meta-Meta-Analysis (Meta-Madness)
• Ioannidis et al. JAMA 1998 2791089-1093.
• Scrutiny of 3 protocols shows extent of agreement
  depends
• Selection of studies (selection bias)
• How disagreements are defined (method of
  analysis)
• Primary endpoint vs. secondary endpoint
• Consensus Meta-analysis and large trials
  disagree 10 to 23 of the time

24
Systematic Research At the Heart of
Evidence-Based Medicine