Principles of Research Synthesis

1
Principles of Research Synthesis
San Francisco Radiation Oncology Conference
February 28 to March 2, 2003
Benjamin Djulbegovic, M.D.,PhD. H. Lee Moffitt
Cancer Center University of South
Florida djulbebm_at_moffitt.usf.edu
2
I The need for research synthesis
3
The need for research synthesis

• Health care decision makers need to access
  research evidence to make informed decisions on
  diagnosis, treatment and health care management
  for both individual patients and populations.
• There are few important questions in health care
  which can be informed by consulting the result of
  a single empirical study.

4
II The problems with traditional review articles
5
The need for research synthesis

• Importance of review articles
• Review articles in medical journals summarize
  large amounts of information on a particular
  topic
• and therefore are useful and popular source of
  information for health care professionals
• review articles have the highest impact factor
• which means, that research, practice and
  policy-decisions are significantly influenced by
  review articles

6
Science of research synthesis problems with
traditional review articles

• Personal views on the available body of evidence
• Selection bias and selective citations monster
• Has been pervasive in medicine, economics and
  social sciences
• Can obscure up to 40-60 of true interventions
  effect
• In 2000, Nobel prize in Economic Science was
  awarded to James Heckman of the University of
  Chicago for his analysis of selection bias, which
  in turn profoundly affected applied research in
  economics as well as in other social sciences
• Lack of reproducibility
• that is, the lack of the key scientific criterion

7
Selective citation bias blind men and elephant
8
Critique of reviews of chemotherapy for ovarian
cancer

• Crx superior
• (by Qualitative analysis) 48/53
• Search strategy 3/53
• Inclusion/exclusion criteria 2/53
• Validity assessment 1/53
• Quantitative Assessment 3/53

Courtesy of Dr. C. Williams
9
Quality of Review Article158 articlesonly 2 met
all 10 criteria.
Ann Intern Med 1999131947-951
10
Research Synthesis terminology

• Systematic review. The application of strategies
  that limit bias in the assembly, critical
  appraisal, and synthesis of all relevant studies
  on a specific topic. Meta-analysis may be, but is
  not necessary, used as part of this process.
• Meta-analysis. The statistical synthesis of the
  data from separate but similar, i.e. comparable
  studies, leading to a quantitativ summary of the
  pooled results.

11
Key Distinctions Between Narrative and Systematic
Reviews, by Core Features of Such Reviews
Systematic Review
Core Feature
Narrative Review
Study question
Often broad in scope
Often a focused clinical question.
Which databases
Data sources and
Comperehensive search of many
were searched and
search strategy
databases as well as so-called gray
search strategy are not typically
literature. Explicit search strategy
provided.
Not usually specified, potentially
Selection of articles for
Criterion-based
Selection,
biased.
study
uniformly
applied.
Article review or
Variable, depending on
who is
Rigorous critical appraisal, typically
appraisal
conducting the review.
using
a data extraction form.
Some assessment of quality is almost
Study quality
If assessed, may not use formal
always included as part of the data
quality assessment.
extraction process.
Quantitative summary (
meta-analysis)

Often a qualitative summary.
Synthesis
if the data can be appropriately pooled
qualitative otherwise.
Inferences
Sometimes evidence-based.
Usually evidence-based
12
Principles of reliable detection of the effects
of health care interventions

• Methods to reduce bias
• Methods to reduce statistical imprecision

13
III Principles of systematic reviews and
meta-analysis
14
Principle 1the need to consider the totality of
evidence

• The world can be only considered as the totality
  of factsfor the totality of facts determines
  what is the case, and also whatever is not the
  case
• L. Wittgenstein
  (Tractus logico-philosophicus), 1921

15
Principle 2 requirement for reproducibility

• Transparent, explicit and systematic approach in
  identifying and synthesizing evidence
• Methods for search for evidence
• Inclusion and exclusion criteria
• Quality assessment

16
Steps of a Systematic Review
Search of
Consultation
personal
Computerized
with experts
files
Databases
Review of
Systematic
reference lists
manual searches
of articles
of key journals
Identify studies
Relevant
Not Relevant
Review for
relevance
Evaluate
methodological quality
Reject
Extract data
Analyze data
Draw Conclusions
17
the QUOROM statement
18
Principles of reliable detection of the effects
of health care interventions

• Systematic bias must be lt the effect of
  intervention which we are trying to detect
• The need for the totality of evidence (published
  and unpublished)
• Random errors (play of chance) must be lt the
  effect of intervention which we are trying to
  detect
• uncertainty/imprecision reduced by pooling all
  available data
• Need for large number of patients/events

19
Rationale for (quantitative) synthesis of all
available evidence

• The rationale for pooling data is clinical and
  not statistical
• Similar interventions for similar conditions will
  produce the similar effects (i.e. in the same
  direction)
• While the effect size may not be the same, it
  will rarely be in the opposite directions
• Meta-analysis attempts to show direction of the
  effect (i.e. help establish generalisability of
  the effect)

20
RR (95 CI Fixed)
A)
Disease population
RCT2
RCT3
RCT1
Test for heterogeneity chi square df 2, p0.1
Test for overall effect Z p0.02
0.8
1
10
Relative risk
Favors new treatment
Favors control
21
RR (95 CI Fixed)
B)
Disease population
RCT2
RCT3
RCT1
Test for heterogeneity chi square df 2, p0.02
Test for overall effect Z p0.05
0.8
1
10
Relative risk
Favors new treatment
Favors control
22
Calculate observed minus expected for each trial
Treated
Control
Obs10 Exp12.5
25
Dead
Obs15
o-e -2.5 v 5.5 odds ratio
0.64 Conf.Int. 0.28-1.46 P 0.29
Alive
175
100
100
200
Courtesy of Dr. K. Wheatley
23
Compare only patients in one trial with patients
in the same trial

• Statistics
• Obsd expd Variance
• Trial 1 (o e)1 V1
• Trial 2 (o e)2 V2
• Trial 3 (o e)3 V3
• All Trials (o e)T VT

Courtesy of Dr. K. Wheatley
24
Compare only patients in one trial with patients
in the same trial

• Statistics
• Obsd expd Variance
• Trial 1 - 2.5 5.5
• Trial 2 - 2.5 5.5
• Trial 10 - 2.5 5.5
• All Trials - 25.0
  55.0
• Odds ratio 0.63
• 95 confidence interval 0.49 to 0.83
• Plt0.001

25
Rationale for (quantitative) synthesis of all
available evidence

• Reduction of bias Comparison of alike with alike
• Use of randomized comparison whenever possible
• Always within the same trial
• Pooling is done by adding trials (not patients)
• Reduction of imprecision and uncertainty
• Particularly important when the effects of
  interventions are of small to moderate size (e.g.
  RRR5-10 or 15-25)
• 20 of reduction in a 50 risk of deathavoidance
  of death in 1 in 10 patients

26
Effect of random errors

• Function of the size of the trial
• Subgroup analysis

27
No evidence or no evidence of an effect?

• Absence of evidence of benefit is not
• evidence of absence of benefit
• Truly negative trial (evidence of no effect) vs.
  false-negative trial (no evidence of an effect)

28
Size of randomized trials in myeloma
29
Effect of chancedata-dependent subgroup
analysis vs. indirect extrapolation of overall
analysis

• Data-dependent subgroup analyses may result to
  importantly biased conclusions and should be
  avoided
• Paradoxically, even effects among specific
  categories of patients may be best assessed
  indirectly by approximation of overall treatment
  effect to the patients into a specific category
• As long as the effect the effect in the specific
  subgroup is not qualitatively different from the
  overall effect

30
Real trial (ISIS-2) EXAGGERATEDLY POSITIVE
mortality effect in a subgroup defined only by
astrological birth sign
Astrological birth sign Atenolol effect on day
0-1 mortality in acute myocardial
infarction Mortality reduction
Statistical comparing
Atenlol significance with control
group (2P) Leo (I.e. born beween 71
23 lt0.01 July 24 August 23) 11 other birth
signs Mean 24 Each gt 0.1
(NS) (taken separately) Any birth sign 30
10 lt0.004 (appropriate overall analysis)