Discussion opener Iain Chalmers

1
Iain Chalmers and Paul Glasziou
2
Overview

• Waste occurs at 4 stages of research question
  design publication the report
• About 50 loss at last 3 stages
• Implies 85 of 100Billion spent on research
  each year is wasted

3
The personal impact of non-publication
4
The 4 stages from question to report
Questions relevant to clinicians patients?
Accessible full publication?
Unbiased and usable report?
Appropriate design and methods?
Low priority questions addressed Important
outcomes not assessed Clinicians and patients
not involved in setting research agendas
Over 50 of studies never published in
full Biased under-reporting of studies with
disappointing results
Over 30 of trial interventions not sufficiently
described Over 50 of planned study outcomes
not reported Most new research not interpreted
in the context of systematic assessment of other
relevant evidence
Over 50 studies designed without reference to
systematic reviews of existing evidence Over
50 of studies fail to take adequate steps to
reduce biases, e.g. unconcealed treatment
allocation
Research waste
5
Stage 1 study questions
Questions relevant to clinicians patients?
Low priority questions studied Important
outcomes not assessed Clinicians and patients
not involved in setting research agendas
Research waste
6
Research priorities among patients with
osteoarthritis of the knee compared with
researchers priorities (Tallon et al. 2000).
7
Survey of patients with rheumatoid arthritis
priority treatment outcome

• What is patients most important problem?
• It was not pain
• It was fatigue!

8
Stage 2 study design
Appropriate design and methods?
Over 50 studies designed without reference to
systematic reviews of existing evidence Over
50 studies fail to take adequate steps to reduce
biases, e.g. unconcealed treatment allocation

• New studies
• Ignore previous studies
• Have avoidable design flaws

Research waste
9
Only 11 of 24 responding authors of trial reports
that had been added to existing systematic
reviews were even aware of the relevant reviews
when they designed their new studies.
10
New trials of aprotinin ignored previous trials
11
Cumulative estimate of the effect of aprotinin on
perioperative blood transfusion, 1987-2002.
12
Avoidable design flaws are common
BMJ 2005330 1057-8.
13
Stage 3 publication
Accessible full publication?
Over 50 of studies never published in
full Biased under-reporting of studies with
disappointing results
Research waste
14
Publication bias and rates
Hopewell S, et al. CDSR 2009
15
About half of trials are unpublished

• Less than half of all studies, and about 60 of
  randomized or controlled clinical trials,
  initially presented as summaries or abstracts at
  professional meetings are subsequently published
  as peer-reviewed journal articles. Scherer RW,
  Langenberg P, von Elm E. Full publication of
  results initially presented in abstracts.
  Cochrane Database of Systematic Reviews 2007,
  Issue 2.

16
Studies that report positive or significant
results are more likely to be published and
outcomes that are statistically significant have
higher odds of being fully reported.
PLoS ONE, August 20083e3081
17
Publication bias can kill!Cowley et al. 1993
When we carried out our study in 1980 we thought
that the increased death rate that occurred in
the drug group was an effect of chanceThe
development of the drug was abandoned for
commercial reasons, and this study was therefore
never published it is now a good example of
publication bias. The results described
heremight have provided an early warning of
trouble ahead.
18
At the peak of their use in the late 1980s, it
has been estimated that these drugs killed as
many Americans every year as were killed during
the whole of the Vietnam war. Moore 1995.
19
Stage 4 Useable report
Unbiased and usable report?
Over 30 of interventions not sufficiently
described Over 50 of planned study outcomes not
reported Most new research not interpreted in
the context of systematic reviews of other
relevant evidence
Research waste
20
What is the treatment?

• The papers description of sodium reduction
• "Individual and weekly group counseling sessions
  were offered initially, with less intensive
  counseling and support thereafter, specific to
  sodium reduction."

TOHP Study BMJ, Apr 2007 334 885
21
What is sodium reduction?

• The papers description
• "Individual and weekly group counseling sessions
  were offered initially, with less intensive
  counseling and support thereafter, specific to
  sodium reduction."
• Previous reference
• (i) an individual session followed by 10 weekly
  group 90 minute sessions with a nutritionist,
  followed by a transitional stage of some
  additional sessions
• (ii) Topics in the weekly sessions included
  Getting Started, sodium basics, the morning meal,
  midday sources of sodium, the main meal, planning
  ahead, creative cooking, eating out, food cues,
  and social support,
• (iii) the sessions included sampling of foods,
  discussion of articles on sodium reduction, and
  problem-solving,
• (iv) patients kept diaries at least 6 days per
  week, and urine sodiums were measured.

22
Is the inadequate description fixable?
Glasziou P, et al. BMJ 20083361472-74
23
Systematic review what specific regimen?

• STUDY meta-analysis of behavioural interventions
  for insomnia adults
• .. confirms the efficacy of behavioral
  interventions for person with chronic insomnia.
• PROBLEM No regimens for behavioural
  intervention described
• Author asked what specific treatment regime (or
  regimes) would you recommend based on your
  review?
• Author response It was found that cognitive,
  behavioral and relaxation therapies all in
  general lead to similar improvements in sleep
  outcomes---although cognitive approaches might
  have been a bit better. The references for these
  studies are found in the article.

Rx Behavioural Intervention
24
So what can we do?

• Training
• Standards (CONSORT)
• Non-pharmacopeia

25
Repository of intervention descriptions is needed

• A Handbook of Non-Drug Interventions

26
The 4 stages from question to report
Questions relevant to clinicians patients?
Accessible full publication?
Unbiased and usable report?
Appropriate design and methods?
Low priority questions addressed Important
outcomes not assessed Clinicians and patients
not involved in setting research agendas
Over 50 of studies never published in
full Biased under-reporting of studies with
disappointing results
Over 30 of trial interventions not sufficiently
described Over 50 of planned study outcomes
not reported Most new research not interpreted
in the context of systematic assessment of other
relevant evidence
Over 50 studies designed without reference to
systematic reviews of existing evidence Over
50 of studies fail to take adequate steps to
reduce biases, e.g. unconcealed treatment
allocation
50
50
50
85 Research waste over 85 Billion / year
27
Summary

• Waste at 4 stages of research question design
  publication report
• About 50 loss at last 3 stages
• Implies 85 of 100Billion spent on research each
  year is wasted

28
(No Transcript)
29
Database of Uncertainties about the Effects of
TreatmentsDUETs www.library.nhs.uk/duetsAn
initiative of the National Therapeutic Ignorance
Service
30
Sources of uncertainties about the effects of
treatments
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
Classification of Discussion sections in RCT
reports published in May issues of Ann Int Med,
BMJ, JAMA, Lancet, and N Eng J Med
36
Classification of Discussion sections in RCT
reports published in May issues of Ann Int Med,
BMJ, JAMA, Lancet, and N Eng J Med
37
Classification of Discussion sections in RCT
reports published in May issues of Ann Int Med,
BMJ, JAMA, Lancet, and N Eng J Med
38
(No Transcript)
39
(No Transcript)
40
An illustration of how researchers and the
system can serve patients and clinicians better
41
What are the effects of systemic corticosteroids
in acute traumatic brain injury?
42
A systematic review of existing knowledge
Alderson P, Roberts I (1997). BMJ 3141855-9
and Cochrane Database of Systematic Reviews.
The review revealed important uncertainty about
whether systemic steroids did more good than
harm.
43
(No Transcript)
44
Addressing an important uncertaintyBecause the
systematic review and a survey of clinical
practice had revealed important uncertainty, a
large, publicly-funded, multicentre randomized
trial was organised to address the
uncertainty.The trial was registered
prospectively on the ISRCTN register, The
protocol for the trial was published in BioMed
Central.
45
Lancet 20043641321-28
46
(No Transcript)
47

• The report of the CRASH trial is exemplary
  because
• it refers to current uncertainty about the
  effects of a treatment, manifested in a
  systematic review of all the existing evidence,
  and in variations in clinical practice
• It notes that the trial was registered and the
  protocol published prospectively
• it sets the new results in the context of an
  updated systematic review of all of the existing
  evidence
• it provides readers with all the evidence needed
  for action to prevent thousands of iatrogenic
  deaths

48
(No Transcript)
49
The problemThe real information needs of
clinicians and patients
50
Working on the Problem
51
Cognitive Style

• Paul
  Iain

52
Mapping the research-practice gap
Published Studies (primary research
studies sound unsound)
Systems (bottomline /- ref) Synopses (user
summary of research) Systematic Reviews
CATs (search appraise synthesis)
Research Questions Funding
53
Austin Bradford Hill, 1965Four questions to
which readers want answers when reading reports
of research. 1. Why did you start? 2. What
did you do? 3. What answer did you get? 4. And
what does it mean anyway?
54
Austin Bradford Hill, 1965Four questions to
which readers want answers when reading reports
of research. 1. Why did you start? 2. What
did you do? 3. What answer did you get?
4. And what does it mean anyway?
55
(No Transcript)
56
(No Transcript)