A brief, birdseye overview of new Cochrane methodologies

1
A brief, birds-eye overview of new Cochrane
methodologies

• Part 1
• Assessing the risk of bias
• GRADE, GRADEpro, and the Summary of
  findings table

The new Cochrane Handbook explains all of this in
detail. If anything in this presentation seems
unclear or is confusing, the Handbook will
clarify it for you. http//www.cochrane-handbook.o
rg
2
WHAT IS HAPPENING?

• You will assess the risk of bias in an included
  study by asking yourself a series of
  questions about it. Your answers will be
  Yes, No, or Unclear.
• You will mark these answers in RevMan.
• You will enter other data in RevMan, as usual.
• RevMan will create a color-coded Risk of Bias
  summary (or graph), which will appear in your
  review.
• The GRADE approach software (GRADEpro) will
  retrieve data from your review
• You will enter some other data, and comments,
  into GRADEpro
• GRADEpro will create a Summary of Findings
  table
• You can import the Summary of Findings table
  into RevMan

3
Assessing the Risk of Bias Key Points

• Numerous tools are available for assessing
  methodological quality of clinical trials.
  Cochrane now explicitly recommends against the
  use of scales yielding a summary score (this
  includes the Jadad scale)
• The Cochrane Collaboration recommends a
  specific tool for assessing risk of bias in
  each included study. This comprises a description
  and a judgment for each entry in a Risk of
  bias table, where each entry addresses a
  specific feature of the study.
• The judgment for each entry involves answering
  a question, with answers Yes indicating
  low risk of bias, No indicating high risk of
  bias, and Unclear indicating either lack of
  information or uncertainty over the
  potential for bias
• Plots of Risk of bias assessments can be
  created in RevMan.

4
Quality or Risk of bias?

• Quality did they do the best they could?
• Bias should I believe the result?
• We never know biases, but there is rationale for
  considering risk of bias
• Key consideration in Cochrane reviews is
  believability risk of bias targets this question
  squarely
• High quality research methods can still leave
  a study at important risk of bias. (e.g. when
  blinding is impossible)
• Some markers of quality in medical research are
  unlikely to have direct implications for risk of
  bias (e.g ethical approval, sample size
  calculation)
• Overcomes ambiguity between quality of reporting
  and the quality of the underlying research
• The new tool principles
• Provides a framework for assessing the whole
  trial
• Explicitly judgmental but separates the facts
  from the judgments
• Transparent, and so repeatable

NB Summary of findings tables use Quality of
evidence to assess something different
5
The new tool items to address
1. Sequence generation (randomization) 2.
Allocation concealment 3. Blinding of
participants, personnel and outcomes 4.
Incomplete outcome data (attrition and
exclusions) 5. Selective outcome reporting 6.
Other (including topic-specific, design-specific)
6
The new tool the questions

• Was the allocation sequence adequately
  generated?
• Was allocation adequately concealed?
• Was knowledge of the allocated intervention
  adequately prevented during the study?
• Were incomplete outcome data adequately
  addressed?
• Are reports of the study free of suggestion of
  selective outcome reporting?
• Was the study apparently free of other problems
  that could put it at a high risk of bias?

7
The new tool how to assess them

• Two components
• 1. Description of what happened
• Possibly including done, probably done,
  probably not done or not done for some
  items
• 2. Review authors judgment
• Whether bias unlikely to be introduced
  through this item (Yes, No, Unclear)
• Yes Low risk of bias
• No High risk of bias
• Blinding and Incomplete outcome data may need
  separate assessments for different outcomes

8
Risk of bias assessment in Cochrane reviews
9
Risk of bias summary
Here Blinding and Incomplete outcomes data
have been assessed for two sets of outcomes
10
Risk of bias graph
11
Summary assessment by outcome
12
What about non-randomized studies?

• For some reviews, the question of interest cannot
  be answered by randomized trials, and review
  authors may be justified in including
  non-randomized studies
• Potential biases are likely to be greater for
  non-randomized studies compared with randomized
  trials, so results should always be interpreted
  with caution when they are included in reviews
  and meta-analyses.
• Particular concerns arise with respect to
  differences between people in different
  intervention groups (selection bias) and studies
  that do not explicitly report having had a
  protocol (reporting bias)

13
Non-randomized, continued

• Cochrane recommends that eligibility criteria,
  data collection and critical assessment of
  included studies place an emphasis on
  specific features of study design (e.g. which
  parts of the study were prospectively
  designed) rather than labels for study designs
  (such as case-control versus cohort)
• Risk of bias in non-randomized studies can be
  assessed in a similar manner to that used for
  randomized trials, although more attention
  must be paid to the possibility of selection
  bias
• Meta-analyses of non-randomized studies must
  consider how potential confounders are
  addressed, and consider the likelihood of
  increased heterogeneity resulting from other
  biases that vary across studies.

14
Bias in non-randomized studies

• Bias may be present in findings from
  non-randomized studies in many of the same ways
  as in poorly designed or conducted randomized
  trials.
• For example, numbers of exclusions in
  non-randomized studies are frequently unclear,
  intervention and outcome assessment are often not
  conducted according to standardized protocols,
  and outcomes may not be assessed blind. The
  biases caused by these problems are likely to be
  similar to those that occur in randomized trials.
  
• In non-randomized studies, use of allocation
  mechanisms other than concealed randomization
  means that groups are unlikely to be comparable.
  These potential systematic differences between
  characteristics of participants in different
  intervention groups are likely to be the issue
  of key concern in most non-randomized studies.
• Statistical methods are sometimes used to counter
  bias introduced from confounding by producing
  adjusted estimates of intervention effects, and
  part of the assessment of study quality may
  involve making judgments about the
  appropriateness of the analysis as well as the
  design and execution of the study.
• The variety of study designs classified as
  non-randomized, and their varying susceptibility
  to different biases, makes it difficult to
  produce a generic robust tool that can be used to
  evaluate risk of bias. Within a review that
  includes non-randomized studies of different
  designs, several tools for assessment of risk of
  bias may need to be created.

15
GRADE What is it?

• The Grades of Recommendation, Assessment,
  Development and Evaluation Working Group (GRADE
  Working Group) has developed a system for grading
  the quality of evidence.
• Over 20 organizations including the World Health
  Organization (WHO), BMJ Clinical Evidence,
  National Institutes of Health and Clinical
  Excellence (NICE) in the UK, many others, have
  adopted the GRADE system in its original format
  or with minor modifications.
• The BMJ encourages authors of clinical guidelines
  to use the GRADE system.
• The Cochrane Collaboration has adopted the
  principles of the GRADE system for evaluating the
  quality of evidence for outcomes reported in
  systematic reviews.

16
GRADE

• For purposes of systematic reviews, the GRADE
  approach defines the quality of a body of
  evidence as the extent to which one can be
  confident that an estimate of effect or
  association is close to the quantity of specific
  interest.
• Quality of a body of evidence involves
  consideration of within-study risk of bias
  (methodological quality), directness of evidence,
  heterogeneity, precision of effect estimates and
  risk of publication bias.
• The GRADE system entails an assessment of the
  quality of a body of evidence for each individual
  outcome.

17
GRADE

• The GRADE approach specifies four levels of
  quality. The highest quality rating is for
  randomized trial evidence.
• Review authors can, however, downgrade randomized
  trial evidence to moderate, low, or even very low
  quality evidence, depending on the presence of
  five factors.
• Usually, quality rating will fall by one level
  for each factor, up to a maximum of three levels.
  
• If there are very severe problems for any one
  factor (e.g. when assessing limitations in design
  and implementation, all studies were unconcealed,
  unblinded, and lost over 50 of their patients to
  follow-up), randomized trial evidence may fall by
  two levels due to that factor alone.
• They can also upgrade evidence from other kinds
  of studies, depending on the presence of three
  factors.

18
GRADE

• Five factors that may decrease the quality level
  of a body of evidence
• Limitations in the design and implementation of
  available studies suggesting high likelihood of
  bias.
• Indirectness of evidence (indirect population,
  intervention, control, outcomes).
• Unexplained heterogeneity or inconsistency of
  results (including problems with subgroup
  analyses).
• Imprecision of results (wide confidence
  intervals).
• High probability of publication bias.

19
GRADE

• Three factors that may increase the quality level
  of a body of evidence
• Large magnitude of effect.
• All plausible confounding would reduce a
  demonstrated effect or suggest a spurious effect
  when results show no effect.
• Dose-response gradient.

20
Good news

• An additional piece of software, GRADEprofiler
  (GRADEpro), is available to assist Cochrane
  review authors in the preparation of Summary of
  findings tables.
• GRADEpro is able to retrieve data from RevMan and
  to combine this with user-entered control group
  risks to produce the relative effects and
  absolute risks associated with interventions.
• It performs many of the calculations necessary
  to present the key results of systematic
  reviews in a table format, and guides users
  through the process of grading the quality of the
  evidence using the GRADE approach.
• Before very long, GRADEpro will actually be
  incorporated into RevMan.

21
Cochrane Summary of findings tables include the
following six elements, using a fixed format

• A list of all important outcomes, both desirable
  and undesirable (up to seven outcomes)
• A measure of the typical burden of these outcomes
  (e.g. illustrative risk, or illustrative mean, on
  control intervention)
• Absolute and relative magnitude of effect (if
  both are appropriate)
• Numbers of participants and studies addressing
  these outcomes
• A rating of the overall quality of evidence for
  each outcome (which may vary by outcome) and
• Space for comments.

22
Summaryoffindingstable
23

• Title
• Question (PICO)

24
Important outcomes (up to 7)
25
Results Number of Participants/studies
26
Results Relative effect
27
Results Baseline risks (Assumed Risk)
28
Results Risk with intervention (Corresponding
Risk)
29
Results Dichotomous outcomes
30
Results Continuous outcomes
31
Results Outcomes not reported/not
measured/ not pooled
32
Comments
33
Quality of the evidence for each outcome
34
THE ENDfor now. Please be sure to consult the
newest version of the Cochrane Handbook, which
will clarify all these things and
more. http//www.cochrane-handbook.org