Selecting Evidence for Comparative Effectiveness Reviews: When to use Observational Studies

1
Selecting Evidence for Comparative Effectiveness
ReviewsWhen to use Observational Studies

• Dan Jonas, MD, MPH
• Meera Viswanathan, PhD
• Karen Crotty, PhD, MPH
• RTI-UNC Evidence-based Practice Center

2
Sources

• AHRQ Methods Guide, Chapters 4 and 8,
  http//www.effectivehealthcare.ahrq.gov/repFiles/2
  007_10DraftMethodsGuide.pdf
• Draft manuscript, Norris et al., Observational
  Studies in Systematic Reviews of Comparative
  Effectiveness.
• Chou R, Aronson N, Atkins D, et al. Assessing
  harms when comparing medical interventions AHRQ
  and the Effective Health Care Program. J Clin
  Epidemiol 2008 Sep 25.

3
Overview

• Why should reviewers consider including
  observational studies (OS) in comparative
  effectiveness reviews (CERs)?
• When should OS be included in CERs?
• What are the differences in considering inclusion
  of OS for benefits as opposed to OS of harms?

4
Current Perspective

• CERs should consider including observational
  studies
• this should be the default strategy
• Reviewers should explicitly state the rationale
  for including or excluding OS

5
Comparative Effectiveness Reviews (CERs)

• Systematic reviews that compare the relative
  benefits and harms among a range of available
  treatments or interventions for a given condition

6
CER Process Overview
7
Hierarchy of Evidence
8
Danger of Over-reliance on RCTs

• May be unnecessary, inappropriate, inadequate, or
  impractical
• May be too short in duration
• May report intermediate outcomes rather than main
  health outcomes of interest
• Often not available for vulnerable populations
• Generally report efficacy rather than
  effectiveness
• AHRQ Evidence-based Practice Centers include wide
  variety of study designs (not only RCTs)

9
Observational Studies (OS)

• Definition Studies where the investigators did
  not assign the exposure/intervention
• i.e. non-experimental studies
• Controlled clinical trials are quasi-experimental
  studies, not OS
• We present considerations for including OS to
  assess benefits and to assess harms separately

10
OS to Assess Benefits

• Often insufficient evidence from trials to answer
  all KQs in CERs (think PICOTS)
• Population may not be available for
  sub-populations and vulnerable populations
• Interventions may not be able to assign
  high-risk interventions randomly
• Outcomes may report intermediate outcomes rather
  than main health outcomes of interest
• Timing may be too short in duration
• Setting may not represent typical practice

11
Group Exercise

• What should reviewers consider when deciding
  whether or not to include observational studies
  in CERs?

12
OS to Assess Benefits

• Reviewers should consider 2 questions
• Are there gaps in trial evidence for the review
  questions under consideration?
• Will observational studies provide valid and
  useful information to address key questions?

13
Are there gaps in trial evidence? Will OS provide
valid and useful information?
14
Group Exercise Include OS?

1. CER of PCI vs. CABG for coronary disease
   identified 23 RCTs. Experts (TEP) raised
   concerns that the studies enrolled patients with
   a relatively narrow spectrum of disease relative
   to those having the procedures in current
   practice
2. Review of antioxidant supplementation to prevent
   heart disease found numerous large clinical
   trials, including over 20,000 elevated-risk
   subjects in the Heart Protection Study. No
   beneficial effects were seen in CV outcomes,
   including mortality. Findings were consistent
   across trials with varying populations, sizes,
   etc.

15
Group Exercise include OS?

• CER of PCI vs. CABG----Need to look for OS
• OS from 10 large cardiovascular registries were
  identified
• These confirmed that the use of the procedures in
  the community included patients with wider
  variation in disease
• For patients similar to those enrolled in trials,
  mortality results in the registries were similar
  to trials (no difference between interventions)
• Relative benefits of the procedures varied
  markedly with extent of disease, raising caution
  about extending trial conclusions to patients
  with greater or lesser disease than those in
  trial populations
• Review of antioxidant supplementation to prevent
  heart disease----Trial data are sufficient

16
Gaps in Trial Evidence PICOTS

• Trial data may be insufficient for a number of
  reasons
• PICOTS
• Populations included (missing certain groups)
• Interventions included
• Outcomes reported (only intermediate)
• Duration
• All trials may be efficacy studies

17
Are Trial Data Sufficient? PICOTS and Beyond

• Risk of bias (internal validity)
• Degree to which the findings may be attributed to
  factors other than the intervention under review
• Consistency
• Extent to which effect size and direction vary
  within and across studies
• Inconsistency may be due to heterogeneity across
  PICOTS
• Directness
• Degree to which outcomes that are important to
  users of the CER (patients, clinicians, or
  policymakers) are encompassed by trial data
• Health outcomes generally most important

18
Are Trial Data Sufficient? PICOTS and Beyond

• Precision
• Includes sample size, number of studies, and
  heterogeneity within or across studies
• Reporting bias
• Extent to which trial authors appear to have
  reported all outcomes examined
• Applicability
• Extent to which the trial data are likely to be
  applicable to populations, interventions, and
  settings of interest to the user
• The review questions should reflect the PICOTS
  characteristics of interest

19
When to Identify Gaps in Trial Evidence

• Identification of gaps in trial evidence
  available to answer review questions can occur at
  a number of points in the review
• When first scoping the review
• Consultation with Technical Expert Panel
• Initial review of titles and abstracts
• After detailed review of trial data

20
CER Process Overview
21
Gaps in Trial Evidence

• Operationally, may perform initial searches
  broadly, to identify both OS and trials, or may
  do searches sequentially and search for OS after
  reviewing trials in detail to identify gaps in
  evidence

22
2. Will observational studies provide valid and
useful information to address key questions?

• Reviewers should
• Refocus the study question on gaps in trial
  evidence
• specify the PICOTS characteristics for gaps in
  trial evidence
• Assess whether available OS may address the
  review questions (applicable to PICOTS?)
• Assess suitability of OS to answer the review
  questions

23
Valid and Useful Information

• Assess suitability of OS to answer the review
  questions
• After gaps have been identified in trial
  literature and that OS potentially fill those
  gaps
• Consider the clinical context and natural history
  of the condition under study
• Assess how potential biases may influence the
  results of OS

24
Clinical context

• Fluctuating or intermittent conditions are more
  difficult to assess with OS
• Especially if there is no comparison group
• OS may be more useful for conditions with steady
  progression or decline

25
Group Exercise

• Here are two very different conditions
• Acute low back pain
• Amyotrophic lateral sclerosis (ALS)
• How might the differences in these conditions
  impact whether OS would provide useful
  information?

26
Group Exercise

• Main considerations here are the natural history
  of the condition under study
• People with acute low back pain often recover
  spontaneously
• A cohort study of treatments for acute low back
  pain cant establish, with any degree of
  certainty, whether the treatments affected
  patient outcomes
• ALS has a course of steady decline
• An uncontrolled cohort study of treatments for
  ALS may well be able to demonstrate meaningful
  effects

27
Potential biases

• Selection bias (and confounding by indication)
• Performance bias
• Detection bias
• Attrition bias

28
Group Exercise

• Suppose youre conducting a CER of medications
  for rheumatoid arthritis (RA)
• You find several retrospective analyses of
  administrative databases comparing outcomes of RA
  patients taking etanercept vs. methotrexate
• Suppose that etanercept is restricted in many of
  the health systems to patients with more severe
  RA who have failed other treatments
• Should you include these OS?
• What considerations will influence your decision?

29
Group Exercise

• Confounding by indication
• A type of selection bias
• When different diagnoses, severity of illness, or
  comorbid conditions are important reasons for
  physicians to assign different treatments
• Common problem in pharmacoepidemiology studies
  comparing beneficial effects of interventions
• Generally would not include this information due
  to a high risk of bias (poor internal validity),
  unless studies had a good way to adjust for
  severity of disease

30
Harms

• Assessing harms can be difficult
• Trials often focus on benefits, with little
  effort to balance assessment of benefits and
  harms
• OS are almost always necessary to assess harms
  adequately
• There are tradeoffs between increasing
  comprehensiveness of reviewing all possible harms
  data and decreasing quality (increasing risk of
  bias) for harms data

31
Trials to Assess Harms

• Randomized controlled trials gold standard for
  evaluating efficacy
• But, relying solely on RCTs to evaluate harms in
  CERs is problematic
• Most lack prespecified hypotheses for harms as
  they are designed to evaluate benefits
• Assessment of harms is often a secondary
  consideration
• Quality and quantity of reporting of harms is
  frequently inadequate
• Few have sufficient sample sizes or duration to
  adequately assess uncommon or long-term harms

32
Trials to Assess Harms

• Most RCTs are efficacy trials
• they assess benefits and harms in ideal,
  homogenous populations and settings
• patients who are more susceptible to harms are
  often under-represented
• Few RCTs directly compare alternative treatment
  strategies
• Publication bias and selective outcome reporting
  bias
• RCTs may not be available

33
Trials to Assess Harms

• Nevertheless, head-to-head RCTs provide the most
  direct evidence on comparative harms
• In addition, placebo-controlled RCTs can provide
  important information
• In general, CERs should routinely include both
  head-to-head and placebo-controlled trials for
  assessment of harms
• In lieu of placebo-controlled RCTs, CERs may
  incorporate findings of well-conducted systematic
  reviews if they evaluated the specific harms of
  interest

34
Unpublished Supplemental Trials Data

• Consider including results of completed or
  terminated unpublished RCTs and unpublished
  results from published trials
• FDA website, http//www.ClinicalTrials.gov, etc.
• Must contemplate ability to fully assess risk of
  bias
• When significant of published trials fails to
  report an important AE, CER authors should report
  this gap in the evidence and consider efforts to
  obtain unpublished data

35
OS to Assess Harms

• OS are almost always necessary to assess harms
  adequately
• Exception is when there are sufficient data from
  RCTs to reliably estimate harms
• May provide best or only data for assessing harms
  in minority or vulnerable populations who are
  under-represented in trials
• Types of OS included in a CER will vary
  different types of OS might be included or
  rendered irrelevant by availability of data from
  stronger study types

36
Hypothesis Testing vs. Hypothesis Generating

• Important consideration in determining which OS
  to include
• Case reports are hypothesis generating
• Cohort and case-control studies are well suited
  for testing hypotheses of whether one
  intervention is associated with a greater risk
  for an adverse event than another and for
  quantifying the risk

Chou et al, JCE 2008
37
Hierarchy of Evidence
38
OS to Assess Harms

• Cohort and case-control studies
• CERs should routinely search for and include,
  except when RCT data are sufficient and valid
• OS based on patient registries
• OS based on analyses of large databases
• Case reports and post-marketing surveillance
• New medications
• Other OS

39
OS to Assess Harms

• Criteria to select OS for inclusion
• there are often many more OS than trials
  evaluating a large number of OS can be
  impractical when conducting a CER
• Several criteria commonly uses in CERs to screen
  OS for inclusion (empirical data lacking)
• Minimum duration of follow-up
• Minimum sample size
• Defined threshold for risk of bias
• Study design (cohort and case-control)
• Specific population of interest

40
Key Take-home Points

• Often insufficient evidence from trials to answer
  all Key Questions in CERs
• CERs should consider including OS default
  strategy
• Should explicitly state the rationale for
  including or excluding OS
• For OS to assess benefits, reviewers should
  consider 2 questions
• Are there gaps in trial evidence for the review
  questions under consideration?
• Will observational studies provide valid and
  useful information to address key questions?
• For harms, should routinely search for and
  include cohort and case-control studies