Conducting a Sound Systematic Review: Balancing Resources with Quality Control

1
Conducting a Sound Systematic Review Balancing
Resources with Quality Control

• Eric B. Bass, MD, MPH
• Johns Hopkins University
• Evidence-based Practice Center

2
Learning Objectives

• Identify steps in a systematic review that have
  greatest implications for resource needs
• Explain how those steps are vulnerable to
  error/bias
• Identify strategies for quality control when
  resources are limited

3
Steps in the systematic review
Establish eligibility criteria for each question
Search literature
Review citations
Extract data
Evaluate study quality applicability
Summarize synthesize evidence
4
Focus the Question(s)

• Formulate in specific measurable terms
• Clarify priorities
• Be realistic about time effort
• Negotiate with sponsor

5
Set Eligibility Criteria

• Study design
• RCT only?
• Comparison group?
• Any observational study?
• Study population
• Study setting
• Size of study
• Year of publication
• Peer-reviewed?
• Language

6
Search Literature

• Select highest yield sources
• MEDLINE
• EMBASE
• Cochrane database of reviews trials
• Consider other sources
• Use sampling to estimate incremental yield
• Use hand searches for quality control
• Citations in eligible articles
• Table of contents of relevant journals
• Query experts

7
Review Citations

• Titles
• Many citations obviously not relevant can be
  excluded quickly
• Abstracts
• Usually sufficient to determine eligibility of
  citations
• Full text of articles
• For some studies, eligibility can only be
  determined by reading full text

8
Review Citations An Example
Title review (n10,475)
Title review 6,863 excluded
Abstract review (n3,612)
Abstract review 3,163 excluded
Full-text review (n449)
Full-text review 386 excluded
63 included articles
9
Review Citations Quality Control

• Use independent dual reviewers
• Assess samples early
• Discuss discrepancies

10
Extract Data

• Set priorities for data extraction
• Anticipate content of final evidence tables
• Resist temptation to extract everything
• Choose data management method
• Systematic review software vs. Access vs. tables
• Consider complexity, consistency, completeness of
  data
• Establish quality control
• Dual review vs. solo review with random checks
• Independent vs. sequential dual review
• Discuss discrepancies

11
Evaluate Study Quality Applicability

• Decide how evaluation will be used
• To determine eligibility of studies
• To give more or less weight to studies
• To improve future research
• Focus on most important aspects of quality
  applicability
• Use established instrument(s)
• Jadad criteria for RCTs
• Establish quality control
• Independent dual review
• Decide how to reconcile discrepancies

12
Summarize Synthesize Evidence

• Assemble evidence tables
• Reassess priorities
• Exclude duplicative data
• Prepare summary of evidence on each question
• Assess strength of evidence
• Use 2 or more reviewers
• Decide how to reconcile differences

13
Summary

• To conduct a sound systematic review with limited
  resources
• Pay attention to priorities when defining
  questions, eligibility criteria elements of
  evidence tables
• Establish quality control measures for each step
  that could introduce error or bias
• reviewing citations
• extracting data
• evaluating quality, applicability overall
  strength of evidence