Kaiser Permanente National Guideline Program Implications of IOM SR Standards

1
Kaiser PermanenteNational Guideline
ProgramImplications of IOM SR Standards

• Wiley Chan, MD
• Physician, Internal Medicine
• Methodologist, KP National Guideline Program

2
Kaiser Permanente Largest Non-Profit Health Care
Program in the United States

• Founded in 1945
• 8 regions in 9 states and District of Columbia
• 8.6 million members (as of 12/09)
• 15,129 physicians (as of 12/09)
• 164,098 employees (as of 12/09)
• KP Care Management Institute (CMI)
• KP National Guideline Program (NGP)

3
Kaiser Permanente Evidence Network

• 15 KP National Evidence-Based Guidelines
• Supported by full Systematic Reviews
• 3.5 Staff FTEs dedicated to KP NGP work
• 1 Principal Consultant , 2 Analysts, 2 Project
  Managers
• 1.5 Physician FTEs dedicated to KP NGP work
• 7 Physician EBM Methodologists
• External vendor Doctor Evidence
• Search Data Extraction
• Technology platform
• Data repository, analysis documentation

4
Kaiser Permanente National Guideline Program
Process Methodology
New Clinical Issue
ScheduledUpdate

• Key to Abbreviations
• CQ Clinical Question
• SR Systematic Review
• GDT Guideline Development Team
• GQ Guideline Quality Committee
• NGD KP National Guideline Directors

Clinical Questions (CQ)

• Overview

Existing Guideline?
No
Yes
CQ Match?
Existing SR by CQ?
Evidence Search
Abstract Article Review
Inclusion/ Exclusion
Data Extraction
Critical Appraisal
No
No
Yes
Yes
Assess Guideline
Assess SR
Qualitative/ Quantitative Synthesis
Assess Evidence Quality
Evidence Summary
Rationale
Recommendations
Acceptable?
Acceptable?
No
No
Assess Implementability of Recommendations
GDT Approval
Internal Review
GQ NGD Approval
Yes
Yes
New Relevant Studies?
National Guideline
Yes
No
Implementation
5
Kaiser Permanente National Guideline Program
Process Methodology
New Clinical Issue
ScheduledUpdate
Clinical Questions (CQ)

• Acceptable External Guideline
• ADAPTE/AGREE

Existing Guideline?
Yes
CQ Match?
Yes
Assess Guideline
Acceptable?
GDT Approval
Internal Review
GQ NGD Approval
Yes
National Guideline
Implementation
6
Kaiser Permanente National Guideline Program
Process Methodology
New Clinical Issue
ScheduledUpdate
Clinical Questions (CQ)

• No Acceptable External Guideline
• Acceptable External Systematic Review
• No New Relevant Studies
• AMSTAR
• GRADE

Existing Guideline?
No
Yes
CQ Match?
Existing SR by CQ?
No
Yes
Yes
Assess Guideline
Assess SR
Evidence Summary
Rationale
Recommendations
Acceptable?
Acceptable?
No
Assess Implementability of Recommendations
GDT Approval
Internal Review
GQ NGD Approval
Yes
New Relevant Studies?
National Guideline
No
Implementation
7
Kaiser Permanente National Guideline Program
Process Methodology
New Clinical Issue
ScheduledUpdate
Clinical Questions (CQ)

• No Acceptable External Guideline
• No Acceptable External Systematic Review
• Internal Systematic Review
• GRADE

Existing Guideline?
No
Yes
CQ Match?
Existing SR by CQ?
Evidence Search
Abstract Article Review
Inclusion/ Exclusion
Data Extraction
Critical Appraisal
No
No
Yes
Yes
Assess Guideline
Assess SR
Qualitative/ Quantitative Synthesis
Assess Evidence Quality
Evidence Summary
Rationale
Recommendations
Acceptable?
Acceptable?
No
No
Assess Implementability of Recommendations
GDT Approval
Internal Review
GQ NGD Approval
National Guideline
Implementation
8
Kaiser Permanente Systematic ReviewProcesses,
Responsibilities, Tools

• Clinical Questions
• KP Population, Intervention, Comparison, Outcome
• Timing, Setting
• Evidence Search
• Doctor Evidence with KP input
• Masters-trained Medical Librarians (MLIS)
• Abstract/Article Review Inclusion/Exclusion
• Doctor Evidence
• Dual Inclusion/Exclusion with 3rd party
  adjudication
• KP Evaluate exclusions

9
Kaiser Permanente Systematic ReviewProcesses,
Responsibilities, Tools

• Data Extraction
• Doctor Evidence
• Dual extraction with adjudication 3rd party
  independent QA
• Masters-trained graduate student data
  extractors
• Contact authors to clarify discrepancies in
  published data
• KP Quality Assurance
• Critical Appraisal
• KP Cochrane Risk of Bias, Doctor Evidence
  platform
• Qualitative/Quantitative Analysis
• KP Doctor Evidence platform

10
Doctor Evidence Meta-Analysis Page
11
Doctor Evidence Meta-Analysis Results
12
Doctor Evidence Meta-Analysis Study Selection
Detailed study- and outcome-specific descriptions
of Cochrane Risk of Bias are exposed by hovering
Select and de-select studies and Recalculate
13
Doctor Evidence Study Summary
Detailed study-specific descriptions of
characteristics and outcomes are exposed by
hovering
14
Kaiser Permanente Systematic ReviewProcesses,
Responsibilities, Tools

• Evidence Grading
• KP GRADE, GRADEPro
• Developing functionality in Doctor Evidence
  platform
• Evidence Summary (SR)
• Rationale
• Recommendations Guideline
• KP GRADE, GRADEPro, Word
• Developing functionality in Doctor Evidence
  platform

15
Kaiser Permanente National Guideline Program
Process Methodology

• Guideline Implementation

National Guideline
Disease Management Accreditation (NCQA)
Clinician Staff Education Tools (Clinical
Library)
Patient Education Tools (KP.ORG, National
Patient Instructions
Regions (EHR, CDS, Operations)

• Key to Abbreviations
• EHR Electronic Health Record
• CDS Clinical Decision Support
• KP.ORG KP publicly-accessible Internet site
• NCQA National Committee for Quality Assurance

16
IOM Systematic Review StandardsProblem Areas for
KP

• 2.6 Develop a systematic review protocol
• 2.7 Submit the protocol for peer review
• 2.7.1 Provide a public comment period for the
  protocol
• 2.8 Make the final protocol publicly available
• 5.2.1 Use a 3rd party to manage peer review
• 5.2.2 Provide a public comment period for the
  report
• 5.3 Publish the final report with free public
  access

17
IOM Systematic Review StandardsChallenges

• Very resource-intensive
• Infinite needs - Finite resources
• Balance between efficiency versus rigor
• Investment in developing expertise
  infrastructure
• Migration of existing SRs CPGs to new methods
• Updating SRs CPGs
• Every 2 years?
• Dynamic updating, based on evidence impact?
• External review
• Patient public involvement in SRs and CPGs

18
IOM Systematic Review StandardsOpportunities

• Improved SR CPG rigor transparency
• Common evidence-grading recommendation strength
  language
• Collaborate with other SR CPG developers
• Prioritize coordinate topics
• Avoid duplication
• Target funding
• Share data extractions?
• Devote more energy to CPG implementation
• Collaborate with performance metric developers

19
Implications of IOM SR Standards
Wiley.Chan_at_kp.org
20
Supplemental Slides
21
IOM Systematic Review StandardsCongruent Areas
for KP

• 2.1 Establish a team with appropriate expertise
• 2.2 Manage bias conflict of interest
• 2.3 Ensure user stakeholder input
• 2.4 Manage bias conflict of interest for
  individuals providing Input
• 2.5 Formulate the topic
• 3.1 Conduct a comprehensive systematic search

22
IOM Systematic Review StandardsCongruent Areas
for KP

• 3.2 Address potentially biased reporting of
  research results
• 3.3 Screen select studies
• 3.4 Document the search
• 3.5 Manage data collection
• 3.6 Critically appraise each study
• 5.1 Prepare final report using a structured
  format
• 5.2 Peer review the draft report

23
IOM Systematic Review StandardsCongruent Areas
for KP

• 4.1 Use a prespecified method to evaluate the
  body of evidence
• 4.2 Conduct a qualitative synthesis
• 4.3 Decide if the SR will include a quantitative
  analysis
• 4.4.1 Use expert methodologists
• 4.4.2 Address heterogeneity
• 4.4.3 Measures of statistical uncertainty
• 4.4.4 Sensitivity analysis

24
Doctor Evidence Evidence Table Qizard