Working Group: Can Six Blind Men Find Apples

1
Working GroupCan Six Blind Men FindApples
Oranges? Measuring Variable Implementation of
QI Interventions Using Multiple Data Sources
2
Presenters

• Alexander S. Young, MD, MSHS
• Elizabeth (Becky) Yano, PhD, MSPH
• Lisa V. Rubenstein, MD, MSPH
• Alison Hamilton, PhD

3
Overview

• 90 minutes presentations
• 60 minutes group discussion and breakout groups
• 30 minutes group consensus on priorities,
  suggested next steps, directions
• Working group moves to plenary
• 5 minute summary presented

4
Overview of Working Group

• Presentations
• Introduction and overview (Alex)
• QUITS smoking cessation trial (Becky)
• TIDES depression collaborative care (Lisa)
• EQUIP evidence-based practice in schizophrenia
  (Alison)

5
Goal of Presentations

• Brief orientation to example QI intervention
• How context matters ? sets us up for variable QI
  intervention deployment
• Process for intentional adaptation of evidence
  into context of local practice
• Types of data sources brought to bear on
  measuring implementation
• including development of a fidelity score
• Triangulation of data sources to tell story

6
QI Intervention (QII) Examples

• QUITS (Quality Improvement Trial for Smoking
  cessation)
• evidence-based quality improvement to implement
  smoking cessation guidelines
• Scott Sherman MD Becky Yano PhD (co-PIs)
• TIDES (Translating Interventions for Depression
  into Evidence-based Solutions)
• depression collaborative care model
• Lisa Rubenstein MD Ed Chaney PhD (co-PIs)

7
QI Intervention Examples (contd)

• EQUIP (Enhancing QUality of care In Psychosis)
• evidence-based quality improvement to implement
  effective treatments in schizophrenia
• Alex Young MD Amy Cohen PhD (co-PIs)

8
QUITS Smoking Cessation Trial

• Evidence base
• SC clinic referrals
• Tobacco quitlines
• PC-based intns

Education toolkit
Local QI plan development
EBQI
Expert review/feedback
Performance feedback
local buy-in priority-setting
Leadership support
9
TIDES Depression Collaborative Care

• Evidence base
• gt20 RCTs
• Depression
• toolkit

Provider/patient education
Depression care manager
EBQI
QI Informatics support
Performance feedback
adaptation priority-setting
Leadership support
10
EQUIP Effective Schizophrenia Care

• Evidence base
• TMAP
• EQUIP-1

Provider/patient education
Quality manager
EBQI
QI Informatics support
Performance feedback
infrastructure priority-setting
Leadership support
11
Context Matters Design for It

• TIDES
• 21 intervention-to-control sites x 3 VISNs (6
  intervention 3 control sites total)
• VISN leaders chose sites, we randomized within
  network (block on network characteristics)
• QUITS
• regional concentration in southwest (3 VISNs)
• matched on size/academic affiliation within VISN
• we chose sites and randomized within network
• EQUIP
• 11 intervention-to-control sites x 4 diverse
  VISNs
• sites chosen based on leadership interest

12
Context Matters Input from Sites

• Attitudes / beliefs / experiences
• perceived need for the intervention
• competing demands
• staff openness to innovation
• Resources
• perceived time to use program and participate in
  implementation
• organizational structure, staffing, prior QI
  experience, informatics support

Source Kirchner JE, Parker LE, Yano EM, COVES
evaluation (2007).
13
Multiple Data Sources Measuring Implementation
TIDES QUITS EQUIP Examples
Semi-structured interviewsleaders, clinicians, mgrs ? -- ? participation, level of implementation
Organizational site surveys PC/MH leaders, SC mgrs ? ? ? clinic structure, processes, change
Field journals -- -- ? group-level dynamics, implementation details
Administrative data ? ? ? visits, Rxs, costs
Patient surveys ? ? ? PHQ9, BASIS, quits
Clinician surveys activity logs ? ? ? knowledge, attitudes, behaviors
Practice checklists (experts) -- ? -- QII components
14
Multiple Data SourcesStrengths and Challenges
Strengths Challenges
Semi-structured interviewsleaders, clinicians, mgrs rich data, diverse perspectives expensive, time-consuming
Organizational site surveys PC/MH leaders, SC mgrs site profiles, fast, easier to analyze limited discovery, key informant view
Field journals detailed contextual data variation between observers
Administrative data readily available, historical value not QII-specific, local coding differences
Patient surveys experience, exposure, outcomes expensive, sensitive to sample
Clinician surveys activity logs implementers! difficult responders
Practice checklists (experts) face validity, eyes on the ground variation between observers
15
Triangulation

• Critical to collect information about
  implementation from multiple sources
• be prepared for disagreement
• perspectives and opportunities for observation
  differ for managers, providers vs. patients
• Recognize differences between exposed sample
  and practice population
• does the enrolled group represent the practice?
• did the intervention penetrate among all
  providers?