When Is Quality Improvement Research

1
When Is Quality Improvement Research?

• Christopher J. Crnich, MD MS
• November 15, 2008

2
Outline

• Hierarchy of evidence
• Misconceptions about the hierarchy
• Evidence of QI research
• Acute care
• Long-term care
• Potential limitations of QI research
• Improving methodological quality of QI research

3
A Short History of Medicine

• 2000 BC Here, eat this root.
• 1000 AD That root is heathen. Here, say this
  prayer.
• 1850 AD The prayer is superstition. Here, drink
  this potion.
• 1920 AD That potion is snake oil. Here, swallow
  this pill.
• 1945 AD That pill is ineffective. Here, take
  this penicillin.
• 1955 AD Oops bugs mutated. Here, take this
  tetracycline.
• 1960-1999 AD Thirty-nine more oops Here, take
  this more powerful antibiotic.
• 2000 AD The bugs have won! Here, eat this root.

Anonymous, World Health Report on Infectious
Diseases 2000
4
Hierarchy of Research Evidence
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
Results from Observational Studies of Combined
Hormone Therapy and from the Women's Health
Initiative and the Heart and Estrogen/Progestin
Replacement Study
Grodstein, F. et al. N Engl J Med 2003348645-650
9
Hierarchy of Research Evidence
10
Hierarchy on Shaky Ground
Benson et al. N Engl J Med 2000 342 1878-1886
11
The Tyranny of the RCT

• "the claims for the RCT have been greatly, indeed
  preposterously overstated. The truth of the
  matter is that the RCT is one of many ways of
  generating information, of validating hypotheses.
  The proponents of the RCT, however, have
  elevated what is in theory a frequent (though by
  no means universal) advantage of degree into a
  gulf as sharp as that between the kosher and the
  non-kosher.
• Charles Fried

12
Weaknesses of RCTs

• Cannot be used to assess effects of potentially
  hazardous exposures
• Traditionally examine efficacy not effectiveness
• Difficulty with context
• Have difficulty analyzing complex interventions

13
Are RCTs the only way to learn?

• The difference between the RCT and the
  observational, retrospective study is not the
  difference between good and bad science, truth or
  falsity, but a difference between varying degrees
  of confidence.
• Charles Fried

14
Traditional View of QI

• Focused on implementing proven intervention in
  the real world (effectiveness)
• Primary objective is to improve patient care,
  gain of knowledge is secondary
• Local focus (not generalizeable)
• Minimal statistical analysis
• Not a viable form of academic productivity

15
Can QI be Research?

• It depends
• From IRB perspective, no
• From academic advancement perspective, yes
• Like RCTs and causal epidemiologic studies QI
  (when done right) can be used to
• Generate new knowledge
• Generalize that knowledge to other settings
• Increasing methodological rigor of QI has been
  the key to academic acceptance

16
Examples of QI Research
17
(No Transcript)
18
NOSOCOMIAL BSIIVD-RELATED BSIs

• 200 million IVDs/year in U.S.
• 0.2-5 risk of IVD-related BSI
• Attributable mortality 5
• Incremental costs 12,000/case
• 500,000 IVD-related BSIs in U.S. each year,
  25,000 deaths

19
OGrady et al. Clin Infect Dis 2002 35(11)
1287-1301
20
Guideline Compliance

• Education program (52)
• Routine exchange of CVCs (25)
• Appropriate hand hygiene (17 60)
• Insertion in subclavian vein (31 60)
• Maximal sterile barriers during insertion (28 -
  58)
• Chlorhexidine for insertion site preparation

Alonso-Echanove et al. Infect Control Hosp
Epidemiol 2003 Braun et al. Infect Control Hosp
Epidemiol 2003 Coopersmith et al. Arch Intern Med
2004 Warren et al. Infect Control Hosp Epidemiol
2006
21
QI Project to Reduce CVC-associated BSIs
P lt 0.0001
P lt 0.0001
P lt 0.0001
P 0.04
Eggiman et al. Lancet 2000 355(9218) 1864-8
22
Crit Care Med 2004 32(10) 2014-20

• Intervention 1 Education of staff
• Intervention 2 Creation of CVC insertion cart
• Intervention 3 Asking if CVC can be removed
  daily
• Intervention 4 Nurse completed checklist
• Intervention 5 Empowerment of nurses

23
Crit Care Med 2004 32(10) 2014-20
24
Program Expansion The Michigan Keystone Project
Pronovost et al. N Engl J Med 2006 355(26)
2725-32
25
Program Expansion The Michigan Keystone Project
Pronovost et al. N Engl J Med 2006 355(26)
2725-32
26
(No Transcript)
27
Antimicrobial Use in LTCFs

• Accounts for 20-40 of the medications used in
  LTCFs
• 3-13 of residents are receiving antimicrobials
  at any time
• 40-70 of residents will receive at least one
  antibiotic during 6 months of follow-up
• 25-75 of antibiotics given for inappropriate
  indications

28
Antimicrobial Use in LTCF
Mylotte, AJIC 1999 2710-19
29
Antimicrobial Use in LTCF
Mylotte, AJIC 1999 2710-19
30
Antimicrobial Use in LTCF
Mylotte, AJIC 1999 2710-19
31
Antimicrobial Use in LTCFs
Mylotte, AJIC 1999 2710-19
32
J Am Geriatr Soc 2004 52(1) 112-6

• Phase 1 Antibiotic utilization review
• Phase 2 Physician contact and guideline
  development
• Phase 3 Intervention deployment
• Pocket cards with Abx use guidelines
• Caregiver lectures
• Weekly antibiotic rounds
• Targeted feedback to outliers

33
J Am Geriatr Soc 2004 52(1) 112-6
P lt 0.001
P 0.08
34
J Am Geriatr Soc 2007 55(8) 1236-42

• Point prevalence study of antibiotic resistance
  (1998)
• Development of antimicrobial utilization methods
  (2000)
• Teaching and guideline intervention
• Targeted internists and nursing leaders
• Case-based education
• Dissemination of algorithms and guidelines

35
J Am Geriatr Soc 2007 55(8) 1236-42
Antibiotic-resistant infections (per 1,000-days)
? 25
36
Limitations of QI Research

• Internal validity problems
• Construct validity problems
• Statistical validity problems
• External validity problems

37
Internal Validity Problems

• Confounding factors, history
• Trends/maturation effects
• Seasonal effects
• Detection bias
• Attrition bias
• Selection bias
• Regression to the mean effects

38
Internal Validity Problems

• Confounding factors, history
• Trends/maturation effects
• Seasonal effects
• Detection bias
• Attrition bias
• Selection bias
• Regression to the mean effects

39
Confounding

• Problem
• Many features of care can change over time
  (intentional and unintentional)
• Reduces ability to infer causal benefit (or lack
  thereof) of intervention
• E.g., MRSA control programs
• Potential solutions
• Make inventory of potential care practices that
  can influence outcome of interest
• Seek to measure major confounders along with
  intervention
• Adjust for confounding in analyses
• When not feasible, fess up in your limitations
  section

40
Determinants of MRSA Control
Amox/Clav
3GCP
Alch HH
Alch Wipes
Macrolide
Fluoroquinolone
ASC
Admit MRSA
Aldeyab et al. J Antimicrob Chemother 2008
62(3) 593-600
41
Confounding

• Problem
• Many features of care can change over time
  (intentional and unintentional)
• Reduces ability to infer causal benefit (or lack
  thereof) of intervention
• E.g., MRSA control programs
• Potential solutions
• Make inventory of potential care practices that
  can influence outcome of interest
• Seek to measure major confounders along with
  intervention
• Adjust for confounding in analyses
• When not feasible, fess up in your limitations
  section

42
Internal Validity Problems

• Confounding factors, history
• Trends/maturation effects
• Seasonal effects
• Detection bias
• Attrition bias
• Selection bias
• Regression to the mean effects

43
J Am Geriatr Soc 2007 55(8) 1236-42
44
J Am Geriatr Soc 2007 55(8) 1236-42
45
J Am Geriatr Soc 2007 55(8) 1236-42
46
Dealing with Maturation Effects

• Multiple data points
• Utilize time series statistical methods for
  analyses
• Segmented regression
• ARIMA (autoregressive integrated moving average)
• Present data visually

47
Internal Validity Problems

• Confounding factors, history
• Trends/maturation effects
• Seasonal effects
• Detection bias
• Attrition bias
• Selection bias
• Regression to the mean effects

48
Emerg Infect Dis 2006 12(6) 894-9

• Spurious associations can occur because of
• Seasonal staffing fluctuations
• Seasonal fluctuations in illness
• Examples
• Initiating a MRSA control program as the
  influenza season was ending
• Fluctuating effectiveness of influenza vaccine
  programs due to year-to-year differences in
  virulence of virus and vaccine-virus match
• Solutions
• Exam effects of interventions across seasons
  (years)

49
Internal Validity Problems

• Confounding factors, history
• Trends/maturation effects
• Seasonal effects
• Detection bias
• Attrition bias
• Selection bias
• Regression to the mean effects

50
Ventilator-Associated Pneumonia Rates UW
Hospital (Hypothetical)
ICP 1
ICP 2
ICP 1
VAP Rate (per 1,000 vent-days)
UCL
Date
51
Internal Validity Problems

• Confounding factors, history
• Trends/maturation effects
• Seasonal effects
• Detection bias
• Attrition bias
• Selection bias
• Regression to the mean effects

52
Attrition Bias
MV Fatalities (per year)
Year
53
Attrition Bias
MV Fatalities (per year)
Miles Driven (100 Billion)
Year
54
Attrition Bias
MV Fatalities (per year)
Fatalities (per 100 Million Miles Driven)
Year
55
Internal Validity Problems

• Confounding factors, history
• Trends/maturation effects
• Seasonal effects
• Detection bias
• Attrition bias
• Selection bias
• Regression to the mean effects

56
Internal Validity Problems

• Confounding factors, history
• Trends/maturation effects
• Seasonal effects
• Detection bias
• Attrition bias
• Selection bias
• Regression to the mean effects

57
Regression to the Mean
Cooper et al. Health Technol Assess 2003 7(39)
1-194
58
Construct Validity
Intervention
? Outcome
59
Construct Validity
ASC Isolation
? MRSA
60
Statistical Conclusion Validity

• Small studies susceptible to type II errors
• Increase duration of data collection
• Increase the number of study sites
• Clustering effects will increase probability of
  committing a type I error
• Multilevel modeling
• Already discussed the problems associated with
  averaging results in study periods

61
External Validity

• Intervention may work only in
• A specific resident population
• A specific type of LTCF
• Solutions
• Implement intervention in a variety of patient
  populations
• Implement intervention in a variety of LTCF
  settings
• Utilization of a cluster-randomized study design

62
Summary

• Carefully plan the following
• Components of the intervention (be as specific
  and detailed as possible)
• Methods for assessing outcome (make sure the same
  method is used before and after the intervention)
• Carefully enumerate
• Potential stealth interventions (ASC HH)
• Potential confounders and methods to measure the
  major ones

63
Summary

• Collect pre-intervention data
• The more the better
• Look closely for trends and stochastic phenomena
• Analyses should
• Be based on multiple data points (not one from
  each study period)
• Adjust for major confounders
• Adjust for differences in LOS and time-varying
  variables
• Adjust for clustering effects if looking at
  communicable diseases or studies performed across
  multiple facilities
• Be forthright with limitations in submitted papers

64
Thank You