AHRQ Quality Indicators

1
AHRQ Quality Indicators

• Developed by Stanford-UCSF Evidence Based
  Practice Center
• Funded by the Agency for Healthcare Research and
  Quality

EPC Team (PSI Development) PI Kathryn McDonald,
M.M., Stanford Patrick Romano, M.D., M.P.H, UC
Davis Jeffrey Geppert, J.D., Ed.M.,
Stanford Sheryl Davies, M.A., Stanford Bradford
Duncan, M.D., M.A., Stanford Kaveh G. Shojania,
M.D., UCSF
Support of Quality Indicators PI Kathryn
McDonald, M.M., Stanford Sheryl Davies, M.A.,
Stanford Patrick Romano, M.D., M.P.H, UC
Davis Jeffrey Geppert, J.D. Ed.M., Stanford Mark
Gritz, PhD, Battelle Greg Hubert,
Battelle Denise Remus, RN PhD, AHRQ Project
Officer
2
Acknowledgements

• Funded by AHRQ
• Contract No. 290-97-0013
• Support of Quality Indicators Contract No.
  290-02-0007
• Presentation funded by AHRQ
• Data used for analyses
• Nationwide Inpatient Sample (NIS), 1995-2000.
  Healthcare Cost and Utilization Project (HCUP),
  Agency for Healthcare Research and Quality
• State Inpatient Databases (SID), 1997 (19
  states). Healthcare Cost and Utilization Project
  (HCUP), Agency for Healthcare Research and
  Quality

3
Acknowledgements

• We gratefully acknowledge the data organizations
  in participating states that contributed data to
  HCUP and that we used in this study the Arizona
  Department of Health Services California Office
  of Statewide Health and Development Colorado
  Health and Hospital Association CHIME, Inc.
  (Connecticut) Florida Agency for Health Care
  Administration Georgia Hospital Association
  Hawaii Health Information Corporation Illinois
  Health Care Cost Containment Council Iowa
  Hospital Association Kansas Hospital
  Association Maryland Health Services Cost Review
  Commission Massachusetts Division of Health Care
  Finance and Policy Missouri Hospital Industry
  Data Institute New Jersey Department of Health
  and Senior Services New York State Department of
  Health Oregon Association of Hospitals and
  Health Systems Pennsylvania Health Care Cost
  Containment Council South Carolina State Budget
  and Control Board Tennessee Hospital
  Association Utah Department of Health
  Washington State Department of Health and
  Wisconsin Department of Health and Family Service.

4
Outline

• Administrative data and quality indicators
• AHRQ Quality Indicators (QI)
• Development of AHRQ QIs
• Risk adjustment MSX smoothing methods
• Application of QIs to research and quality

5
History of AHRQ QIs/PSIs

• Healthcare Cost and Utilization Project (HCUP)
• HCUP discharge data collection (1988)
• HCUP Quality Indicators
• Mortality for Inpatient Procedures
• Complication Rates
• Potentially Inappropriate Utilization
• Potentially Avoidable Hospital Admissions

6
Refinement of HCUP QIs

• Refinement commissioned by AHRQ in 1999
• Completed by UCSF-Stanford EPC
• Two related projects
• Two technical reviews
• Refinement of the HCUP Quality Indicators
• Measures of Patient Safety Based on
  Administrative Data
• Three indicator sets, AHRQ QIs
• Inpatient Quality Indicators (IQIs)
• Prevention Quality Indicators (PQIs)
• Patient Safety Indicators (PSIs)

7
Administrative Data Quality Improvement

• Opportunities
• Coding practices improving
• Data availability improving (e.g., less
  truncation)
• More specific codes
• Large data sets improve precision
• Comprehensive all hospitals
• Quality screening feasible

• Obstacles
• Coding errors introduce noise
• Lack of information on timing, comorbidity vs.
  adverse events
• Varying number of secondary diagnoses fields can
  cause bias
• Heterogeneous severity within single code

8
Administrative Data

• State Inpatient Databases
• Includes ICD-9-CM dx and procedure codes, DRG,
  dates, age, sex, payer, race, discharge
  disposition, hospital and/or patient zip codes
• 1995-2002
• 33 States
• 80 of all U.S. hospital discharges
• 18 states available for purchase
• In 27 state sample, approximately 3200 hospitals

9
Administrative Data

• Nationwide Inpatient Sample (NIS)
• Sampling of State Inpatient Databases
• 1988-2001
• 7.5 million discharges/1,000 hospitals/33 States
• Approximates 20 sample of nonfederal acute care
  hospitals
• Discharge level weights applied for national
  estimates
• Available for purchase

10
HCUPnet

• http///hcupnet.ahrq.gov/
• Web-based tool to query NIS and KIDS databases,
  1993-2001
• Pre-run tables for 1997-2001
• Query based on ICD-9-CM, DRG or CCS
• Information on hospitalizations, charges, length
  of stay, mortality, discharge status
• Stratification by age, sex, race, income,
  insurance, hospital characteristics
• Rank order hospitalizations

11
Outline

• Administrative data and quality indicators
• AHRQ Quality Indicators (QI)
• Development of AHRQ QIs
• Risk adjustment MSX smoothing methods
• Application of QIs to research and quality

12
Sample AHRQ QI definition
13
Prevention Quality Indicators (PQIs)

• Defined using area population as denominator
• Potentially avoidable hospitalizations or
  ambulatory care sensitive conditions
• Conditions for which good outpatient care can
  potentially prevent the need for hospitalization
  or for which early intervention can prevent
  complications or more severe disease
• Public health, comprehensive health care systems
• Based on existing, validated indicators set, but
  modified and updated

14
Prevention Quality Indicators (PQIs)

• Bacterial pneumonia
• Dehydration
• Pediatric gastroenteritis
• Urinary tract infection
• Perforated appendix
• Low birth weight
• Angina without procedure
• Congestive heart failure
• Hypertension
• Adult asthma
• Pediatric asthma
• Chronic obstructive pulmonary disease
• Diabetes short-term complication
• Diabetes long-term complication
• Uncontrolled diabetes
• Lower-extremity amputation among patients with
  diabetes

15
Inpatient Quality Indicators (IQIs)

• Defined using both hospital admissions and area
  population as denominator
• Inpatient mortality for certain procedures and
  medical conditions
• Utilization of procedures for which there are
  questions of overuse, underuse, and misuse
• Volume of certain procedures
• Risk-adjusted using APR-DRGs
• Potential for internal quality improvement
  purposes
• Based on existing, validated indicators

16
Inpatient Quality Indicators (IQIs)

• Mortality Rates for Conditions
• Acute myocardial infarction (2 versions)
• Congestive heart failure
• Gastrointestinal hemorrhage
• Hip fracture
• Pneumonia
• Stroke
• Mortality Rates for Procedures
• Abdominal aortic aneurysm repair
• Coronary artery bypass graft
• Craniotomy
• Esophageal resection
• Hip replacement
• Pancreatic resection
• Pediatric heart surgery

• Hospital-level Procedure Utilization Rates
• Cesarean section delivery (primary and total)
• Incidental appendectomy in the elderly
• Bi-lateral cardiac catheterization
• Vaginal birth after Cesarean section (2 versions)
• Laparoscopic cholecystectomy
• Area-level Utilization Rates
• Coronary artery bypass graft
• Hysterectomy
• Laminectomy or spinal fusion
• PTCA
• Volume of Procedures
• Abdominal aortic aneurysm repair
• Carotid endarterectomy
• Coronary artery bypass graft
• Esophageal resection
• Pancreatic resection
• Pediatric heart surgery
• PTCA

17
Patient Safety Indicators (PSIs)

• Defined using hospital admissions as denominator
• Inpatient complications of care and potential
  patient safety events
• Potential for internal quality improvement
  purposes, monitoring of patient safety events
• Novel indicators, based on concepts reported in
  the literature

18
Patient Safety Indicators (PSIs)

• Provider-level Patient Safety Indicators
• Accidental puncture or laceration during
  procedure
• Complications of anesthesia
• Death in low mortality DRGs
• Decubitus ulcer
• Failure to rescue
• Foreign body left in during procedure
• Iatrogenic pneumothorax
• Selected infection due to medical care
• Postoperative hemorrhage or hematoma
• Postoperative hip fracture
• Postoperative physiologic and metabolic
  derangements
• Obstetric trauma vaginal delivery with
  instrument
• Obstetric trauma vaginal delivery without
  instrument
• Obstetric trauma cesarean section delivery

• Postoperative pulmonary embolism or deep vein
  thrombosis
• Postoperative respiratory failure
• Postoperative sepsis
• Transfusion reaction
• Postoperative wound dehiscence in abdominopelvic
  surgical patients
• Birth trauma injury to neonate
• Area-level Patient Safety Indicators
• Foreign body left in during procedure
• Iatrogenic pneumothorax
• Infection due to medical care
• Technical difficulty with medical care
• Transfusion reaction
• Postoperative wound dehiscence in abdominopelvic
  surgical patients

19
PQI Rates
Source SID, 2000. AHRQ Prevention Quality
Indicators SAS Software Version 2.1 Revision 3.
20
IQI Rates
Source SID, 2000. AHRQ Inpatient Quality
Indicators SAS Software Version 2.1, Revision 3.
Release pending.
21
PSI Rates
Source NIS, 2000. AHRQ Patient Safety Indicators
SAS Software Version 2.1 Revision 2. Release
pending.
22
Outline

• Administrative data and quality indicators
• AHRQ Quality Indicators (QI)
• Development of AHRQ QIs
• Risk adjustment MSX smoothing methods
• Application of QIs to research and quality

23
Methods

• Evaluation framework
• Literature review
• Identification of indicators
• Gray literature/interviews
• Identification of indicators
• Literature review
• Evidence for indicators
• Empirical analyses
• ICD-9-CM coding review (PSI only)
• Clinical panel reviews (PSI only)

24
Evaluation Framework

• Face validity does the indicator capture an
  aspect of quality that is widely regarded as
  important and subject to provider or public
  health system control?
• Precision is there a substantial amount of
  provider or community level variation that is not
  attributable to random variation?
• Minimum Bias is there either little effect on
  the indicator of variations in patient disease
  severity and comorbidities, or is it possible to
  apply risk adjustment and statistical methods to
  remove most or all bias?
• Construct validity does the indicator perform
  well in identifying true (or actual) quality of
  care problems?
• Fosters real quality improvement Is the
  indicator insulated from perverse incentives for
  providers to improve their reported performance
  by avoiding difficult or complex cases, or by
  other responses that do not improve quality of
  care?
• Application Has the measure been used
  effectively in practice? Does it have potential
  for working well with other indicators?

25
Literature ReviewIdentification of Indicators

• Systematic review to identify indicators
• Thousands of articles screened
• Over 200 abstracted
• Only 20 articles actually described indicators,
  most of which had overlapping indicators
• Grey literature searched to identify over 200
  indicators

26
Empirical Analyses

• Used novel statistical methods to measure
• Precision/Reliability
• Bias
• Inter-relatedness of indicators
• Precision criteria of 1.0 or more systematic
  variation among providers
• Then, literature review conducted

27
Literature ReviewEvidence for Each Indicator

• Identified and reported evidence for
• Face validity
• Precision and reliability
• Potential bias
• Construct validity
• Fosters true quality improvement (gaming)
• Current use

28
PSIs Methods Development of Candidate Indicator
List

• Background literature review
• Little evidence in peer reviewed journals
• Complications Screening Program
• Miller et al. Patient Safety Indicators
• Review of ICD-9-CM code book
• Codes from above sources grouped into indicators
  and assigned denominators
• Review of CSP evidence to retain indicators
• Final refinements of indicators

29
PSIs Methods Review of Candidate Indicators

• Literature review of potential indicators
• Coding validity/consistency
• Construct validity
• ICD-9-CM coding review
• Clinical panel review (face validity)
• Results used to define final set of indicators

30
PSIs Methods Clinical Panel Review

• Intended to establish consensual validity
• Modified RAND/UCLA Appropriateness Method
• Doctors of various specialties/subspecialties,
  nurses, specialized (e.g., midwife, pharmacist)
• Initial rating, followed by conference call,
  followed by final rating
• Rated indicator on
• Overall usefulness
• Present on admission
• Preventability of complication
• Likelihood due to medical error
• Extent indicator subject to bias
• Eight multispecialty panels, three surgical
  panels (5-9 members on each panel)

31
Example reviewsMultispecialty Panels

• Overall rating
• Not present on admission
• Preventability (4)
• Due to medical error (2)
• Charting by physicians (6)
• Not biased (3)

32
PSIs Methods Final Selection of Indicators

• Indicators for which overall usefulness rating
  was high
• Some changes in indicator set based on coding
  review and operationalization concerns (e.g.,
  reopening of surgical site)
• Empirical analyses of nationwide rates,
  variation, impact of risk adjustment, and
  relationship between indicators

33
Outline

• Administrative data and quality indicators
• AHRQ Quality Indicators (QI)
• Development of AHRQ QIs
• Risk adjustment MSX smoothing methods
• Application of QIs to research and quality

34
Risk-Adjustment Criteria

• User-specified criteria for evaluating
  risk-adjustment systems
• 1) Open systems preferred
• 2) Data collection costs minimized and
  well-justified
• 3) Multiple-use coding system
• 4) Official recognition

35
Evidence on DRG-based Systems

• Open systems
• Widely adopted by state agencies
• Based on existing data collection systems
• Use for reimbursement ensures improved data
  quality
• Evidence suggests at least equivalent performance
  across broad spectrum of conditions
• Studies underway to examine alternatives

36
3M APR-DRG

• All-patient refined (956 categories in version
  15.0, including pediatrics)
• Severity of illness subclass that reflect
  presence of co morbidity/complication and level
• Risk of mortality subclass
• Differential impact of secondary diagnosis by
  condition

37
Evidence on 3M APR-DRG

• All-patient refined (956 categories in version
  15.0, including pediatrics)
• Severity of illness subclasses that reflect
  presence of co morbidity/complication and level
• Risk of mortality subclasses
• Differential impact of secondary diagnosis by
  condition

38
Evidence on 3M APR-DRG

• Better empirical performance than DRG-based
  alternatives on predicting mortality (especially
  for surgical patients patients at large, urban,
  teaching hospitals)
• Better empirical performance than DRG-based
  alternatives on predicting resource use
  (especially for medical patients patients over
  65, at children, teaching hospitals)
• Better at reflecting the distribution of patient
  severity at the extremes

39
Risk-Adjustment Conclusions

• No single system based on administrative or
  clinical data is clearly superior
• DRG-based systems perform as well, and often
  better, than alternatives
• Data enhancements may improve performance (e.g.,
  condition present on admission, key clinical
  variables)

40
Risk-Adjustment ModelInpatient Quality Indicators

• Direct standardization
• Indirect standardization
• RA (OR / ER) PR
• (RA risk adjusted OR observed ER
  expected PR population)

41
Risk-Adjustment Model

• Expected rate Assuming the hospitals case-mix
  and the population rates
• Risk-adjusted rate Assuming the populations
  case-mix and the hospitals rates

42
Risk-Adjustment Model

• Linear regression model
• observed rate hospital effect
• demographic effect
  
• condition effect
  error
• Model estimated on the SID, 2000 (25 million
  discharges)

43
Risk-Adjustment Model

• IQI Age, sex, APR-DRG (with risk of mortality
  or severity of illness subclass)
• (linear with hospital fixed effects)
• PQI Age and sex
• (linear with area fixed effects)
• PSI Age, sex, modified CMS DRG and AHRQ
  comorbidity (logistic)

44
How it Works CABG Mortality
45
How it Works CABG Mortality
46
MSX Smoothing Model

• Observed quality measure true quality (signal)
  error (noise)
• Smaller hospitals and/or less frequent conditions
  have more noise
• Difficult to compare hospitals, trend over time,
  and identify best practices
• Confidence intervals reflect but do not address
  the problem

47
Key Features of MSX Approach

• Removes noise uses redundancy over time and
  among measures
• Improves forecasts predicting current quality
  based on past performance
• Reduces dimensionality appropriately - allows
  meaningful summary measures
• Reveals and helps reduce biases, identify best
  practices

48
Outline

• Administrative data and quality indicators
• AHRQ Quality Indicators (QI)
• Development of AHRQ QIs
• Risk adjustment MSX smoothing methods
• Application of QIs to research and quality

49
Caveats of Use

• Validity of data
• Validity of coding
• Present on admission
• Outpatient care
• Linking of admissions and impact of LOS
• Incomplete risk adjustment

50
Using the AHRQ QI

• State monitoring of rates
• Hospital quality improvement
• National Healthcare Quality Report
• PQIs and PSIs
• CMS Pay for Performance Demonstration Project
• Postoperative Hemorrhage or Hematoma
• Postoperative Metabolic and Physiologic
  Derangement
• Romano et al
• PSI National trends, (HA, Mar/Apr 03)

51
Using the AHRQ QI

• Kovner
• QI and nurse staffing
• Miller
• PSI and Costs and LOS
• Alexander/Shortell
• PSI and Quality improvement culture
• Baker
• PSI and Patient safety culture hospital
  characteristics
• Rosen
• VA hospitals, QI and other measures (NSQuIP)
• Volpp
• QIs and new resident work hours

52
Technical Reports

• Development of Quality Indicators, risk
  adjustment and MSX methods documented in
• Davies S, Geppert J, McClellan M, McDonald KM,
  Romano PS, Shojania KG. Refinement of the HCUP
  Quality Indicators. Technical Review Number 4.
  Rockville, MD (Prepared by the UCSF-Stanford
  Evidence-based Practice Center under Contract No.
  290-97-0013) Agency for Healthcare Research and
  Quality 2001. Report No. 01-0035
• McDonald KM, Romano PS, Geppert J, Davies S,
  Shojania KG. Measures of Patient Safety Based on
  Hospital Administrative Data The Patient Safety
  Indicators. Technical Review Number 5. Rockville,
  MD (Prepared by the UCSF-Stanford Evidence-based
  Practice Center under Contract No. 290-97-0013)
  Agency for Healthcare Research and Quality
  August 2002. Report No. 01-0038.

53
For More Information on AHRQ QIs

• Quality Indicators Additional information and
  assistance
• E-mail support_at_qualityindicators.ahrq.gov
• Website http//qualityindicators.ahrq.gov/
• QI technical reports, documentation and software
  is available on the website
• User Support is provided under contract by
  Battelle Memorial Institute, Stanford University
  and University of California at Davis