Assessing Patient Safety through Administrative Data: Adapting and Improving Existing Systems

1
Assessing Patient Safety through Administrative
DataAdapting and Improving Existing Systems

• Patrick S. Romano, MD MPH
• Professor of Medicine and Pediatrics
• UC Davis School of Medicine
• Sacramento CA, USA
• June 29, 2006

2
Acknowledgments

• Support for Quality Indicators II (Contract No.
  290-04-0020)
• Mamatha Pancholi, AHRQ Project Officer
• Marybeth Farquhar, AHRQ QI Senior Advisor
• Mark Gritz and Jeffrey Geppert, Project
  Directors, Battelle Health and Life Sciences
• Kathryn McDonald (PI) and Sheryl Davies (project
  manager), Stanford University
• Other clinical team members Douglas Payne
  (medicine), Garth Utter (surgery), Shagufta
  Yasmeen (obstetrics gynecology), Corinna
  Haberland (pediatrics), Banafsheh Sadeghi
  (research assistant)

3
Overview

• General approaches to assessing inpatient safety
• Rationale for using administrative data
  strengths and limitations
• Background about the AHRQ Quality Indicators
  program
• Development and maintenance of the AHRQ Patient
  Safety Indicators (PSIs)
• OECD international expert panel review
• International interest in the AHRQ PSIs
• Practical issues associated with international
  application of the AHRQ PSIs

4
Taxonomy of patient safety measures
Donabedians classification Examples
Structural measures Hospital design Staffing (intensity, training) Decision support systems Safety culture
Process measures Medication errors (incorrect dosing, inappropriate use) Medical errors Near misses
Outcome measures Adverse events (potentially preventable complications, medical injuries)
Zhan et al., Med Care 200543I42-I47
5
General approaches to assessing inpatient safety

• Analyze administrative data (adverse events,
  selected types of medical errors)
• Review medical records (adverse events, selected
  types of medical errors)
• Collect confidential provider reports of
  incidents or safety events (passive
  surveillance of medical errors or near misses)
• Conduct active surveillance or real-time
  observation (same)
• Survey patients
• Survey employees or managers on organizational
  capabilities or climate (culture of safety)

6
Ethnographic observation to identify adverse
events and errors
480 of 1047 patients (46) experienced a mean of
4.5 events
Andrews LB, et al. Lancet 1997349309-13. Ethnog
raphers trained in qualitative observational
research attended regularly scheduled attending
rounds, resident work rounds, nursing shift
changes, case conferences, and other scheduled
meetings (e.g., MM conferences, QA meetings) on
3 units at one teaching hospital.
7
Video recording to identify errors in pediatric
trauma resuscitation
Mean of 5.9 errors per resuscitation, with 93
agree-ment between 2 reviewers. Mean of 2.2
errors in each seriously injured child, with 20
capture on medical records
Oakley, E. et al. Pediatrics 2006117658-664
8
Rationale for using administrative data

• Limitations
• Limited/no information on processes of care and
  physiologic measures of severity
• Limited/no information on timing (comorbidities
  vs. adverse events)
• Heterogeneous severity within some ICD codes
• Accuracy depends on documentation and coding
• Data are used for other purposes, subject to
  gaming
• Time lag limits usefulness

• Opportunities
• Data availability improving
• Coding systems and practices improving
• Large data sets optimize precision
• Comprehensiveness (all hospitals, all payers)
  avoids sampling/selection bias
• Data are used for other purposes, subject to
  auditing and monitoring

9
AHRQ Quality Indicators (QIs)

• Developed through contracts with UC-Stanford
  Evidence-based Practice Center
• Use existing hospital discharge data, based on
  readily available data elements
• Incorporate severity adjustment methods
  (APR-DRGs, comorbidity groupings) when possible
• Offer free, downloadable software (SAS, Windows)
  with documentation, biennial updates, and user
  support through listserve, newsletters, national
  meetings, web seminars, e-mail system
• User feedback drives continuous improvement

10
AHRQ Quality Indicators
Inpatient QIs Mortality Utilization Volume
Prevention QIs (Area Level) Avoidable
Hospitalizations Other Avoidable Conditions
Patient Safety Indicators Complications Failure-t
o-rescue Unexpected death
11
Structure of indicators

• Definitions based on
• ICD-9-CM diagnosis and procedure codes
• Inclusion/exclusion criteria based upon DRGs,
  sex, age, procedure dates, admission type
• Numerator number of cases flagged with the
  complication or situation of interest
• e.g., postoperative sepsis, avoidable
  hospitalization for asthma, death
• Denominator number of patients considered to be
  at risk for that complication or situation
• e.g. elective surgical patients, county
  population from census data
• Indicator rate numerator/denominator

12
AHRQ QI development General process

• Literature review (all)
• To identify quality concepts and potential
  indicators
• To find previous work on indicator validity
• ICD-9-CM coding review (all)
• To ensure correspondence between clinical concept
  and coding practice
• Clinical panel reviews (PSIs, pediatric QIs)
• To refine indicator definition and risk groupings
• To establish face validity when minimal
  literature
• Empirical analyses (all)
• To explore alternative definitions
• To assess nationwide rates, hospital variation,
  relationships among indicators
• To develop methods to account for differences in
  risk

13
Literature review to find candidate PSI indicators

• MEDLINE/EMBASE search guided by medical
  librarians at Stanford and NCPCRD (UK)
• Few examples described in peer reviewed journals
• Iezzoni et al.s Complications Screening Program
  (CSP)
• Miller et al.s Patient Safety Indicators
• Review of ICD-9-CM code book
• Codes from above sources were grouped into
  clinically coherent indicators with appropriate
  denominators

14
Coding (criterion) validity based on literature
review (MEDLINE/EMBASE)

• Validation studies of Iezzoni et al.s CSP
• At least one of three validation studies (coders,
  nurses, or physicians) confirmed PPV at least 70
  among flagged cases
• Nurse-identified process-of-care failures were
  more prevalent among flagged cases than among
  unflagged controls
• Other studies of coding validity
• Very few in peer-reviewed journals, some in gray
  literature

15
Construct validity based on literature review
(MEDLINE/EMBASE)

• Approaches to assessing construct validity
• Is the outcome indicator associated with explicit
  processes of care (e.g., appropriate use of
  medications)?
• Is the outcome indicator associated with implicit
  process of care (e.g., global ratings of
  quality)?
• Is the outcome indicator associated with nurse
  staffing or skill mix, physician skill mix, or
  other aspects of hospital structure?

16
ICD-9-CM coding consultant review

• All definitions were reviewed by an expert coding
  consultant from the American Health Information
  Management Association, with special attention to
  prior coding guidelines
• Central staff of ICD-9-CM were queried as
  necessary
• Definitions were refined as appropriate

17
Face validity Clinical panel review

• Intended to establish consensual validity
• Modified RAND/UCLA Appropriateness Method
• Physicians of various specialties and
  subspecialties, nurses, other specialized
  professionals (e.g., midwife, pharmacist)
• Potential indicators were rated by 8
  multispecialty panels surgical indicators were
  also rated by 3 surgical panels

18
Face validity Clinical panel review (contd)

• All panelists rated all assigned indicators on
• Overall usefulness
• Likelihood of identifying the occurrence of an
  adverse event or complication (i.e., not present
  at admission)
• Likelihood of being preventable (i.e., not an
  expected result of underlying conditions)
• Likelihood of being due to medical error or
  negligence (i.e., not just lack of ideal or
  perfect care)
• Likelihood of being clearly charted
• Extent to which indicator is subject to case mix
  bias

19
Evaluation framework
Medical error and complications continuum
Medical error
Nonpreventable Complications

• Pre-conference ratings and comments/suggestions
• Individual ratings returned to panelists with
  distribution of ratings and other panelists
  comments/suggestions
• Telephone conference call moderated by PI and
  attended by note-taker, focusing on
  high-variability items and panelists suggestions
  (90-120 mins)
• Suggestions adopted only by consensus
• Post-conference ratings and comments/ suggestions

20
Example reviewsMultispecialty panels

• Overall rating
• Not present on admission
• Preventability (4)
• Due to medical error (2)
• Charting by physicians (6)
• Not biased (3)

21
Final selection of indicators

• Retained indicators for which overall
  usefulness rating was Acceptable or
  Acceptable-
• Median score 7-9
• Definite or indeterminate agreement
• Excluded indicators rated Unclear, Unclear-,
  or Unacceptable
• Median score lt7, OR
• At least 2 panelists rated the indicator in each
  of the extreme 3-point ranges

22
Candidate PSIs reviewed

• 48 indicators reviewed in total
• 37 reviewed by multispecialty panel
• 15 of those reviewed by surgical panel
• 20 accepted based on face validity
• 2 dropped due to operational concerns
• 17 experimental or promising indicators
• 11 rejected

23
Accepted PSIs

• Selected postop complications
• Postoperative thromboembolism
• Postoperative respiratory failure
• Postoperative sepsis
• Postoperative physiologic and metabolic
  derangements
• Postoperative abdominopelvic wound dehiscence
• Postoperative hip fracture
• Postoperative hemorrhage or hematoma
• Selected technical adverse events
• Decubitus ulcer
• Selected infections due to medical care
• Technical difficulty with procedures
• Iatrogenic pneumothorax
• Accidental puncture or laceration
• Foreign body left in during procedure

• Other
• Complications of anesthesia
• Death in low mortality DRGs
• Failure to rescue
• Transfusion reaction (ABO/Rh)
• Obstetric trauma and birth trauma
• Birth trauma injury to neonate
• Obstetric trauma vaginal delivery with
  instrument
• Obstetric trauma vaginal delivery without
  instrument
• Obstetric trauma cesarean section delivery

24
Pediatric Quality Indicators

• Inpatient Indicators
• Accidental puncture and laceration
• Decubitus ulcer
• Foreign body left in after procedure
• Iatrogenic pneumothorax in neonates at risk
• Iatrogenic pneumothorax in non-neonates
• Pediatric heart surgery mortality
• Pediatric heart surgery volume
• Postoperative hemorrhage or hematoma
• Postoperative respiratory failure
• Postoperative sepsis
• Postoperative wound dehiscence due to medical
  care
• Transfusion reaction

25
PSI risk adjustment methods

• Must use only administrative data
• APR-DRGs and other canned packages may adjust for
  complications
• Final model
• DRGs (complication DRGs aggregated)
• Modified Comorbidity Index based on list
  developed by Elixhauser et al. (completely
  redesigned for Pediatric QIs)
• Age, Sex, Age-Sex interactions

26
Pediatric QI Risk Adjustment

• Reason for admission/type of procedure
• DRGs (with/without CC collapsed)
• Other (e.g., diagnostic/therapeutic procedure
  categories for accidental injury)
• Comorbidity
• Special pediatric-oriented comorbidity list
• Gender, age groups
• lt29 d, 29-60 d, 61-90 d, 91-365 d, 1-2 yrs, 3-5
  yrs, 6-12 yrs, 13-17 yrs
• Low birth weight categories (neonates)
• 500 gram categories (500-2500 g)

27
OECD Health Care Quality Indicators Project

• Includes 21 countries, WHO, European Commission,
  World Bank, ISQua, etc.
• Five priority areas
• Cardiac care
• Diabetes mellitus
• Mental health
• Patient safety
• Prevention/health promotion and primary care

28
OECD Indicator Selection Criteria

• Importance
• Impact on health
• Policy importance (concern for policymakers and
  consumers)
• Susceptibility to being influenced by the health
  care system
• Scientific soundness
• Face validity (clinical rationale and past usage)
• Content validity
• Feasibility
• Data availability on the international level
• Reporting burden

29
OECD Review Process

• Patient safety panel constituted with 5 members
  (Dr. John Millar, Chair)
• 59 indicators from 7 sources submitted for review
  (US, Canada, Australia)
• Modified RAND/UCLA Appropriateness Method
• Panelists rated each indicator on importance and
  scientific soundness (2 rounds with intervening
  discussion)
• Retained 21 indicators with median score gt7
  (scale 1-9) on both domains rejected indicators
  with median score 5 on either domain

30
OECD expert panel ratings of PSIs
31
AHRQ panel ratings of PSI preventabilityvery
similar to OECD ratings
a Panel ratings were based on definitions
different than final definitions. For Iatrogenic
pneumothorax, the rated denominator was
restricted to patients receiving thoracentesis or
central lines the final definition expands the
denominator to all patients (with same
exclusions). For In-hospital fracture panelists
rated the broader Experimental indicator, which
was replaced in the Accepted set by
Postoperative hip fracture due to operational
concerns. b Vascular complications were rated as
Unclear (-) by surgical panel multispecialty
panel rating is shown here.
32
US rates of OECD-endorsed PSIs
Patient Safety Indicator 2003 events 2003 rate per 1,000
COMPLICATIONS OF ANESTHESIA 7,406 0.775
DECUBITUS ULCER 198,752 23.365
FOREIGN BODY LEFT IN DURING PROC 2,741 0.086
INFECTION DUE TO MEDICAL CARE 43,591 2.052
POSTOPERATIVE HIP FRACTURE 1,511 0.279
POSTOPERATIVE PE OR DVT 80,477 9.883
POSTOPERATIVE SEPSIS 10,435 10.463
ACCIDENTAL PUNCTURE/LACERATION 97,058 3.574
TRANSFUSION REACTION 151 0.005
BIRTH TRAUMA -INJURY TO NEONATE 22,061 5.412
OB TRAUMA - VAGINAL W INSTRUMENT 55,502 189.576
OB TRAUMA - VAGINAL W/O INSTRUMENT 116,707 45.219
33
Primary uses of the AHRQ PSIs

• Internal hospital quality improvement
• Individual hospitals and health care systems,
  hospital associations
• Trigger case finding, root cause analyses,
  identification of clusters
• Evaluate impact of local interventions
• Monitor performance over time
• External hospital accountability to the community
• National, State and regional analyses
• National Healthcare Quality/Disparities Reports
• Surveillance of trends over time
• Disparities across areas, SES strata, ethnicities

34
Relative change from 1999-2000 to 2002-2003 in
observed and risk-adjusted AHRQ PSI rates
Patient Safety Indicator change Observed change Risk-adjusted
COMPLICATIONS OF ANESTHESIA 14.7 13.7
DECUBITUS ULCER 12.1 11.7
FOREIGN BODY LEFT IN DURING PROC 4.5
INFECTION DUE TO MEDICAL CARE 13.8 11.0
POSTOPERATIVE HIP FRACTURE -8.4 -12.2
POSTOPERATIVE PE OR DVT 25.3 26.6
POSTOPERATIVE SEPSIS 15.6 14.7
ACCIDENTAL PUNCTURE/LACERATION 3.1 3.9
TRANSFUSION REACTION 13.2
BIRTH TRAUMA -INJURY TO NEONATE -8.3 -8.3
OB TRAUMA - VAGINAL W INSTRUMENT -10.1 -9.4
OB TRAUMA - VAGINAL W/O INSTRUMENT -15.3 -14.9
35
Newer uses of the AHRQ PSIs

• Testing research hypotheses related to patient
  safety
• Housestaff work hours reform
• Nurse staffing regulation
• Public reporting by hospital
• Texas, New York, Colorado, Oregon, Massachusetts,
  Wisconsin, Florida, Utah
• Pay-for-performance by hospital
• CMS/Premier Demonstration (278 hospitals, focus
  on 2 postop events after THA/TKA)
• Anthem of Virginia (focus on monitoring any two)
• Hospital profiling
• Blue Cross/Blue Shield of Illinois

36
International inquiries regarding the AHRQ QIs
Canada 58
Spain 3
Italy 15
Australia 7
Belgium 5
South Africa 1
Philippines 1
Slovenia 1
Taiwan 3
Switzerland 1
Romania 3
New Zeland 4
Argentina 2
Portugal 1
United Kingdom 1
Japan 3
Germany 7
France 1
Indonesia 2
Saudi Arabia 2
Guyana 1
37
International inquiries regarding the AHRQ QIs
Quality Indicator Module Number
Prevention Quality Indicators 15
Inpatient Quality Indicators 46
Patient Safety Indicators 74
Pediatric Quality Indicators 1
No specific module 51
38
Practical issues in international implementation
of AHRQ PSIs

• ICD-9-CM to ICD-10 conversion
• Entirely different coding structure
• Three new chapters
• 12,420 codes versus 6,969
• Nation-specific versions (CA, AU, GM)
• No internationally accepted coding system for
  procedures

39
Practical issues in international implementation
of AHRQ PSIs

• Variation in documentation and coding practices
• Variation in other data definitions
• Principal versus primary diagnosis
• Number of diagnosis codes
• Procedure dates
• External cause of injury codes
• Type of admission (elective, urgent, emergency)
• Variation in how administrative data are
  collected and used
• DRG-based payment versus global budgeting versus
  service-based payment

40
Coding of secondary diagnoses in the USA

• For reporting purposes the definition for "other
  diagnoses" is interpreted as additional
  conditions that affect patient care in terms of
  requiring
• clinical evaluation or
• therapeutic treatment or
• diagnostic procedures or
• extended length of hospital stay or
• increased nursing care and/or monitoring.
• All conditions that occur following surgeryare
  not complications there must be more than a
  routinely expected condition or occurrence there
  must be a cause-and-effect relationship between
  the care provided and the condition

41
ICD-9-CM Coding Procedures

• Coding of procedures
• The UHDDS requires all significant procedures
  to be reported A significant procedure is
  defined as one that meets any of the following
  conditions
• Is surgical in nature
• Carries an anesthetic risk
• Carries a procedural risk
• Requires specialized training.
• What about central venous catheters?

42
International initiatives

• Conversion efforts are underway, but need to be
  coordinated internationally
• Undertake detailed meta-analysis of national data
  systems
• Review international variation in coding rules
  and procedures
• Improve data systems (e.g., present at
  admission coding in USA) and develop data on
  accuracy
• Prioritize indicators based on likelihood of
  international comparability

43
International collaborative meeting of health
services researchers using administrative data
Calgary, Alberta, June 2005 supported by CIHR
forthcoming in BMC HSR
44
Conversion of Elixhauser comorbidity list from
ICD-9-CM to ICD-10, ICD-10-CA
Quan H, et al., reported at AcademyHealth 2006
45
German mapping of PSIs from ICD-9-CM to ICD-10-GM
Saskia E. Droesler and Juergen Stausberg
46
PSI incidence comparisonGermany vs. USA
German population rate (log) 2004
US population rate (log) 2002
47
Developing data on accuracy and relevance AHRQ
PSIs in Childrens HospitalsSedman A, et al.
Pediatrics 2005115(1)135-145
PSI No. reviewed (total events) Preventable (PPV ) Nonpreventable Unclear
Complications of anesthesia 74 (503) 11 (15) 37 25
Death in low-mortality DRG 121 (1282) 16 (13) 89 16
Decubitus ulcer 130 (2300) 71 (55) 47 10
Failure to rescue 187 (5271) 15 (8) 148 11
Foreign body left in 49 (235) 25 (51) 14 10
Postop hemorrhage or hematoma 114 (1571) 40 (35) 51 23
Iatrogenic pneumothorax 114 (1113) 51 (45) 42 21
Selected infection 2 to med care 152 (7291) 63 (41) 45 39
Postop DVT/PE 126 (1956) 36 (29) 61 29
Postop wound dehiscence 41 (232) 19 (46) 16 6
Accidental puncture or laceration 133 (4020) 86 (65) 19 26