Performance Reports

1
Performance Reports

• Andy Bindman MD
• Department of Medicine, Epidemiology and
  Biostatistics
• UCSF

2
Calculating an Individuals Risk

• Solve the multivariate model incorporating an
  individuals specific characteristics
• For continuous outcomes the predicted values are
  the expected values
• For dichotomous outcomes the sum of the derived
  predictor variables produces a logit which can
  be algebraically converted to a probability (enat
  log odds /1 enat log odds )

3
Aggregating to the group level

• Sum observed events (eg deaths) for sub-group
• Sum expected probability of events for same
• Probabilities used to calculate expected events
  derived from entire data set and applied to
  individuals in sub-group (eg defined by care
  site)
• The overall expected number of events must equal
  the observed number of events but this need not
  be the case at the level of subgroups

4
Comparing observed and expected outcomes to
assess quality

• Observed events or rates of events
• Expected events or rates of events
• Better quality implied when observed is lower
  than expected (worse quality when observed higher
  than expected)

5
Risk adjusted rates

• Standardizes rates across sub-groups so that they
  can be directly compared with a single number
• Observed rate/expected rate of subgroup
• x overall observed rate

6
Observed CABG Mortality Rates, NY 1989-1992
7
Calculating Expected CABG Mortality Rates in New
York by Year

• Pool all 4 years of CABG patients
• Develop risk adjustment model for CABG patients
• Apply risk adjustment model for CABG patients
  sub-grouped by year to determine expected number
  of deaths for each year
• Divide expected number of deaths by number of
  cases per year to get expected death rate

8
Observed and Expected CABG Mortality Rates, NY
1989-1992
9
Annual Risk Adjusted Mortality Rate for CABG in
New York

• Observed rate per year/expected rate per year
• X average death rate over 4 year period (3.1)

10
Observed, Expected and Risk-Adjusted CABG
Mortality Rates, NY 1989-1992
11
What Happened with CABG Surgery Over Time in New
York?

• Operated on sicker patients
• Observed mortality rate declined over time
• Risk adjusted mortality rate declined even more
• Did quality of CABG care improve over time?

12
NY CABG Risk Adjustment Model

• Well designed model
• C index .787 Hosmer-Lemeshow chi square p.16
• Mortality is not a subjective outcome- hard to
  fake
• Gaming might be possible with coding some
  predictors

13
Interpreting Risk Adjusted CABG Outcomes

• Public reporting on hospital CABG mortality began
  in 1989
• Low volume hospitals had higher mortality rates
  and some stopped performing CABG over time
• Process indicators of cardiac care (beta blocker
  post MI) also improved in NY hospitals over time
• Hospitals documented more co-morbidities over
  time resulting in inflated expected death rates
• Some sick NY cardiac patients operated on in NJ

14
Applying Results To Providers

• Possible to aggregate observed and expected rates
  of events to hospital, physician, or some other
  provider level grouping
• Statistical problems arise when total number of
  expected events are small
• Minimum of five expected events per group as a
  rule of thumb

15
Naming Names

• Assigning assessments of quality to specific
  providers increases the stakes
• Need to demonstrate validity of analytic approach

16
Reporting Results

• Public reporting vs internal quality improvement
• Data users tend to want gradations of quality
  along a continuum (excellent to poor)
• However denoting those within a 95 confidence
  interval of the expected as average is less
  sensitive to noise in data

17
Bootstrap ProcedureDeriving Confidence Intervals

• Multiple (e.g. 1000) random samples of same size
  of original derived from original sample with
  replacement
• Calculate expected rate for each new sample
• Create frequency distribution of expected rates
• Empirically derive 95 CI (950 of 1000 centered
  around mean)

18
(No Transcript)
19
Consistency in the evidence

• Differences between observed and expected may be
  due to things other than quality
• Are the results consistent over time
• Are results consistent with prior expectations
  such as volume-outcome relationships
• Confirmation through very different types of
  evidence is a major goal- external validation

20
Observed / Expected Mortality
21
Volume-Outcome

• Relationship between high volume providers and
  better outcomes
• Most often studied in relationship to procedures
• Consistent with notion that practice makes perfect

22
Hospital Volume and CABG Mortality in California
Hospitals Using Registry, 2000-02
23
External validation of data

• Link hospital discharge data with CABG registry
  data
• Looking for missing cases, deaths and highly
  predictive risk factors
• 221 cases in discharge data not reported to
  registry
• 26 additional deaths (498 total)
• 63 undercodes and 123 overcodes of cardiogenic
  shock
• 29 overcodes of salvage (51 total)
• Direct auditing
• Deaths
• Highly weighted predictors particularly if
  subjective

24
Growing Number of Quality Initiatives Provide
Opportunity for Cross Comparisons

• AHRQ Quality Indicators
• JCAHO ORYX Hospital Core Performance
• CMS Hospital Quality Alliance
• National Quality Forum (NQF)
• Leapfrog
• NCQA HEDIS

25
JCAHO - Hospital AMI indicators

• Process elements
• Aspirin at arrival, Aspirin at discharge, ACEI
  for LVSD, Smoking cessation counseling,
  Beta-blocker at discharge, Beta-blocker at
  arrival, Thrombolytic within 30 minutes, PCI
  within 120 minutes

26
Who uses these reports and how

• Patients
• Slow to catch on
• More important for those without other ways to
  judge quality
• Managers
• Aim to improve quality to avoid naming and
  shaming
• Payers (health plans)
• Selective contracting
• Pay for performance

27
How much do these reports matter?

• California has lowered isolated CABG mortality by
  1 (from 3 to 2 ) during public report period
• Approximately 20,000 procedures per year
• Reduction from 600 to 400 deaths
• Average survival 5 years
• Even if half the change is due to gaming, 500
  life years saved

28
My Reflections on Performance Reports

• View the risk adjusted estimates as yellow
  flags, not smoking guns
• Risk models will probably improve with ability to
  add more clinical data available through
  electronic records
• Research may not need to be perfect to bring
  about public health benefits
• Attempts to improve quality need to consider
  unintended consequences on access/efficiency

29
Outcomes Research Opportunities

• Validate risk adjustment models for new
  conditions
• Are health care outcomes changing over time and
  if so why?
• How can performance reports on health outcomes be
  used to create better health care quality?