Combining multiple indicators of clinical quality

1
Combining multiple indicators of clinical quality
David Reeves Stephen Campbell John Adams Paul
Shekelle Evan Kontopantelis Martin
Roland National Primary Care RD
Centre University of Manchester RAND, California
2
Aims
To compare methods for computing a single
composite quality score for a health
provider To use methods being used in
pay-for-performance schemes and quality
assessment studies To use transparent methods
understandable to clinicians
3
The Datasets
Data from audits of random samples of GP patient
records GAP Study 23 conditions (most common
in primary care) 179 quality indicators 16
general practices 100 patients per
practice QUASAR Study 3 conditions (CHD,
Asthma, Diabetes) 46 quality indicators 60
practices, up to 60 patients per practice
4
The Indicators
Indicators developed using a modification of the
RAND Appropriateness method. Uses a combination
of scientific literature review and expert
panels.

• Example diabetes mellitus indicators
• The diagnosis of diabetes should be clearly
  identifiable on the electronic or paper records
  of all known diabetics.
• HbA1C levels should be checked in diabetic
  patients at least every 12 months.
• All diabetic patients should have an annual
  fundal examination.
• Diabetic patients with sustained proteinuria
  should be currently prescribed treatment with ACE
  inhibitors unless contraindicated.

5
Indicator data for 149 hypertensive patients (GAP)
Indicator
Pat ient
6
Unconditional and Conditional Indicators
An unconditional indicator applies to all
patients with a disease condition A conditional
indicator applies to a subset of those
patients Conditional indicators often apply to
small numbers, but can be disproportionately
important eg headache patients with focal
neurological signs should be referred to a
specialist
72 of indicators in dataset A were
conditional 54 of indicators in dataset B were
conditional
7
Five Methods for Computing Quality Scores (1)
Criteria-referenced methods All-or-none of
patients for whom all indicators were
met Advocates argue for no compromise in pursuit
of excellence. Recently adopted by the Centers
for Medicare Medicaid Services 70 Standard
of patients for whom gt70 of indicators
met Less stringent version of above
8
Five Methods for Computing Quality Scores (2)
Absolute score methods Overall Percentage - of
all opportunities for care met Used to reward
hospitals in the Centers for Medicare and
Medicaid Services and Premier Hospital Quality
Incentive Demonstration Project. Indicator
average mean rate at which each indicator
was met Used to evaluate quality of care in 4,000
hospitals involved in the US Hospital Quality
Alliance. A related method forms the basis for
the pay for performance scheme in the UK family
practitioner contract. Patient average
mean of indicators met per patient Used to
evaluate change in quality of care over a 5-year
period in a representative sample of UK general
practices.
9
Criteria-referenced quality scores
? triggered and met ? triggered but not
met
10
Absolute score methods (1)
Overall total met total triggered x 100
54
11
Absolute score methods
Patient Average
Indicator Average
12
Results Graph of 5 types of quality score for
practices in dataset A
13
Results Graph of 5 types of quality score for
practices in dataset B
14
Effect on Rank Position
Dataset A Rank positions changed markedly for
40-50 of practices. Dataset B All-or-none gave
markedly different rank positions for 20 All
other methods gave quite similar rank positions.
15
Top and bottom quartiles of practices from
dataset A
Practices in red appear in both top and bottom
quartiles
16
Standard errors of measurement
17
Comparison of methods by disease condition
18
Conclusions (1)

• Different ways of aggregating the data can
  produce different scores/rankings
• In a pay-for performance scheme, providers may
  get quite different rewards depending upon the
  method used
• Incentive schemes need a composite measure that
  accurately reflects the schemes objectives

19
Conclusions (2)

• Agreement varies by dataset
• Different methods have different pros/cons
• Criteria-referenced scores/ranks less stable
• All-or-none scores suggest quality very low
• However low scores allow room for improvement!

20
Factors that influence agreement between
absolute-score methods

• infrequently triggered indicators
• Variation in indicator and patient pass rates
• Associations between rates at which indicators
  are triggered and rates at which they are met

(Note If all indicators are triggered by all
patients then, Patient average Indicator
average Overall )
21
Pros and Cons of the various methods
22
Question
Is there a better method? Eg Item Response
model But Can IRT cope with such sparse
matrices?