Vulnerability of

1
Vulnerability of Evidence Based Guidelines
The Tyranny of Small Numbers
2
Guidelines, Care Plans, Pathways, Protocols,
Order Sets
Starting point for research and a basis for
starting improvement
3
Indications for Operation I
Always the hardest problem Geographic/health
system variations Hysterectomy, T A, but
Kentucky is still 1
- in back surgery
4
Indications for Operation II
Does lowering of Mb Mt permit softening of
IFO? Laparoscopic GB ? HIDA only Laparoscopic
fundoplication or purple roofs
5
Indications for Operation III
Dr. Gray Sr. ? Hysterectomy Hill Only source of
Premarin in 1965 No death last 14 years of a
very busy practice
6
Practice Guidelines I
Standard of practice and the Witness
stand General, vague
Precise detailed
Errors of Omission
7
Practice Guidelines II
Similar to informed consent Organizational views
well-trained, good judgment, collegial
precise protocols
few guidelines
a good doctor better
8
Practice Guidelines III 2006
Carefully stated guidelines with caveats permit a
well-trained surgeon with good judgment
to perform better more consistently
9
nature
Vol 43 20 October 2005
NATURE article slide
35 16 49
of authors said they had a conflict of interest
of some kind
authors helped to write guidelines on illnesses
relevant to companies in which they owned stock.
..bodies that produce guidelines maintain that
there just arent enough experts without
conflicts of interest.
of guidelines did not include any details of
authors conflicts of interest
10
Pages of Confession
11
Who purchased and distributed 500,000 copies?
12
In fact, most published guidelines promulgated by
our best societies/associations fluctuate
between being marginally corrupt or partially
uninformed.
13
Guidelines for the Future
Carefully stated, specific exceptions and
caveats consensus Physicians judgment, honest
second opinions Symbols for industry
influenced Meta-analyses forest, funnel or
fudged
14
HSR has largely ignored elective surgery as a
very good monitor of quality practice Patients
and often MD decide concur Surgeon assesses and
agrees Many outliers intrinsically corrected
Event is focus of many processes Outcome clear
in a few weeks
15
A problem with elective surgery as a study item
in HSR is that it is, defined in traditional
terms, pretty safe and good, and given sensible
indications, associated with few complications
and infrequent deaths
16
In fact, with or without prophylaxis most wounds
heal without infection or hernia most patients
avoid DVT DE most patients do not die and
return to their previous lifestyle in
unimproved health
17
Tyranny of small numbers
Death is the most objective, if least desired,
Small numerators Very large (and not so
homogenous) denominators Statistical
significance, Yates correction and clinical
validity
18
The essential nature of reasonable volumes for
measurement For the surgeon uncommon For the
hospital frequent Captain of the ship that
runs aground or a team builder?
19
Risk adjustment essential for physician and
hospital acceptance Public may never understand
20
Why surrogates or process measures? SCIP
2004 every patient provided 17 events The
5,339 patients permitted potential 80,000
process measures and still yielded few, if
any, strongly defensible outcomes
21
SCIP 2004 deaths after colorectal resections
0/171 2/138 5/167
14/444
7/476
248/5455
Risk stratification and operative complexity
enhance the differences
22
Which would you choose? Would you make the same
choice if it involved doubling your reported
complication rate?
23
Risk stratification, outliers and 95
confidence limits Diagnostic advances and the
advent of previously undetected
pre-existing or succeeding illnesses
24
SCIP 2004 deaths after colorectal resections
32 / 5818

• likely not preventable
• 15 likely preventable

25
Top Five Factors Associated with 32 Deaths Among
5818 SCIP 2004 Patients
Raw Number Risk Factor
Age gt 65 years 25 ASCVD 19 ASA Class 4 or
5 15 COPD 10 Functional dependence 9
all plt0.05
26
Top Five Factors Associated with 32 Deaths Among
5818 SCIP 2004 Patients
Percent Risk Factor
Age gt 65 years 78 ASCVD 59 ASA Class 4 or
5 52 Recent weight loss 41 Functional
dependence 38
all plt0.05
27
Top Five Factors Associated with 32 Deaths Among
5818 SCIP 2004 Patients
Raw Number Process measures
Antibiotic start time 17 Antibiotic stop time
13 Normolthermia 11 Antibiotic choice
8 VTE prophylaxis 6 Glucose gt200 6
-plt0.05
all others plt0.01
28
Top Five Factors Associated with 32 Deaths Among
5818 SCIP 2004 Patients
Percent Risk Factor
Antibiotic start time 63 HOB elevation 56 New
b blockade / vasc 50 New b blockade / CAD
47 Antibiotic stop time 45
plt0.01
too few
p0.09
p0.09
plt0.01
29
Surprises Occurring in 32 Deaths Among 5818 SCIP
2004 Patients
Functional dependence WBC gt 11,000 Prophylactic
antibiotic choice
plt0.05
plt0.05
plt0.01
30
Are the 15 preventable deaths good medicine or
are they the equivalent of 15 private plane
crashes or one crash with 15 passengers?
31
The small number that surgery needs to embrace
and accept internally is zero.
32
Welcome transparency as the corporate we
better surgeons, better nurses, better
hospitals, better health plans Pursue that
ultimate small number Eliminate imperfections on
a personal and systematic basis
33
Better is not a number. Soon is not a time. Try
is permission to fail.