Risk Informed Evaluation of Patient Safety Training

1
Risk Informed Evaluation of Patient Safety
Training

• Anthony D. Slonim, MD, DrPH
• Vice President Medical Affairs
• Carilion Medical Center
• Senior Staff, Departments of Medicine and
  Pediatrics
• Carilion Clinic
• Professor, Medicine and Pediatrics
• Virginia Tech-Carilion School of Medicine

2
Objectives

• Decision-making science How do we normally make
  decisions?
• Picking up on Level 4Can we train to improve our
  decision making results?

3
Decision Making Science
4
Bayes Theorem

• P (A/B)P(B/A) P(A)
• P(B)

5
Pattern Recognition
Is there flashing in the squares?Your mind will
play tricks on you!
How many squares do you see? Jumping to
conclusions too quickly.
6
Decision-Making

• Medical Decision-Making Process
• Perception/Data gathering (training-H and P,
  Labs, Rads)
• Amber light is showing
• Interpretation (training-pattern recognition and
  probability)
• Amber light means prepare to stop, maybe
• Decision making (based on probability
  experience)
• Stop or go
• Action taking (reflex/gut level
  response/programming)
• Hit the brake or accelerator

Marx D and Slonim AD Assessing patient safety
risk before the injury occurs An Introduction to
Socio-Technical Probabilistic Risk Assessment.
Quality and Safety in Healthcare 2003 12 Suppl
2 33-38.
Marx, D
7
Medical Decision-Making
Perception/Data Gathering
Make a Dx and Treat
X
Interpretation
Action
Get Help Cardiology Consultation
Triage Nurse
ED Nurse
Do more testing-which test? (Pre-test
probabilities)
Physician
Decision-Making
8
Medical Decision-Making
What are the results? Did we get it right?
Perception/Data Gathering
Make a Dx and Treat
X
Interpretation
Action
Get Help Cardiology Consultation
Triage Nurse
ED Nurse
Do more testing-which test? (Pre-test
probabilities)
Physician
Decision-Making
9
Expert Decision MakingPractice, Practice,
Practice

• Expert pattern matching against large mental
  library, quick, accurate if confirm correct
  answer
• Novice library is empty slow, error prone
  process
• Certain Diagnoses are Favored- Frequent, Recent,
  Serious
• Heuristics fixating on the wrong pattern

10
Pattern Recognition
11
Picking up on Level 4Can we train for results?
12
Kirkpatricks Levels

• Level I Reactions
• How well trainees liked training
• Level II Learning
• The extent to which trainees understand and
  retain principles, facts, and techniques
• Level III Behavior
• The extent to which behavior changes as a result
  of training
• Level IV Results
• Impact of training on organizational criteria

13
Elements of Quality Programs
Program Identification Prioritization
Quality Functions
Data Analytics
Process Improvements
Change Management

Improved Outcomes
Research
Education / Training
14
Why is there a safety problem ?

• Considerable variation in practice
• Based on opinion or consensus
• Evidence-based guidelines-unsupported
• Failure to create fail-safe processes
• Our providers may not know their work
• Policies and procedures
• Were learning to work together
• Were not sure of the results were looking for

15
Process Analysis

• Processes
• A series of sequential steps governing
  interactions
• Between patients and providers
• Between providers and providers
• Examples of process analysis techniques
• Root cause analysis-retrospective
• HAACP (hazard analysis and crit control points)
• FMEA (failure mode effects)
• PI methodology

16
Low-frequency, High Impact Events

• Low frequency, high-impact events
• Variable processes and practices
• Wrong site surgery
• The abduction of children from hospitals
• Deaths or major harm
• Process analysis helps to identify risk and
  prioritize interventions
• Decision support helps to guide decision making

17
Probabilistic Risk Assessment

• A hybrid between process analysis and decision
  support
• Identifies risk points and directs to
  interventions
• Is hierarchical and probabilistic
• Allows disentanglement of patient level risks,
  provider level risks, and system level risks
• Assigns probabilities for prioritization of risk
  reduction strategies
• Includes sociotechnical components into the models

18
Conceptual Framework
Probabilistic Risk Assessment
The Institution
The Providers
Quantitative Methods
Qualitative Methods
The Patient
19
The Prospective Risk Model
The Top Three Risks
20
Training Evaluation

• Definition
• The systematic collection of descriptive and
  judgmental information necessary to make
  decisions related to instructional activities
• Ensures training
• Meets its stated objectives
• Changes trainee attitudes
• Increases trainee knowledge
• Develops trainee skills
• Transfers results to the job

21
Training Evaluation

• Important variables to consider
• Organizational Factors
• Individual Factors
• Trainee Knowledge, Skills, and Attitudes
• Training Transfer
• Organizational Outcomes

22
Merging Kirkpatrick and ST PRA

• Socio-Technical Probabilistic Risk Assessment
• Good for examining low base rate events (Six
  Sigma)
• Models contributing causes
• Procedural tasks
• Team tasks
• Identifies the impact of an intervention
• Evidence base
• Empirically based
• Adjust and test the model
• Monte Carlo
• Changes in the likelihood of outcomes

23
Traditional Approaches

• Quick Wins
• Fire-fighting
• Burn-out / Fatigue
• Difficult-to-Sustain, Short-Term Results

Impact
Typical Results
Time
24
Quality Fusion Approach
Quality Fusion Results
Impact
Typical Results
Time
25
Example
26
What is Escalation?

• Failure to rescue associated with
• Interpretation problems
• Throughput problems
• Put another way
• When you do not realize the patient is in trouble
  OR you know the patient is in trouble, but you
  dont respond as needed.

27
Common Course
Ideally, we track the illness. As the patient
gets worse (line goes up), we respond. As the
patient improves we adjust.
The patient condition


The provider team response
28
Going Off Course
The defect rate in our model is caused by
failures to properly track the course of the
illness.
The patient condition

The provider team response

29
Never On Course
Sometimes, were off course right from the
beginning and its difficult to get back on
course.
The patient condition


The provider team response
30
The Prospective Risk Model
The Top Three Risks
31
Conclusions

• A focus on results helps providers and patients
• Training on risk points can improve performance
• Leads to better results
• Requires alterations in decision making
• Enhances empiric data for better understanding
  training