DISASTER DRILL EVALUATION

1
DISASTER DRILL EVALUATION
Gary B. Green, MD, MPH, FACEP Office of Critical
Event Preparedness and Response (CEPAR) Johns
Hopkins University, Baltimore, Maryland
2
The Johns Hopkins University Evidence-based
Practice Center (EPC)supported by the Agency for
Healthcare Research Quality
The Johns Hopkins University Office of Critical
Event Preparedness and Response (CEPAR)
Johns Hopkins University School of Medicine,
Department of Emergency Medicine
3
Objectives

• Understand the critical role of disaster
  simulations (drills) and their evaluation in
  disaster preparedness efforts
• Discuss the data collection instruments, methods,
  and analytical techniques used to perform an
  objective, quantitative disaster drill evaluation
• Describe the challenges faced in the actual
  performance of disaster drill evaluations and
  successful strategies to overcome these challenges

4
Why do disaster drills?

• Training
• Skills-based
• Evaluation
• Raise Awareness
• Administration, Staff, Public
• Team Building
• Fulfill regulatory requirements
• JCAHO, others

5
Learning Opportunities

• A drill affords opportunity to learn
• Emergency Operations (Disaster) Plan
• Gain/Improve Skills-Based Competence
• New, modified job requirements
• Operations in unfamiliar and difficult settings
• Use of infrequently used equipment
• Deal with Large Numbers of Patients
• Teamwork
• Problem Solving, Improvisation

6
Drilling as Formative Evaluation

• "The purpose of evaluation may be
• formative
• (to provide ongoing feedback so that the learners
  or curriculum can improve)
• or
• summative
• (to provide a final "grade" assessment of the
  performance of the learner or curriculum)"
• Kern DE, Thomas PA, Howard DM, Bass EB.
  Curriculum Development for Medical Education A
  Six Step Approach. The Johns Hopkins University
  Press, Baltimore 1998, pp 6-7.

7
Continuous Quality Improvement (CQI)
Process Applied to Disaster Preparedness Capacity
Building
8
Basic Principles of Drill Evaluation

• Drill/Evaluation Planning
• Components of disaster response to be evaluated
• Drill evaluation tools techniques

9
1. Define Drill Goals and Boundaries

• What are Specific Goals for this Drill?
• Clinical response training
• Incident Command System (ICS) effectiveness
• Chemical, biological, radiation incident response
• Efficiency of response
• Define Boundaries of Drill Activities
• Jurisdictions involved
• Single or multi-agency
• Interface with outside agencies?
• Hospital, pre-hospital, both?
• Medical only, law enforcement, others?
• Decontamination activities included?

10
2. Consider Resources Available

• Adequate time before drill?
• Buy-in by key stakeholders?
• Live volunteers available?
• Distinct evaluation team?
• Time and place available for debriefing?
• Continuing normal services during drill?

DO NOT BE AMBITIOUS BEYOND RESOURCE AVAILABILITY!!
11
3. Select Specific System Response Components to
be Evaluated

• Clinical Operations
• Efficiency (Drill Flow)
• Logistics (materials, facilities, transportation)
• Other Operational Components
• SR, Security, PR, etc
• Incident Command System
• Most critical component
• Defines borders of other components
• System Integration
• Interagency coordination, communications
• Human Resources
• Others as defined by local plan

12
4. Select Appropriate Evaluation Methods and
Instruments Based Upon Goals and Resources

• Smart observers
• Smart casualties
• Post-drill interviews
• Group debriefing
• Self-assessment
• Photo, audiotape and/or videotape assessment

• Checklists
• Scribes
• Victim based recording
• Cross-evaluation
• Provider and/or victim flow tracking
• Computer modeling

13
Combined Approach to Evaluation

• Clinical Care
• Trained observers
• Provider recording (triage tags, etc.)
• Smart patients
• Standardized Victim Scenarios
• Drill flow
• Victim Movement
• Victim/Provider interactions
• Entrance/exit observers
• Scene Diagrams
• Photos/Videotape analysis
• Incident Command, System Integration
• Direct observation difficult
• Pattern Analysis of response
• Self assessment cross-evaluation
• Plan Compliance (Qualitative Methods)
• Checklists
• Evaluators narrative comments
• Videotape review

14
Clinical Care Evaluation

• Standardized Victim Scenarios
• Clearly written victim descriptions
• Expert panel consensus process
• Triage Level Primary and Secondary Triage
• Critical Actions Scene, Treatment Area
• Victims moulaged per pre-written scenarios
• Scenarios may change during event
• Victim Based Recording
• Smart Victims
• Observer Based Recording
• 6 victims/observer maximum

15
Victim Scenario Card (Triage Area)



16
2
Zone III
1
17
9
14
Zone I
Zone II
19
20
4
17
Evaluation of Drill Flow

• Assess Efficiency of Operations
• Time-based Data Collection
• Relative rather than absolute times important
• Flow Measured at Key Locations (Bottlenecks)
• Can capture all movements with exact times for
  Victims, Providers and Resources
• Scene entry/exit, decon, triage, treatment areas
• Clear Identification System Critical
• Allows Pattern Analysis

18
Mean Response Times by Triage Level and Zone
19
Evaluation of ICS

• Most Difficult Component to Objectively Evaluate
• Multiple simultaneous communications and
  decisions
• Impractical to directly observe, record time and
  content of every communication
• Observers and/or scribes can capture
  pre-determined key communications and actions
• Pattern Analysis of Drill Flow
• Assess active management of response components
• Assess system response to situational changes
• Structured Self Assessment Cross-Evaluation
• Adequacy, clarity of communications
• Understanding of functional roles

20
Unit Leader Survey

1- Strongly Agree, 2 Neutral, 3 Strongly
Disagree
21
Drill Participant Survey
1- Strongly Agree, 2 Neutral, 3 Strongly
Disagree
22
Closing Perspectives

• Plan drills targeted to specific goals
• Include evaluation in plans for all drills
• Use evaluation methods congruent with goals of
  drill available resources
• Seek adequate support for stronger evaluation
  designs
• Evaluation results should drive all future
  planning and training activities

23
Questions?