Evaluation of Hospital Drills: Using the Tool

1
Evaluation of Hospital Drills Using the Tool

• Amy Kaji, MD, MPH
• November 16th, 2005
• Acute Care College Medical Student Seminar

2
Hospital Disaster Drills

• Why are drills necessary?
• Hospitals will be called upon to provide care to
  the ill, injured, exposed, and concerned
• Joint Commission on Accreditation of Healthcare
  Organizations (JCAHO) Requirement
• May help train employees
• Tests aspects of hospital response

3
Hospital Disaster Drills

• Types
• Computer simulation
• Tabletop exercises
• Operationalized drills involving specific victim
  scenarios
• Evaluations
• Can help maximize the value of the drill
• Based on accurate observation
• Benefit of standardization

4
Johns Hopkins University Evidence-based Practice
Center (JHU EPC)

• Developed an evaluation tool for hospital drills
• Assess impact of drill as hospital response
  progresses and develops
• Presents topics for evaluation in a systematic
  manner
• Identify strengths and weaknesses in hospital
  disaster preparedness
• Promotes targeted efforts to strengthen
  preparedness
• NOT intended to be used to determine whether a
  hospital passes or fails in its planning endeavors

5
The Evaluation Modules

• Developed by a multi-disciplinary team of experts
  at JHU EPC
• Based upon systematic review of published reports
  on hospital disaster drills and literature
  relevant to bioterrorism preparedness
• Had expert input from federal, state, and local
  agencies
• Piloted the modules in two multi-hospital
  regional disaster drills in the summer of 2003

6
Guiding Principles in Developing Evaluation
Modules

• Need for observing multiple hospital zones
• Evaluation of a disaster drill requires an
  understanding of drill activities in all areas of
  the hospital
• Four zones identified include
• Incident Command
• Decontamination
• Triage
• Treatment
• Addendum if Biological or Radiological Scenario
• Decontamination Zone Module needed for radiation
  chemical drills not recommended in biological
  drill

7
Guiding Principles in Developing the Evaluation
Modules

• Need for documentation of time points
• Recording time points of drill activities is a
  widely accepted method of evaluation
• Labor-intensive if excessive number of time
  points and may detract from overall evaluation
• Limited, specific time points thus predetermined

8
Guiding Principles in Developing the Evaluation
Modules

• Need for documenting clinical care outcomes
• Track the volume of victims in each zone and
  adequacy of provisions made for them, including
  space, staff, supplies, etc.
• Modules are not intended to collect individual
  victim level data
• Modules monitor the zone and outcome for the zone
  as a whole, not for each victim

9
Guiding Principles in Developing the Evaluation
Modules

• Need for debriefing (after-action review)
• Obtain feedback from participants, including
  organizers, staff, and victims
• Allows for discussion of issues that span more
  than one zone
• Evaluate and integrate cross-zone issues at a
  post-drill debriefing session

10
Guiding Principles in Developing the Evaluation
Modules

• Need for ease and flexibility of care
• Designed to be readily understood, easy to use,
  and applicable to many different drill scenarios
• Items on form are arranged by subject, and this
  is the same across all modules to facilitate
  analysis
• Decision to include a module is dependent upon
  the specific scenario and drill

11
Guiding Principles in Developing the Evaluation
Modules

• Need for safety and security
• Consider planned drill activities i.e., use of
  unfamiliar equipment
• Protect actual patients on the premises
• Safety of drill victims and healthcare workers
• Contingency plan to stop the drill in case of an
  actual emergency
• Consider designating a safety officer to monitor
  the drill and its participants

12
Recommended Use of Evaluation Modules and Addenda
( X USE IT!)
13
Internal Structure of Evaluation Modules

• Zone forms for Incident Command, Decontamination,
  Triage, and Treatment have same structure and
  subject headers
• Time points
• Zone description
• Personnel
• Zone operations
• Communications
• Information flow
• Security
• Victim decontamination and tracking
• Victim flow
• Personal protective equipment (PPE) and Safety
• Equipment and Supplies
• Rotation of Staff
• Zone disruption

14
Internal Structure of Evaluation Modules

• Numbering of questions on the forms
• C denotes questions that are common to several
  zones
• Questions that are zone-specific are identified
  by a unique letter code
• 120 questions per module
• Coding on the forms and use of comment boxes
• Yes Y No N Unclear U Not applicable NA
• NA is not a substitute for missing information,
  negative information, or to avoid writing a
  comment
• Color coding for the modules and addenda
• Assists in organizing and tracking modules and
  addenda before and after a drill

15
Common Sample Questions from an Evaluation Module
(Time Points)

• C1. Time the drill began ___AM/PM/U
• C2. Time the hospital disaster plan was initiated
  in this zone ___AM/PM/U/Not initiated
• C3. Time this zone was ready to accept
  victims ___AM/PM/U
• C4. Time when this zone was notified that
  incident command was operational___AM/PM/U/Not
  notified
• C5. Time the drill ended in this zone ___AM/PM/U

16
Common Sample Questions from an Evaluation Module
(Zone Description)

• C8. Was the boundary for this zone defined? Y/N/U
• C9. If this zone had a defined boundary, how was
  it defined? (Check all that apply)
• Barricade
• Security personnel
• Sign
• Tape
• Vehicle
• Wall (permanent or temporary)
• No boundary
• Other (specify) __________________

17
Common Sample Questions from an Evaluation Module
(Personnel)

• C11. Did someone take charge of this zone?
  Y / N / U   
• C12. If someone took charge of this zone, how
  many minutes after the drill activities in this
  zone began did this person take charge? (Check
  one)  
• O lt 10 min
• O 10 - 29 min
• O 30 - 59 min 
• O 1 - 2 hrs
• O gt 2 hrs
• O NA   
• C13. If someone took charge of this zone, was it
  the officially designated person?
• Y / N / U / NA

18
Common Sample Question from an Evaluation Module
(Personnel)

• C14. How was the person in charge of the zone
  identified? (Check all that apply)
• a.    Arm band
• b.    Hat
• c.    Name tag 
• d.    Verbal statement
• e.    Vest
• f.     Not identified 
• g.    Other physical identification (specify)
  __________________________________________

19
Common Sample Questions from Evaluation Module
(Zone Operations)

• C29. Was the space allocated for the zone
  adequate? Y/N/U
• C30. If not enough space for the zone, where did
  zone activities overflow to? (Check all that
  apply)
• Adequate space allotted
• Conference room
• Hallways
• Outside hospital
• Treatment/victim care areas
• Waiting rooms
• No overflow
• NA
• Other (specify) ______________________________
• C31. Was this zone used for the same functions
  during non-drill operations? Y/N/U

20
Common Sample Questions from Evaluation Module
(Zone Operations)

• C33. Did clinical staff interact directly with
  families of victims? Y/N/U/NA
• C34. Were families of victims referred to
  specially designated staff? Y/N/U/NA
• C35. How was victims privacy ensured? (Check all
  that apply)
• Curtains
• Individual areas
• Privacy screens
• Not ensured
• Other (specify) ____________________________

21
Common Sample Questions from Evaluation Module
(Communications)

• Communication device (s) If device not present ,
  circle N in column a and go to the next line.
  
• A. Was device present? Y/N/U
• B. If present, available
• C. If present, was it used in drill? Y/N/U
• D. Comments (note problems)
• C36. 2 way radio/phones
• C37. Direct line
• C42. Numeric paging
• C44. Text paging
• C45. E-mail and internet access
• C47. Intercom
• C49. Runner

22
Common Sample Questions from Evaluation Module
(Information Flow)

• C54. How was this zone notified of the event?
  (Check all that apply)
• FAX
• Runner
• Telephone
• Not notified
• Other (Specify)__________
• C55. Who notified this zone of the event?
• Drill organizer
• Incident command center
• Media
• Other hospital staff
• Outside source
• Victims arriving
• Not notified
• Other (specify)____________

23
Common Sample Questions from Evaluation Module
(Security)

• C60. Were security present in this zone? Y/N/U
• C62. If security personnel were present, what
  type of security? (Check all that apply and
  provide approximate numbers)
• FBI ___
• Hospital Security ___
• Local Police ___
• State Police ___
• NA
• Other (specify) ______________

24
Common Sample Questions from Evaluation Module
(Victim Documentation and Tracking)

• C74. Were all incoming victims registered and
  given a unique identification or medical record
  number? (check one)
• Yes, before entering this zone
• Yes, on entering this zone
• No, not while in this zone
• Unclear
• C76. Was a central list of victims generated for
  this zone? Y/N/U
• C77. Were the triage markers on the victims
  clearly visible? Y/N/U

25
Common Sample Questions from Evaluation Module
(Victim Flow)

• C81. Did a bottleneck develop in this zone?
  Y/N/U
• C84. Were the paths leading to the next zone
  marked? Y/N/U
• C86. Were the lowest acuity victims directed by
  staff to an area separate from higher acuity
  victims? Y/N/U
• C87. What proportion of victims had treatment
  delayed because of zone staffing shortage? (Check
  one)
• None
• Less than half
• At least half (but not all)
• All
• Unclear

26
Common Sample Questions from Evaluation Module
(Personal Protective Equipment and Safety)

• If needed, were these items for standard
  precautions available for the healthcare workers?
  Y/N/U
• Used by staff? Y/N/U
• Adequate supply? Y/N/U
• Eye protection
• Waterproof gowns
• Isolation gowns
• Gloves

27
Common Sample Questions from Evaluation Module
(Equipment and supplies)

• TX15. Were medications needed for the treatment
  of victims available within the
  hospital? Y/N/U/NA
• TX16. Were medications requested from and outside
  source? Y/N/U
• Were needed medical supplies available?
• TX19. Bandages Y/N/U/NA
• TX20. Basic airway equipment Y/N/U/NA
• TX22. Blood drawing supplies Y/N/U/NA
• TX23. Burn Packs Y/N/U/NA
• TX24. Cleaning supplies for contaminated
  equipment
• Y/N/U/NA
• TX25. Crash carts Y/N/U/NA
• TX26. Intravenous fluids Y/N/U/NA

28
Common Sample Questions from Evaluation Module
(Rotation of staff)

• C98. Was there a staff rotation /shift
  change? Y/N/U
• C99. If there was a staff rotation, did the
  officially designated person in charge of the
  zone change? Y/N/U/NA
• C101. What method of shift changing was used?
  (Check one)
• Group shift change
• Staggered shift change
• NA
• Other (specify) _____________
• C102. How were incoming staff updated? (Check all
  that apply)
• Group briefing
• Individual briefing
• Written notes
• Not updated
• NA
• Other (Specify)_____________

29
Common Sample Questions from Evaluation Module
(Zone disruption)

• C103. Was there a plan to relocate this zone if
  necessary? Y/N/U
• C104. Did this zone close at any time during the
  drill? Y/N/U
• If no, disregard the remainder of this section.
  STOP. This zone module is complete
• C105. If the zone closed during the drill, what
  was the reason for closing? (Check all that
  apply)
• Contamination
• Other safety concerns
• Space
• Other (Specify) ________________

30
Description of Modules and Objectives

• Pre-drill Module
• Incident Command Center Zone Module
• Decontamination Zone Module
• Triage Zone Module
• Treatment Zone Module
• Group Debriefing Module

31
Pre-drill Module

• Should be used in all disaster drills during the
  planning stages
• Form is designed to collect the following
• Goals and objectives for the scope of evaluation
• Background information
• Information on areas that hospital wishes to
  evaluate
• Resources required
• If multi-hospital or regional drill, each site
  must work closely with overall coordinators

32
Incident Command Center Zone Module

• Designed to reliably collect information about
  operations of the incident command system (ICS)
• Should be used in all disaster drills when
  evaluating the ICS
• Form is designed to assess the following
• Command structure in the zone
• Adequacy of staffing in the ICS
• Communication and information flow from hospital
  areas to the ICS
• Communication with outside agencies
• Adequacy of security, safety provisions, and
  physical space

33
Decontamination Zone Module

• Designed to collect information re functioning
  of decontamination area
• Used if scenario involves radiation or chemical
  exposure and decontamination is needed
• Form assesses the following
• Command structure in the zone
• Communication and information flow in the zone
• Victim and staff safety in the zone
• Adequacy of staffing and physical space in the
  zone
• Appropriateness of equipment and PPE
• Victim flow in the zone

34
Triage Zone Module

• Designed to collect information re functioning
  of treatment areas
• Should be used whenever drill objectives include
  evaluation of patient care activities beyond
  triage
• Appropriate for use in emergency department-based
  treatment areas or in other clinical areas

35
Triage Zone Module

• Form designed to assess the following
• Command structure in the zone
• Communication and information flow in the zone
• Victim and staff safety in the zone
• Relation of physical characteristics of zone to
  treatment activities
• Efficacy of treatment operations
• Adequacy of materials and supplies in the zone
• Victim flow in the zone

36
Debriefing

• Debriefing is integral part of drill process
• Main objective of debriefing is to identify
  issues not captured by evaluation modules
• Facilitators should create an open,
  non-judgmental atmosphere
• Should occur in all drills to obtain feedback
  from participants and observers

37
Debriefing

• Different approaches to debriefing exist
• One method conduct a session with all
  participants and observers present and ask a
  series of general questions about the drill
• Another method conduct a group debriefing
  session with the participants from that zone

38
Group Debriefing Module

• Contains open-ended questions designed to
  facilitate discussion after completion of drill
• Questions may be added or deleted
• Designed to cover all issues, including incident
  command structure, communications, security,
  decontamination, triage, and treatment
• Documenting the debriefing
• A scribe should be assigned to record
• Videotaping / audiotaping debriefing may help
  capture comments but should not hinder open
  exchange

39
Group Debriefing Module Questions

• Did you feel you were notified of the disaster in
  a timely fashion?
• Did the incident command center work effectively?
• Did anyone receive incorrect information from the
  incident command center? If not correct, what
  specifics do you recall about incorrect
  information?
• Was the information from the incident command
  center received by other zones in a timely way?

40
Group Debriefing Module Questions

• Were there problems with information flow within
  the hospital?
• Were memorandums of understanding with outside
  agencies (e.g., police) activated?
• Did nurses and physicians respond quickly to the
  disaster call?
• Was the zone set up when the first mock victim
  arrived?
• Was security in place before the first mock
  victim arrived?

41
Group Debriefing Module Questions

• Did people have a good understanding of their
  roles, as defined in the disaster plan?
• Did the decontamination system work effectively?
• Did you have any problems with the
  decontamination equipment? Functioning properly?
  Adequate number of units? Participants used
  correctly?
• Were there delays in decontamination? If so, what
  triggered those delays?

42
Description of Addenda

• Four addenda are part of the hospital disaster
  drill evaluation
• Biological Incident Addendum
• Radiation Incident Addendum
• General observation and documentation addendum
• Victim tracking addendum.
• Used to supplement the zone forms
• Example for a radiation exposure drill, the
  Radiation Incident Addendum is added to Incident
  Command Center, Decontamination, Triage, and
  Treatment Zone Modules

43
Biological Incident Addendum

• Designed to collect additional information that
  address response to a biological incident
• Should be added to end of each Incident Command,
  Triage, and Treatment Zone modules
• Should be used in all drills that address a
  biological incident

44
Biological Incident Addendum

• Form is designed to assess the following
• Awareness that biological agent cause of illness
• Whether appropriate personnel were contacted
• Whether health and safety needs of staff were met
• Whether health and safety needs of existing
  patients were met
• Whether health and safety needs of victims were
  met
• Availability of special medications and supplies

45
Biological Incident Addendum

• BA11. Was the Center for Disease Control and
  Prevention (CDC) informed that a biological
  illness was present?
• BA14. Was occupational health informed that a
  biological illness was present?
• BA22. If the cause of the illness was not known
  prior to victim arrival, how long after the first
  victim arrived was the cause of the illness
  identified? (Check one)
• lt1 hour
• 1-4 hours
• 5-8 hours
• gt8 hours
• Never identified
• NA

46
Biological Incident Addendum

• BA24. What resources were used to make the
  diagnosis? (Check all that apply)
• Consultation with an in-hospital expert
• Consultation with an expert from another hospital
• Consultation with an expert from local health
  department
• Consultation with an expert from state health
  department
• Consultation with the CDC
• History and physical exam by the treating
  physician
• Microbiological data
• Radiological data
• NA
• Other (specify) ______________

47
Biological Incident Addendum

• BA26. Was isolation required for the suspected
  biological illness involved? Y/N/U (Isolation is
  required for smallpox, plague, viral hemorrhagic
  fever, certain pneumonias or rashes, and other
  symptoms suggestive of a contagious infectious
  outbreak)
• BA27. If isolation was required, were victims
  transported into an isolation room? Y/N/U
• BA28. If they were transported to an isolation
  room, was the room under negative
  pressure? Y/N/U/NA

48
Biological Incident Addendum

• BA29. Were there enough isolation rooms?
• BA30. If not enough isolation rooms, how were
  victims isolated? (Check all that apply)
• Conversion of other rooms/area (specify)_________
  __
• Existing isolation room in other area
  (specify)__________
• Overflow victims not isolated
• Victims with the same suspected biological
  illness placed in the same isolation room
• NA
• Other

49
Radiation Incident Addendum

• Designed to gather information in response to
  radiation-related incident
• Should be added to end of each zone module
• Should be used in all drills that address
  radiation exposure

50
Radiation Incident Addendum

• Form is designed to assess the following
• Awareness that radiation was cause of illness
• Whether appropriate monitoring personnel were
  contacted
• Whether health and safety needs of staff were met
• Whether health and safety needs of existing
  patients were met
• Whether health and safety needs of victims were
  met
• Availability of special supplies

51
General Observation and Documentation Addendum

• Designed for use by an additional observer to
  document detailed activities in a single unit
• Example
• During a chemical drill, an additional observer
  could be assigned to area where PPE is donned to
  document time required to dress, appropriateness
  of dress, etc.
• Has a front page and a continuation page, which
  may be copied as necessary

52
General Observation and Documentation Addendum

• Instructions Enter time and describe any
  activities in this zone related to drill.
  Include
• Response to patients
• Information received
• Any activities (such as real emergencies) that
  may delay or prevent drill activities in this
  zone

53
Victim Tracking Addendum

• Designed for use by an additional observer to
  track victims through the drill
• May be used within one zone for a large group of
  victims, OR
• Observer can follow victims across zones from
  beginning to end of drill disposition from
  medical perspective
• Useful if there is victim descriptions and
  emergency medical procedures are well delineated
• Has front page continuation page

54
Victim Tracking Addendum
55
Observers

• Value and success of drill, depend on observers,
  who must be able to understand and record events
• Observer selection is therefore critically
  important
• Observers must be trained to use evaluation
  modules
• Documentation by observers provides information
  for evaluation
• Record the type and number of victims, as well as
  the care given or not given to victims
• May record personal views but should note it as
  opinion

56
Role of Observer

• Observers must not have any role other than that
  of evaluating the drill
• Individual who normally works in zone may
  function as an observer, but during the drill,
  he/she may not assist with any activities
• Must not respond to questions from drill
  participants about the drill
• To qualify as observers, volunteers drawn from
  outside the hospital must have knowledge of
  hospital functions

57
Observers

• Background knowledge required
• General knowledge of operations of zone
• Specific medical knowledge not required

58
Observers

• Number of observers needed in each zone
• One observer should be present in each zone
  continuously
• Additional observers may be needed to observe
  numerous staff or victims
• Should use victim tracking or general observation
  and documentation addendum

59
Observers

• Number of observers needed in each zone
• If extensive time point data collection is
  needed, an additional observer should be assigned
  to this task
• Specific time points and reasons for collection
  should be outlined before drill
• If clinical process or outcome data is needed,
  additional observers will be needed
• Observers must have sufficient clinical knowledge
  to report on decision making

60
Observer Training

• Training session
• Receive zone assignments
• Review relevant zone modules
• Achieve familiarity with content of evaluation
  modules and zone configuration
• Explain all questions and response sets
• Address questions about forms
• Instruct how to be an effective observer

61
Final Observer Training Points

• All observations are confidential
• Observer should evaluate drill without
  obstructing flow of drill
• Observers may ask participants questions to
  clarify observations
• Questions should be asked in an unobtrusive
  manner
• Observers should refrain from asking questions
  that may alter drill actions

62
Final Observer Training Points

• Observers must not participate in drill
  activities
• If questioned about a drill issue by a drill
  participant, the observer should state that they
  are evaluating and unable to answer
• Each question on each module should have a
  response
• The response NA should be indicated only when the
  question does not apply

63
References

• Cosgrove SE, Jenckes MW, Kohri D, Hsu EB, Green
  G, Feuerstein CJ, Catlett CL, Robinson KA, Bass
  EB. Evaluation of Hospital Disaster Drills A
  Module-Based Approach. Prepared by Johns Hopkins
  University Evidence-based Practice Center under
  Contract No. 290-02-0018. AHRQ Publication No.
  04-0032. Rockville, MD Agency for Healthcare
  Research and Quality. April 2004.