Disaster Recovery Planning in Florida National Association for Public Health Information Technology

1
Disaster Recovery Planning in FloridaNational
Association for Public Health Information
TechnologyOctober 19, 2005Debbie
SatterwhiteKristin GreenFlorida Department of
Health
2
Disaster Recovery Planning in Florida

• Decentralized environment
• Windows and UNIX servers - no mainframe
• Approximately 400 sites statewide - 11 HQ
  Divisions, 67 County Health Depts, 22 Childrens
  Medical Services offices, Labs, Vital Statistics,
  Pharmacy, TB Hospital, Environmental Health units
• DR IT Personnel 1 at HQ, 11 regional Disaster
  Preparedness Consultants (DPCs)
• DR Team Building with part-time representatives
  from all areas of IT an on-going process
• Terminology
• COOP overall agency continuity of operations
  plan
• COOP-IT the IT piece of COOP recovery of
  critical computer applications and essential IT
  services
• COOP-IT Plans
• Template (used statewide) with process and
  appendices for data
• Appendices include notification lists, assembly
  points/alternate sites, Incident Command teams,
  essential services/personnel, critical
  applications, recovery library info
  (hardware/software inventories, backup tapes),
  procedures, drive away kits, and network info
• Appendices collect documentation normally
  maintained by the IT sections (where
  possible, they try to avoid re-creating the
  wheel)
• Required for each site hosting servers
  approximately 100 department wide

3
Critical Computer Applications

• Business units and HQ management reviewed
  computer applications to determine the
  criticality and the Recovery Time Objectives
• DOH Recovery Time Objective
• Priority 1 (Mission Critical) 1 hour
• Priority 2 (Mission Essential)
  12 hours
• Priority 3 (Business Unit Essential) 3-5
  days
• Priority 4 (Important, not time sensitive)
  not recovered
• during the disaster window
• Note we had originally planned to recover all
  applications, but after the 2004 hurricane
  season, the business units determined all
  applications were not needed and list was cut to
  about a third of the original size.

4
Backup Site

• FDOH establishes in-house recovery site
• site rated to withstand Cat 5 hurricane
• capable of hosting all critical applications
• battery backup, diesel generator and large
  capacity fuel tank
• 14 racks with virtual servers, SANs, email
  archiving, tape safes
• Used for HQ testing and recovery/failover
• Used for field office testing
• Provides a field office recovery option

5
Mobile testing equipment

• 5 sets of equipment
• includes server and tape backup unit in portable
  racks
• generators
• shared by 2 DPC regions
• Satellite equipment added in 2005
• Provides field sites (CHDs and CMS offices) IT
  equipment for annual disaster tests without
  impacting normal work
• Testing is coordinated and scheduled by 11 DPCs
• Minimizes employee time and travel requirements
• Provides production-ready recovery alternatives

6
Testing DR Plans

• Testing is the KEY!!!
• Shows if applications can be successfully
  recovered
• Identifies problem areas many lessons learned
• Allows IT and business unit personnel to
  participate and learn
• Testing vs. Practicing (a practice is less
  intimidating)
• Training employees should occur before a
  test/practice of the plan
• Cross-train back-up employees

7
Hurricanes

• Uses a special subset of the plan with no
  relocation of staff or processing (to backup
  site)
• IT Incident Command structure and teams activated
  and essential staff identified/scheduled
  (Incident Commander, Operations, Planning,
  Finance/Admin and Logistics/Facilities)
• Service Desk/On-Call staff an essential function
• Hurricane activities tracked thru Service Desk
  tickets
• Deployments to hurricane area tracked thru EOC
  missions
• Supports activation of the departments COOP
• Supports the departments responsibilities for
  Emergency Support Function (ESF) 8 - Health
• Supports the departments IMT (Incident
  Management Team) and states EOC (Emergency
  Operations Center)

8
Other Lessons Learned

• To make DR planning a part of daily activities
  for
• GOVERNANCE determine criticality of new
  applications up front priority 1 applications
  require redundancy and impact project cost
• CHANGE MANAGEMENT include assessment of DR
  impacts as new applications are rolled out and
  enhanced
• DR planners to work more closely with department
  COOP planners and facility staff for
• What computer applications are needed to support
  a business units essential functions?
• Where does a business unit plan to work if
  primary facilities are inoperable? (network
  connectivity issues and facility issues)