When Things Go Boom At Work

1
When Things Go Boom At Work!

• Disaster Management
• at
• BCIT
• CANHEIT 2005 McGill University

2
Disaster Planning Certainties

• There is nothing like a good disaster to get the
  attention (and money) of senior executives and
  the Board.
• Your organization is never fully ready or
  prepared for a disaster of any magnitude!

3
When Things Go Boom At Work

• What happened
• On April 13th,2004 at approximately 220pm a
  steam boiler attached to NE6 exploded
• Adjacent buildings sustained significant blast
  and debris damage
• Utilities (water, electricity, fire surpression,
  alarms) were impacted
• Injuries were minor and few

4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
The Aftermath

• Buildings nearby were evacuated and isolated
  pending safety investigation
• Power shutdown to NE1, NE4, NE5, NE6, NE25, NE27
• Fire Suppression and Security compromised
• Significant network loss in most north side
  buildings

9
Educational Programs

• Classes cancelled
• All night school and day school on North side
  next day
• Classes in damaged buildings relocated where
  possible
• Delay in permanent power to NE27 impacted ability
  for program completion
• NE4 and NE6 programs without specialized
  equipment for weeks

10
Work Interruptions

• NE1
• Financial Services, Human Resources (30 offices,
  1 day)
• NE9
• Facilities Management, Purchasing, Warehouse
  Operations (50 offices, 2 days)
• NE25
• Research Center, Internet Engineering Lab, GAIT
  Lab (26 offices, 10 days temporary offices)

11

• ATM GigE
• fibre backbone
• to 10 core hub
• locations
• SE12 is central
• hub to Internet
• and all other
• campus links

12
IT Issues

• Network hub locations for North side had no power
  no communications for most of North buildings.
• Access to hub locations severely restricted
• No access initially, then only with environment
  suits
• Passive fiber patching necessary to bypass

13
(No Transcript)
14
IT Issues

• Staff workstations in quarantined areas
• Temporary work areas needed
• Replacement workstations for 20 staff
• Data stored on local drives not available

15
Lessons Learned..

• There are never enough helpers around when you
  need them
• The aftermath is often as bad or worse than the
  actual disaster
• Communications are critical
• Ensure the data and voice infrastructure have
  redundancy
• Saving data to network drives is good backups
  are better!

16
Disaster Planning at BCIT

• Disaster Management Team established in 2002,
  using ICS structure
• Institute Plan well under way but still not
  complete
• Small steps are better than no steps
• Planning for business continuity is the
  responsibility of all areas, not just IT
• Do scenario planning tabletop or simulation

17
Business Continuity

• Executive Sponsorship critical
• Establish a working committee
• Do an institutional assessment
• Divide and conquer spread the work
• Centralize all plans
• Maintenance and testing
• Communicate, communicate, communicate

18
Develop the Plans

• Start with overall business continuity checklist
  and assessment
• Consider threats by type
• Natural
• Human/Personnel
• Technical

19
Develop the Plan

• Identify Business Unit Processes and Responses to
  Threats
• Operational Risks
• Technology Risks
• Legal Risks
• Financial Risks
• Reputation Risks
• Strategic Market Risks

20
Business Continuity Methodology Proposal for
BCIT
Phase I Data Gathering and Analysis
Phase II Solutions Development
Phase III Solutions Implement
KEY FACTORS ET/IT Plan People Data Apps
Software Servers Comm. Physical
Utilities Prevention
Resource Requirements
Projects Classification
Risk Assessment
Business Impact Analysis
Contracts and Provisioning
Disaster Avoidance Strategies
Operational Impact Analysis
DRP Management
Recovery Strategies
Financial Impact Analysis
DRP Training Testing
Strategy Solutions Recommendation
Business Requirements
DRP Maintenance
21
GET STARTED !

• The next disaster is just around the corner.

22
Some Useful Sites

• http//www.ncsu.edu/ehs/BCP/index.php
• NC State Business Continuity Plan
• http//dri.org
• Disaster Recovery Institute
• http//www.drj.com
• Disaster Recovery Journal

23
Contact Information

• Ian McLeod, Director, Computer Resources, BCIT
• Ian_McLeod_at_bcit.ca
• [phone]