Emergency Preparedness: On Shaky Ground

1
Emergency PreparednessOn Shaky Ground

• NICSA West Coast Regional Meeting
• May 31, 2006

2
Panel Members
Wayne Behrens, Director, World Wide, Business
Continuity Planning Franklin Templeton
Investments Stephen J. Dodson, Chief Operating
OfficerParnassus Investments David Rosenberg,
Vice President, Portfolio Operations
AnalyticsCharles Schwab Investment
Management John Sensenich, Director, Product
Management SunGard Availability Services
3
Business Imperative Availability

• Your goal is to ensure that information is
  available wherever/whenever it is needed.
• Overriding Challenge
• - Maximize value and productivity from
  technology investments
• - Minimize potential impact of any disruption to
  daily business
• Focus on your primary business drivers
• Customer Satisfaction

4
Market Drivers
Current market trends include

• Tolerance for downtime
• Technology costs
• Technology options
• Demand for information
• Regulatory environment

5
Regulatory Convergence Risk

• Business goals are
• to assess risks correctly
• to allocate resources efficiently for risk
  mitigation
• Effective management of information assets has
• tangible payoff
• yields more resilient organization
• more cost-effective to deploy comprehensive
  program rather than scattered point solutions

6
Whats Changed?

• Ubiquity of technology
• The new economy
• Changes in the threat landscape
• Consolidation and new risk
• Erupting corporate scandals

7
New Categories of Risk
Interconnected Economy
CIO
Governance Management
Privacy Info Security
Liability for IT Failures
8
Emergence of Convergence

• Regulatory
• Requirements
• Authenticity
• Long-term preservation, availability
• Tighter control over records

• Business
• Requirements
• On-Line Access
• Ease of management
• Business continuity
• Lower TCO

The Challenge Address both, with limited
financial and human resources
9
Information Un-Availability
A Harris Interactive Poll showed that most
Fortune 1000 executives already have experience
with network disruption.
Harris Interactive Poll, Disaster Preparedness
and Information Availability in Post 9/11
Corporate America
10
Todays Business Threats

• Then
• Fires
• Hurricanes
• Tornadoes
• Earthquakes
• Floods
• Power failure

• Now
• Cybercrime and denial of service
• Terrorist targets of opportunity
• Wireless devices
• Trading partner connectivity
• Public infrastructure concerns (telcos, airlines,
  globalization)
• Protection of human capital

Characteristics Intentional, difficult to
quantify, ambiguous boundaries, trust dimensions
Characteristics Statistically predictable,
quantifiable, insurable, well-understood
11
Common Disasters
Source Harris Interactive / SunGard
12
SunGard Disaster Experience - Declaration Events
13
Worldwide Widespread Disasters Only
Widespread Disasters SunGard Alerts SunGard
Declarations 1992 - Chicago Flood 10 5 1992 -
Hurricane Andrew 18 0 1992 - Hurricane
Iniki 6 0 1993 - World Trade Center
Bombing 0 13 1994 - Northridge
Earthquake 14 6 1995 - Hurricane
Opal 26 6 1996 - East Coast Blizard 6 2 1997
- Grand Forks Flood 0 4 1998 - Canadian Ice
Storm 4 7 1998 - Hurricane Georges 75 25 1999
- Hurricane Floyd 189 58 2000 - Wall Street
Bomb 0 2 2000 - Ft. Worth Tornado 0 2 2001 -
Seattle Earthquake 6 4 2001 - September 11
Attacks 105 121 2002 - Pre-Winter Ice
Storm 5 5 2003 Northeast Power
Outage 155 66 2003 Hurricane
Isabel 216 4 2004 - British Telecom
Fire 11 7 2004 Hurricane Charley 111 10 2004
Hurricane Frances 231 37 2004 Hurricane
Ivan 281 15 2004 Hurricane Jeanne 144 18 200
5 Hurricane Dennis 97 7 2005 London
Bombings 84 28 2005 Hurricane
Katrina 128 32 2005 Hurricane
Rita 153 27 2005 Hurricane Wilma 111 21
Totals 2386 532
14
Business Continuity Strategy
What do you think is currently the weakest link
in your continuity strategy, planning and
recovery efforts?
15
Business Continuity Strategy

• Information Availability
• Uninterrupted access to critical systems/data
  that run business operations
• Range of options for less time sensitive
  applications
• Delivered with local access to global
  capabilities
• Provisioned through a combination of
• Business continuity services network,
  facilities, software, mobile solutions
• Managed IT services Managed storage, security,
  monitoring, outsourcing
• Professional services information availability
  assessment, business impact analysis, security
  assessments

16
Philosophy

• Our ability to successfully respond to a business
  interruption is directly related and dependent on
  the planning process. By identifying the risks,
  the dependencies, and mission critical activities
  we are able to craft a flexible plan that ensures
  that we are prepared to quickly respond to any
  event that reduces business operations to an
  unacceptable level.
• In the event of an interruption Charles Schwab
  Investment Managements (CSIM) priorities are
• Safety and Welfare of Employees
• Restoration of Mission Critical Functions
• Restoration of Deferrable Functions

17
Key Plan Components

• Data Back-Up and Recovery
• Mission Critical Applications Systems
• Financial Operational Assessment
• Alternate Communications between CSIM and its
  Customers
• Alternate Communications between CSIM and its
  Employees
• Business Constituent, Bank Counterparty impact
• Regulatory Reporting Communication with
  Regulators

18
Defining Mission Critical

• The SEC NASD generally define Mission Critical
  to have the following meaning
• Any system or data that is necessary, depending
  on the nature of the members business, to ensure
  prompt and accurate processing of securities
  transactions, including but not limited to, order
  taking, order entry, execution, comparison,
  allocation, clearance and settlement of
  securities transactions, the maintenance of
  customer accounts, access to customer accounts
  and delivery of funds and securities.
• CSIM has defined its mission critical activities
  as follows
• Ability to invest available cash and maturities
  on a daily basis in appropriate securities for
  the investment mandate
• Ability to complete trade execution, delivery of
  securities and settlement of cash
• Ability to properly account for securities in the
  funds books and records
• Ability to calculate and distribute each Funds
  daily Net Asset Value
• Ability to fulfill any regulatory obligations

19
Additional Considerations

• Business Continuity Planning should also
  contemplate the following scenarios
• Length of Business Disruption
• Short-term business disruption (less than 24
  hours)
• Medium term disruption (24 72 hours)
• Our facilities but not technology are impacted
• Our technology is impacted but not facilities
• Both technology and facilities are impacted
• Long-term disruption (72 hours)
• Plan in the absence of telecommunications or
  technology
• Absence of key personnel
• Manual procedures for all mission critical
  activities
• Mission critical vendor business disruption
• Policy in responding to industry-wide events
  (e.g. failure of DTC, etc)

20
Preparing Employees

• CSIM BCM Organization
• Emergency Recovery Coordinators
• Business Unit Representatives
• Floor Wardens
• Emergency Response Team
• Communication
• Telephone Systems
• BCM Hotline
• Call Trees
• Email

• Education
• Monthly Emails
• Brown Bag Lunches
• Corporate Sponsored Events
• Testing
• Call Tree Tests
• Annual Relocation Testing
• Ad Hoc Tests
• Exercises
• Table Top Exercises
• Business Impact Analysis
• Plan Review and Updates

21
Corporate Business Continuity Management
22
Corporate Support

• Manage relationship with SunGard the primary
  provider of recovery locations for Schwab
• Manage relationships with other key recovery
  vendors (e.g. Iron Mountain) who are critical to
  the recovery process
• Corporate BCM Hotline and Emergency Notification
  System
• Sponsor annual relocation tests
• Coordination of the Emergency Response Team and
  if needed the Emergency Command Center
• Provide resources and guidance as it relates to
  the creation of business recovery plans
• Corporate BCM team works with Schwabs Executive
  Committee to secure the necessary resources to
  support the firms efforts.

23
A high level summary of the lessons learned by
Franklin Templeton Investments from Hurricane
Wilma
24
Hurricane Wilma Debrief

• Who was Wilma
• Impact of Wilma
• What went right
• Crisis Management Lessons
• Business Continuity Lessons
• Technology Lessons
• Facility Lessons

25
Who the Heck was Wilma?

• Hurricane Wilma is the lesser known cousin of
  Katrina. However, Wilma was the most intense
  hurricane ever recorded in the Atlantic basin, it
  devastated parts of the Yucatán Peninsula as well
  as southern Florida.
• 62 deaths were attributed to Wilma and damage is
  estimated at 12.2 billion in the U.S., making
  Wilma the sixth costliest storm in U.S history.
• When Wilma reached Ft. Lauderdale, she was a
  category 2 hurricane with sustained winds of 110
  mph to 130 MPH.

26
Impact of Wilma on Franklin

• Wilma reached Ft Lauderdale early Monday morning.
  By Monday afternoon we had received preliminary
  damage assessments indicating that hundreds of
  windows were blown out of our building, mostly on
  the north west section of the building.
• We were unable to occupy the building from
  October 24 through November 18.
• No loss of life and no injures to employees.
  Some damage to a few employees homes. Many
  employees homes were without power for days or
  weeks.
• 500 Employees were impacted, with over 230 being
  relocated to other sites Toronto, St.
  Petersburg, Miami, New York, Short Hills, San
  Mateo, and Rancho Cordova.   

27
What Went Right

• In general, our Crisis Management, Business
  Continuity and Technology Disaster Recovery plans
  worked well and Wilma caused virtually no
  disruption to our customers.
• The Ft Lauderdale Emergency Management Team took
  charge of the situation in Ft Lauderdale.

• All business units were able to follow their
  business continuity plans and recover their
  operations.
• Technology operations were restored to include
  two critical applications.

28
Crisis Management Lessons

• When the local Emergency Management Teams (EMTs)
  are in the midst of the incident and working
  literally in the dark, they do not always have
  the ability to fully coordinate the local
  recovery operations for the site.
• Each of the major roles in our crisis management
  teams need to have a specific checklist.

• We need put in place a pre-plan to track and deal
  immediately with the relocation of employees to
  other sites.
• You can not over communicate. Despite the fact
  that we tried very hard to be proactive in
  communicating to employees we still heard a
  number of complaints in this area.

29
Business Continuity Lessons

• Over 40 laptops were left in the FTL office when
  the hurricane hit. There had been so many
  hurricane threats during the year many employees
  did not feel that Wilma would really hit, or
  cause this much damage if it did.

• The standard for Business Continuity Plans needs
  to be expanded to address a month long outage.
• Shifting work to alternate sites for the first
  3-5 days without moving employees.
• Plan for an incident to last over a month to
  include a month end.
• A number of issues arose in regards to employees.
• Pay during the outage.
• Bonus for extraordinary efforts.
• Relocation of children elders
• Childcare and Schooling

30
Technology Lessons Learned

• We were surprised by the number of business units
  that still relied upon hard copy of faxed
  documents. We need a better process and
  documentation of FAX rerouting requirements.
• We need to establish better written procedures
  for forwarding 800 numbers

• There is a desire from the business for a better
  disaster recovery solution for email and
  Blackberry servers.
• Environmental monitoring of server rooms needs to
  be tied into a central control point to insure it
  is remotely accessible.

31
Facilities Lessons Learned

• We should have drilled more rigorously on damage
  assessments. For the first couple of days, the
  damage assessments were verbal and led us to
  believe the damage was much more extensive than
  it was. It turned out only 15 to 20 of the work
  areas were damaged.
• Conversely, our repair and re-occupancy time
  estimates were wildly optimistic.

• Based on initial reports, we planned for a
  week long disruption. It turned out we were out
  for almost a month
• Need to have a working knowledge prior to the
  incident of what local agencies will require to
  re-occupy a building
• Keep in mind the fire marshal and the building
  inspectors are not always in sync.

32
Closing
In the end it was the knowledge, flexibility and
perseverance of our people who really carried the
day and made the recovery a success.