A Disaster Recovery Methodology

1
A Disaster Recovery Methodology
David Frohman and Robert Todd
2
When Disaster Strikes
Are You Ready?
3
Methodology Overview

• Methodology - Introduction to DRM (Disaster
  Recovery Methodology)
• Phases of DRM
• Sample Recovery Plans
• Cases - Successes
• Management Survey
• Salary Survey
• Exercise

4
Objective of DRM
Reduction in Chaos! Provides a Standard and
Consistent Manner for Handling Systems issues in
a Disaster An Information Systems Function
5
Methodology Purpose

• Provide Management with Key Information
• Obtain Management Commitment
• Define Requirements in Terms of Business
  Functions
• Documentation of Impact
• Focus on Impact Identification and Orderly
  Recovery
• Select Balanced Teams
• Develop User-Friendly Contingency Plan
• Integration

6
Types of Disasters

• Operations
• Security
• Data Protection
• Replication and Reliability
• Responsiveness
• Legal

7
Minimizing the Impact

• Recognizing
• Reacting
• Recovering
• Restoration

8
Degree of Complexity?

• What Degree of complexity is needed?
• What are our current recovery strategies?
• What recovery personnel resources are available?
• What is the allowable recovery budget?

9
DRM Phase 1 - Project Initiation

• Obtain the Commitment of Top Management
• Create a Project Team
• Schedule Departmental Interviews (Discovery)
• Communication to Stakeholders
• Essential Personnel Requirements
• Budget Consideration

10
Phase II - Information Gathering (Discovery)

• Recovery Team Description
• Support Staff
• List Key Contacts
• Use, Location of Critical Information
• Telecommunications Requirements
• Discussions on Implementation
• Distribute Questionnaires
• Review Report of Assets

11
Phase III - Business Impact Assessment

• Create Business Impact Report
• Identify Tasks at Hand
• Perform Risk Assessment
• Identify Interactions Between Disaster Components
• Asset Considerations

12
Phase IV - Design(Detailed Definitions of
Requirements)

• Create Recovery Objectives
• Develop Recovery Strategies
• Create Recovery Standards
• Location of Recovery Teams
• Required Workspace
• Establish Priorities
• Communications

13
Shadow System Journaling
Shadow Opens TCP Connection to Server
Shadow Sends Server the First Journal the Name
and Address of the Last Entry Received
Starting Address of Record
NO
Is There New Information?
Wait for New Record
YES
NO
The Server Sends New Journal File Information.
Shadow Updates Databases and Shadow Journal File
End of Journal File ?
YES
NO
Is There Another Server Journal File ?
Server has Reached Address of Last Journal Entry
YES
Next Journal File
14
Phase V - Implementation

• Update Departmental Working Documents
• Communications
• Documentation
• Definition of Recovery Teams
• Recovery Standards

15
Phase VI - Testing and Refinement

• Testing Goals
• Testing Strategies
• Testing Procedures
• Re-evaluation and Updates
• Employee Training

16
Phase VII - Maintenance Program

• Maintenance of Plans
• Must Reflect Changes to Environment
• Periodic Re-evaluation of Plan
• May Require Revisit of earlier Phases
• Changes Must be in Revision
• Ongoing Training
• Interdepartmental/Site Activity Communications

17
Methodology Types

• Organization/Sector Specific Methodology
• Considers Unique Business Needs
• Specific to Implementation
• Can Incorporate Customer and Partner Involvement
• Standardized Methodology
• Universal Adaptability
• Ongoing Updates
• Other Organization Experiences
• Proven Effectiveness
• Hybrid Methodology
• Addresses Both Business Needs and Standardization

18
Sample Disaster Recovery Plans
19
University of Toronto DRP

• A guide to the development of a disaster recovery
  plan.
• Primary objective is to enable an organization
  the ability to survive a disaster and to
  reestablish normal business operations.
• To survive a disaster an organization must be
  able to resume normal business operations in a
  reasonable time frame.
• Believes contingency planning is a business
  objective not a data processing issue.
• An effective recovery plan is a maintained, or
  and ongoing live recovery plan.

20
Colorado State University EOP

• The Emergency Operation Plan was developed for
  CSU to be prepared in the the event of an
  emergency.
• Attempts to guide university personnel to be
  prepared in ways to mitigate hazards, assist in
  response, and complete prompt recovery in the
  event of a disaster.
• Each department has the responsibility of
  creating its own disaster recovery plan.
• Is a generalized plan for coordination, roles and
  responsibilities of university personnel.
• Was developed under the Comprehensive Emergency
  Management methodology.

21
DRJ Internet DRP

• Free low end Disaster Recovery Plan provided to
  users of Disaster Recovery Journal.
• Is intended for individual P.C. users or users of
  a small network.
• Contains a methodology to complement a
  pre-existing DRP.
• Contains sample forms, and checklists, for a
  disaster recovery plan.

22
BS7799 and ISO 17799

• ISO 17799 is based on British Standard BS7799.
• ISO 177992000 defines 127 security controls
  divided into 10 major areas.
• BS7799-21999 is complementary to ISO 17799 and
  is a standard for senior management monitor and
  control their security.
• Third party modular software planning tools are
  available to implement the standard methodologies
  and reach certification.
• One such tool is COBRA.

23
Example Case 1

• Solectron Experiences Applied to DRM

24
Case Study Solectron

• Founded in Milpitas, California in 1977.
• 57,000 employees in 52 locations, 19 countries,
  worldwide.
• Year 2000 revenues 9.2 billion.
• Supply chain facilitator for custom electronics
  manufacturing. Also provides technology,
  manufacturing, and services to other industries.
• Interview with corporate recovery manager Raelene
  Wong.

25
Motivation

• No Regulations in the manufacturing sector to
  ensure disaster preparedness.
• Loss of customer and shareholder confidence if
  unable to manufacture products.
• Result of growth constantly acquiring new sites.

26
Solectron Phase I, Initiation

• Received incredible support from our executive
  management and from the site general managers
  They recognize the importance of this to our
  employees, customers, and shareholders.

27
Solectron Phase II, Discovery

• Initial plan development utilizing consultants.
  Shifted responsibility to internal forces to get
  the ownership and support needed.
• Needed a fast initial implementation so sat down
  with site general managers and determined each
  site should have its own Disaster Response and
  Recovery Program (DRRP) coordinators.

28
Solectron Phase III, BIA

• Purchased software tools to help identify
  critical business functions and threats.
• Utilized internal forces such as the Finance
  department to determine critical business
  functions and costs should a disaster occur.
• Determined business impacts can also be assessed
  by having site coordinators contact local
  governments to determine threats in the region.

29
Solectron Phase IV, Design

• DRRP coordinators design systems to address needs
  such as first hour response and crisis
  management.
• Design global database repository for site
  coordinators to share status reports over the
  web.
• Kept the number of site plans to a minimum to
  avoid confusion of which plan to implement in the
  event of an emergency.
• Design a corporate web site containing sample
  plans, training information, and corporate policy.

30
Solectron Phase V, Implementation

• Initially applied Solectrons methodology to
  corporate headquarters as a basis for
  implementing to other sites.
• New acquisitions are evaluated and introduced to
  policy/methodology for immediate implementation.
• Applies brunt of the work to individuals to
  alleviate stress on the coordinators.
• Global status reports sent to customers.

31
Solectron Phase VI, Testing

• Requires sites to conduct a number of exercises
  per year.
• Includes a full-building evacuation.
• Includes exercises required by local ordinances.
• All is documented, recorded, and reported.
• Includes all aspects of recovery, including data
  recovery.

32
Solectron Phase VII, Maintenance

• Red/yellow/green status reports used.
• As timelines for plan implementation, or
  maintenance, approaches reports turn from red, to
  yellow, and eventually green upon completion.
• The report goes out to everyone, including
  customers, every two weeks.
• Additionally, corporate auditors perform annual
  inspections, which act as a general managers
  report card to the board of directors.

33
Solectron Phase VII, Contd

• Regional training is performed on an regular
  basis.
• Allows individual sites to compare notes.
• Worldwide facilities council conference calls
  take place every month.
• Implementation of a monthly newsletter.

34
Solectrons Monthly Newsletter
35
Example Case 2

• Gillette Management, Inc.

36
Handout

• Gillette Management, Inc. is a manufacturer of a
  wide range of personal products.
• Had informal BCP in place for more than 20 years.
• In July, 1998 Gillette signed with IBM to provide
  services to ensure continuity.
• Two months later Hurricane Georges struck the
  Puerto Rico offices.
• Operations continued seamlessly routed through
  the New York offices.
• Emphasizes the importance of preparedness!

37
Management Survey
38
Post 9/11 Survey

• Survey included in the November 26, 2001 issue of
  InformationWeek.
• Survey of 250 I.T. and business managers
  responsible for business continuity plans.
• Survey fielded by PricewaterhouseCoopers.

39
Continuity Plan Hurdles

• 64 extend business continuity plans across the
  entire enterprise.
• 36 in I.T. only.
• Only 2 in 5 extend BCP to involve both I.T. and
  business leaders.
• 2 use action groups.

40
Continuity Planning Expenditures

• After the September 11th attacks 10 of companies
  plan to significantly increase expenditures.
• 50 plan to increase spending.
• 40 plan to stay the same or less.
• 50 state expense is a hurdle to effective
  management.
• The sight of all the paper fluttering down from
  the WTC towers hit home with some management!

41
Owner of Contingency Plans

• 12 of survey respondents say I.T. own the
  business continuity plan.
• 40 say place planning with corporate executives.
• 32 say I.T. maintains the plan.
• 50 of I.T. managers say BCP is a collaborative
  venture.
• Only 33 of executive management agree BCP is a
  collaborative venture.

42
Business Continuity Plan Usage

• 28 of the respondents state their business
  continuity plan was used in the last 12 months.
• Of the 28 plans used 66 described the crisis as
  severe or extremely severe.

43
Plan Review Schedule

• 20 stated continuous reviews.
• 22 stated several times per year.
• 30 stated annually.
• 25 stated seldom or not at all.
• 3 stated dont know.

44
Disaster Plan Recovery Time

• 14 had hot backup or standby systems available.
• 40 mentioned it would take days or longer.
• 7 stated systems would not be back online for
  weeks or longer!

45
Other Survey Information
46
Other Survey Information (contd)
47
Other Survey Information (contd)
48
Salary Survey
49
PrimeSearch 1999 Salary Survey Results

• Data Sources
• Online Surveys.
• Attendees of the Spring 2000 Disaster Recovery
  Journal Conference in San Diego.
• Continually reviews the methods in which data is
  collected to provide the best information
  possible for the industry.
• Dated a year but specialized and most recent
  available.
• May still be close as a result of a weak economy
  and strong need for BCP after 9/11 attacks.
• Previous InformationWeek poll indicated 60 of
  organizations plan to increase budgets in DR/BCP.

50
Certified Non-Certified Professionals

• Identifies variance between a certified and
  non-certified Business Continuity Planner.
• Average is the averaging of the non-certified
  data above and certified data below.

51
Degreed vs. Non-Degreed

• 69 of the respondents had received a college
  degree.
• 70 of the respondents the stated the previous
  year they held a college degree.
• The gap diminished during this survey.

52
Salary by Location

• Boston appears to be the highest paying city for
  a certified DR/BC Planner.

53
Salary by Region

• For professionals not living the previous cities
  salaries are broken out into region.
• It appears the Northeast is the best region to
  live for a DR/BC planning professional.

54
Salary by Sector

• Consulting appears to lead in wages according to
  sector.
• Interesting to see governments total last as a
  result of no bonus paid!

55
Salary by Working Experience

• The previous years respondents stated they had an
  average of 21 years of working experience.
• This years respondents stated they had an average
  of 15.6 years of working experience.
• This shift in experience could be attributed to
  the addition of personnel due to Y2K activities.

56
Salary by DR/BC Experience

• The previous years respondents stated they had an
  average of 9 years of DR/BC experience.
• This years respondents stated they had an average
  of 6.7 years of DR/BC experience.
• Again, this shift in experience could be
  attributed to the addition of personnel due to
  Y2K activities.

57
Sample DR/BC Titles

• Disaster Recovery Coordinator
• Business Continuity Planner
• Business Continuity Coordinator
• Business Recovery Services Manager
• Data Center Recovery Services Manager
• Data Center Recovery Services - Sr. Analyst
• Trading Floor Recovery Services Manager
• Trading Floor Recovery Services - Sr. Analyst

58
Questions?
59
Exercise
60
Disaster Recovery Scenario 1

• You work for XYZ Corporation in Kent Ohio and are
  a member of their Disaster Recovery Team. Youve
  just been informed that fire has destroyed a good
  portion of the business where both the I.S. and
  Finance Departments are housed. Using the Asset
  Sheets provided, develop a Simplified Plan to
  restore the processing environment!

61
References

• http//www.drj.com
• http//www.utoronto.com
• http//www.colostate.edu
• http//www.securityauditor.com
• http//www.gammassl.co.uk
• http//www.contingencyplanning.com
• http//www.informationweek.com
• http//www.primesearch.com
• http//www.infoworld.com

62
References Contd

• http//www.contingencyplanning.com/article_index.c
  fm?article297
• http//www.contingencyplanning.com/article_index.c
  fm?article184
• http//www.computeamtx.com/services/disaster_recov
  ery/http//www.idra.com/
• http//www.alliancedatacom.com/disaster-frame-rela
  y.htm
• http//www.solinet.net/presvtn/leaf/displan.htm
• http//www.e-dbms.com/downloads/documentation
• http//www.disaster-resource.com