Disaster Recovery Larry Pedrazoli

1
Business Continuance
Disaster Recovery
2
MISSION
IM
POSSIBLE
IM
3
Learning Objectives

• Benefits of Planning
• Distinguish between Business Continuance and
  Disaster Recovery
• Disaster Recovery Overview
• Where do We Start
• Things to Remember
• Critical Success Factors
• Final Thought

3
4
4
4
5
Benefits

• Prioritizes applications/systems
• Establishes definitive recovery time frames for
  all business functions
• Decreases potential exposures
• Eliminates confusion and errors
• Minimizes decision-making during a disastrous
  event
• Reduces reliance on certain key individuals
• Proves the reliability of standby/backup systems
• Provides training materials for new employees
• Allows you to maintain service levels consistent
  with established expectations of you
  organizations customers

5
6
Key Question
OK, so which is it? Business Continuance
or Disaster Recovery?
6
7
Business Continuance

• Ensure that the applications and information that
  are necessary to run the business are always
  available and accessible
• Available
• Systems are physically available
• Application performance is acceptable and
  consistent
• Relates to planned and unplanned outages

7
8
Disaster Recovery

• A consistent copy of critical data offsite that
  can be recovered after an unplanned outage
• Disaster Recovery Includes
• Business Resumption
• Application Recovery
• Data Recovery

8
9
Disaster Recovery
A Disaster is
Any unplanned interruption of critical business
processes for an unacceptable period of time.
9
10
Disaster Recovery Planning
A Matter of Business Survival...

• Business Interruptions Result In
• Loss of Revenue
• Erosion of Customer Confidence
• Loss of Market Share
• Short and Long Term Negative Image

10
11
Disaster Recovery Planning
To Survive a Disaster You Need

• A Place to GO
• Vital Data
• A Plan to Follow
• Well Trained People

12
Disaster Recovery Planning
Key Phases

• Prevention
• Response
• Resumption
• Recovery
• Restoration

13
Prevention

• Prevention
• Response
• Resumption
• Recovery
• Restoration

Measures and activities that will lessen the
possibility or impact of an adverse incident
occurring. Protect assets and helps manage risk.
14
Response

• Prevention
• Response
• Resumption
• Recovery
• Restoration

Reaction to an incident or emergency to assess
damage or impact and to ascertain the level of
containment and control activity
required. Addresses the policies, procedures and
actions to be followed in the event of an
emergency
15
Resumption

• Prevention
• Response
• Resumption
• Recovery
• Restoration

Process of planning for and/or implementing the
resumption of only the most time-sensitive
business operations immediately following a
disaster.
16
Recovery

• Prevention
• Response
• Resumption
• Recovery
• Restoration

Process of planning for and/or implementing
expanded operations to address less
time-sensitive business operations immediately
following an interruption or disaster.
17
Restoration

• Prevention
• Response
• Resumption
• Recovery
• Restoration

Process of planning for and/or implementing
procedures for the repair or relocation of the
primary site and its contents, and for normal
operations at the primary site.
18
Disaster Recovery Planning
Where Do We Start?
19
Disaster Recovery Planning

• Self-AssessmentWhich are you?

20
Disaster Recovery Planning
Phase 1 Executive Buy-In

• Management understanding and support at the
  executive business level
• Executive Sponsorship

21
Disaster Recovery Planning
Phase 2 Education and Awareness
Executive Buy-in Executive Sponsor
Establish a Business Continuity Council

• Accounting
• Compliance
• Customer Service
• Marketing

• Facilities Management
• Administration
• Clinicians
• Contracting and Provider Relations

Provide Training
22
Disaster Recovery Planning
Phase 3 Build the Plan

• Define Scope
• Perform Risk Analysis
• Perform Impact Analysis
• Determine/Approve Strategies
• Build Teams

23
Disaster Recovery Planning
Phase 4 Develop and Test the Plan

• Organize and document a plan
• Develop testing criteria
• Test plan
• Evaluate test results
• Update/Revise as necessary
• Management approval of plan

24
Disaster Recovery Planning
Phase 5 Maintain and Test Plan

• Make modifications for
• Changes in software/hardware
• Updating production applications and functions
• Operating Procedures
• Personnel and organizational changes
• Other events impacting the recovery plan
• Ongoing Training and Testing

25
Disaster Recovery Planning
Risk Analysis
26
Risk Analysis
The process of identifying the most probable
threats to an organization and analyzing the
related vulnerabilities of an organization to
these threats.
Major Risk Factors

• Key personnel loss
• ASP failure
• Toxic contamination
• Aircraft crash
• Gas leak
• High winds
• Snowstorm

• Utility failures
• Network failure
• Fire
• Flood
• Tornado
• Hurricane
• Bomb Threat

27
Risk Analysis
28
Risk Analysis
THE EQUATION Probability Impact Cost Risk
Rating

• The PROBABILITY of occurrence for a particular
  event
• The potential IMPACT to your organization if the
  event occurred
• The estimated COST of implementing a solution

29
Risk Analysis
THE EQUATION Probability Impact Cost Risk
Rating
30
Disaster Recovery Planning
Business Impact Analysis(BIA)
31
Business Impact Analysis
The process of analyzing all core business
functions and establishing an optimized timetable
for recovery.

• Maps data flow
• Identify maximum tolerance for downtime.
• Identify interdependencies.
• Determine the recovery priorities of the
  organization.

• Provides baseline for
• Justification for costs associated with recovery.
• Developing recovery strategies.
• Developing Support Level Agreements

32
Business Impact Analysis
33
Conceptual Recovery Time Lime
34
Recovery Strategies
35
Recovery Strategies
HOT SITE Facility containing computers and
necessary peripheral equipment that may be
occupied by a subscriber immediately after a
disaster declaration to restore its own systems,
applications and data. WARM SITE Facility
partially equipped with hardware, communications
interfaces, power sources, and environmental
conditioning. COLD SITE Facility with
computer-ready space held in reserve for the
users own systems.
36
Disaster Recovery Planning
37
Things to Remember

• Cant physically get there
• Martial law
• Airports closed
• Bridges and tunnels closed
• Recover from tapes stored across the river?
• Forget it!
• Thousands of people working from home
• Email is mission critical!

38
Things to Remember

• Family first, then business
• Vendor support personnel and capabilities are
  extremely important
• Include in formulating and testing recovery plans
• Disperse your Disaster Recovery support teams
• DR staff in one place is vulnerable

39
Disaster Recovery Planning
Critical Success Factors

• Management understanding and support at the
  executive business level
• Alignment of disaster recovery objectives with
  business objectives
• Maintain cradle to grave process
• Ruthless prioritization
• Appropriate resource allocation
• Using technology as the enabler

40
Final Thought
41
Disaster Recovery Planning
Business Impact Analysis(BIA)
42
Business Impact Analysis
The process of analyzing all core business
functions and establishing an optimized timetable
for recovery.

• Maps data flow
• Identify maximum tolerance for downtime.
• Identify interdependencies.
• Determine the recovery priorities of the
  organization.

• Provides baseline for
• Justification for costs associated with recovery.
• Developing recovery strategies.
• Developing Support Level Agreements

43
Business Impact Analysis
End-User Questionnaire Highlights

• Department Overview
• Workflow Interdependencies
• Computer Resources
• Application Impact Analysis

44
Department Overview

• Department Overview
• Workflow Interdependencies
• Computer Resources
• Application Impact Analysis

• Identify department, location, and at least two
  representatives from each department.
• Develop a comprehensive list of applications used
  in the department.
• Describe the business function(s) of the
  department.
• Gather information about the departments daily
  business hours, revenues generated, transaction
  volume, and any peak or high demand periods.

45
Workflow Interdependencies

• Department Overview
• Workflow Interdependencies
• Computer Resources
• Application Impact Analysis

• Identify the departments and organizations that
  send work to the department.
• Determine what routes or channels of
  communication are used to send that incoming work
  and estimate the percentage that comes via each
  route or channel.
• Gather the same information in 1 and 2 for work
  sent by the department.

46
Computer Resources

• Department Overview
• Workflow Interdependencies
• Computer Resources
• Application Impact Analysis

• Gather information on the computing equipment in
  the department and how it is used.
• Begin exploring the reliance that the department
  has on the computing equipment, e.g., What data
  entry backlog would there be if it was
  unavailable for one day?

47
Application Impact Analysis

• Department Overview
• Workflow Interdependencies
• Computer Resources
• Application Impact Analysis

• Basic description of each application, including
  what it does, what business functions it
  supports, if it handles PHI, and who the
  department contacts are for the application.
• Estimate the level of departmental business
  interruption associated with the application
  being unavailable through various time
  thresholds.
• Estimate the associated data entry backlog that
  would result and how many staff hours it would
  take to eliminate the backlog.

48
Application Impact Analysis

• Department Overview
• Workflow Interdependencies
• Computer Resources
• Application Impact Analysis

• Evaluate the downtime procedures associated with
  the application, asking questions like have the
  procedures been used before?, how did they work?,
  and how long can the department function using
  them?
• Evaluate any regulatory, legal, financial,
  customer service, and public image problems that
  could arise as a direct or indirect result of the
  application being unavailable through various
  time thresholds.

49
Business Impact Analysis
Other Questionnaires and Interviews

• Executive and Administrative Staff
• IS Staff supporting
• Major Clinical Applications
• Interfaces
• Network Configuration
• Hardware/Servers
• Facility Operations

50
Gundersen Lutheran Inventory

• Initial Y2K inventory had 700 items
• Updates for current environment resulted in 170
  applications
• Interviews identified 122 server based
  applications and their associated runtime
  environments
• Analysis identified 64 applications as mission
  critical

51
Application and System Criticality

• Critical - Applications or resources impacting
  patient care. (No downtime is acceptable).
• Essential - Applications or resources with
  patient health information, patient care data,
  billing, payroll and physical security.
  (Limited downtime).
• Necessary - Department-level applications and
  systems. (Some downtime is acceptable).
• Desirable - Applications and resources that can
  tolerate a significant loss of availability.
  (Recover when normal operations is
  re-established).

52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
Top 10 Components

• DataWarehaus
• Clinical Workstation
• Softlab
• SMS
• PP / Cycare
• Lawson
• Wellsoft
• Orsos
• Pharmacy
• Internet/Intranet/Email

59
Critical Component Analysis
Critical
Recovery Time Objective
Essential
Necessary
Desirable
60
Risks and RecommendationsCurrent Strengths

• Knowledgeable staff.
• Limited backup and restore strategies for most IT
  components are in place.
• Telecom and pager systems meet current business
  needs.
• Establishment of a second (backup) ISP is nearly
  complete so as to ensure Internet availability.
• The Data Warehouse is housed on a high
  availability system.
• IS is upgrading to a Netware release that
  enhances server security and availability.

61
Risks and Recommendations Current Opportunities

• Migrate appropriate desktop-based applications to
  a server environment.
• Assess and consolidate server environment.
• Cluster servers to improve availability.
• Expand the use of proactive monitoring of
  applications, systems and interfaces.
• Implement standardized backup and recovery
  procedures across all systems.

62
Risks and Recommendations Current Weaknesses

• The Data Warehouse is the greatest risk area
  across Gundersen Lutherans IT environment due to
  no offsite redundant environment.
• Interface availability and related data
  integrity.
• Critical applications have no offsite redundancy
  and limited internal failover capability.
• Need to build business continuance and disaster
  recovery into the life cycle of every IT
  component.

63
Business Impact Analysis Project Moving Forward
Next Steps

• Business Impact Analysis - Phase 2 Objectives
• Complete the comprehensive IT component audit.
• Provide detailed input for determination of
  service level objectives.
• Complete strategies for decreasing risk related
  to critical applications and IT components.
• Additional Issues
• Identify critical desktop applications and local
  databases.
• Include critical standalone systems in the BIA.
• Consolidate the IS BC/DR plan with departmental
  plans.

64
The End