Operational Recovery Planning

1
Operational Recovery Planning

• Presented by the California State Information
  Security Office

2
Agenda

• Introductions name and agency
• CA State Information Security Office
• Definitions
• Four Types of Continuity Plans
• Review of BL 07-03 ORP Changes
• ORP-COOP/COG Alignment
• Discuss Test Scenarios

3

4
State Information Security Office

• Vision
• Leading the way to secure the State's
• information assets.
• Mission
• To manage security and operational
• recovery risk for the State's
• information assets by providing
• statewide direction and leadership.

5
Definitions

• Emergency Response
• Business Continuity Planning (BCP)
• Operational Recovery Planning (ORP)
• Continuity of Operations (COOP)
• Continuity of Government (COG)

6
Emergency Response

• The immediate reaction and response to an
  emergency situation commonly focusing on ensuring
  life safety and reducing the severity of the
  incident.
• Definition from Disaster Recovery Journal (DRI)
  website at http//www.drj.com/glossary/

7
Business Continuity Planning (BCP)

• Process of developing and documenting
  arrangements and procedures that enable an
  organization to respond to an event that lasts
  for an unacceptable period of time and return to
  performing its critical functions after an
  interruption.
• Similar terms  business resumption plan,
  continuity plan, contingency plan, disaster
  recovery plan, recovery plan.
• Definition from Disaster Recovery Journal (DRI)
  website at http//www.drj.com/glossary/

8
Operational Recovery Planning (ORP)

• The management approved document that defines the
  resources, actions, tasks and data required to
  manage the technology recovery effort.  Usually
  refers to the technology recovery effort.  This
  is a component of the Business Continuity
  Management Program. 
• DISASTER RECOVERY PLAN (also known as
  Operational Recovery Plan)
• Definition from Disaster Recovery Journal (DRI)
  website at http//www.drj.com/glossary/

9
Continuity of Operations (COOP)

• The activities of individual departments and
  agencies and their sub-components to ensure that
  their essential functions are continued under all
  circumstances. This includes plans and
  procedures that delineate essential functions
  specify succession to office and the emergency
  delegation of authority provide for the
  safekeeping of vital records and databases
  identify alternate operating facilities provide
  for interoperable communications and validate
  the capability through tests, training, and
  exercises.
• Office of Emergency Services (OES)

10
Continuity of Government (COG)

• The preservation, maintenance, or reconstitution
  of the institution of government. It is the
  ability to carry out an organizations
  constitutional responsibilities. This is
  accomplished through succession of leadership,
  the pre-delegation of emergency authority and
  active command and control.
• Office of Emergency Services (OES)

11
Relationship of Plans
12
Inter-Dependencies
13
Three Phases of Continuity
Departments
Emergency Response - Life Safety First 72 Hours
Restoration Business back to normal
IT Operational Recovery up to 30 days
Planning, Documenting, Testing, and Training
Business Recovery up to 30 days
Damage Assessment First 72 hours
Phase I
Phase II
Phase III
14
IMPLEMENTATION OF PLANS

• Disruption of business occurs and you are
  informed, next steps
• 1. Emergency Response safety and security of
  staff.
• 2. Securing the site.
• 3. Activate COOP/COG Plan to ensure the
  continuation of essential functions.
• 4. Implementation of the communication plan.
• 5. After assessing incident, determine if
  implementation of BCP ORP is required.
• 6. Contact SISO to report incident.
• 7. Implement BCP and ORP

15
Budget Letter 07-03

• SAM Section 4843 Operational Recovery Planning
• Use results from risk analysis and business
  impact analysis to identify critical business
  functions.
• Include the operational recovery considerations
  and costs in FSRs.
• Develop ORP as part of a complete continuity
  program.

16
Budget Letter 07-03 Continued

• SAM Section 4843.1 Agency Operational Recovery
  Plan
• Rewritten to clarify and enhance operational
  recovery requirements.
• Removal of minimum components from policy.
• SIMM 65A ORP Documentation for Agencies
  Preparation Instructions
• Requires ten minimum components in ORP.
• Additional three components for agencies without
  a BCP or COOP/COG.

17
ORP Documentation Revised

• Components to be included in the ORP updated in
  January 2007.
• The April and July quarterly filers must provide
  a cover sheet indicating where the information
  for each topic area in SIMM 65A is located in the
  agencys Operational Recovery Plan.
• All components listed in SIMM 65A must be
  addressed and included in agencies ORPs
  beginning in October 2007.

18
Changes for ORP Development

• Overall
• Requires more details
• New Components
• Backup and offsite storage
• Data Center Services
• Contact information
• Removed from SAM and Policy
• Damage Recognition
• Preparation of cost-benefit analysis
• Selection of alternative
• SIMM Section 140A

19
New Requirements

• ORPs must describe
• Agency Administrative Information
• Critical Business Functions/Applications
• Recovery Strategy
• Backup and Offsite Storage Procedures
• Operational Recovery Procedures
• Data Center Services
• Resource Requirements
• Assignment of Responsibility
• Contact Information
• Testing

20
Supplemental Requirements

• Agencies that have not developed and implemented
  a full business continuity plan or COOP/COG must
  also address and include the following in their
  ORP
• Damage Recognition and Assessment
• Mobilization of Personnel
• Primary Site Restoration and Relocation

21
Agency Administrative Information

• A communication plan should include strategy on
• How information will flow (escalation)
• Decision making processes
• Interrelationship among agency resources for
  response, recovery and resumption

22
Example - Escalation Process

• Single site, minor impact. User calls into Help
  Desk with possible virus infection. Communication
  Plan strategy includes
• Process to dispatch field support to check PC
• If infected, take steps to identify and
  quarantine
• notify ISO and IT Management
• Eradicate virus
• Verify virus has not spread

23
What would you do?

• Multiple site, major impact. The virus outbreak
  has spread from your headquarters to your remote
  offices and is running rampant. The anti-virus
  software will not eradicate it and all the
  systems in your agency are being impacted.
• What would your communication plan need to
  include?

24
Communication Plan

• Document
• Who to contact and under what circumstances
• Lists name, phone , cell , home , email
  address
• Includes Chain of Command Management, other
  pertinent staff (ISO, ORP Coordinator, etc), and
  contractors
• Distribute to applicable staff
• Providing training to staff
• Collect when duties change or staff leaves

25
Sample Call Lists

• Wallet size cards
• Name, work , cell , home , email
• Call Tree
• Manager calls supervisor
• Supervisor calls his/her staff

26
Critical Business Functions/Applications

• This section includes a description of
• Critical business functions and their supporting
  applications
• Maximum Allowable Outage (MAO) for each
  application
• Recovery priorities

27
Example - Critical Business Function

• Single site, minor impact. Help Desk identifies
  that the services on the email server are not
  working. As a critical business function,
  recovery strategy includes
• Process for IT staff to check services
• If denial of service, follow internal procedures
  to identify and mitigate.
• Notify ISO and IT Management

28
What would you do?

• Multiple site, major impact. The email server has
  crashed, there are both hardware and software
  failures. Rebuilding the server will require
  replacement hardware, which will take several
  days to acquire and configure.
• What would your Critical Business Functions /
  Applications need to include?

29
Procedures for Critical Functions

• Document
• Critical Business Functions
• Recovery Procedures
• Responsible individuals or team for recovery
• Distribute procedures to applicable staff
• Provide training

30
Sample Procedure

• Repair/replace hardware
• Restore database structure
• Restore post office
• Restore gateway connectivity
• Rebuild database
• Keep users/management informed

31
Recovery Strategy

• Recovery strategy should include alternate
  recovery site/sites that include
• Location of all sites
• Requirements of facilities/equipment
• Contact numbers

32
What would you do?

• Single site, minor impact Your department is
  located in several locations. A building adjacent
  to one location has a fire, the fire did not
  spread to your site. The Fire Dept and Law
  Enforcement block the street, so there is no
  access into your building.
• What would your recovery strategy need to
  include?

33
Recovery Strategy

• Communication plan for employees, management, and
  contractors.
• List all office locations.
• Identify the alternate location. If multiple
  locations are available, prioritize them.
• Address what functions could be restored at each
  site.
• Determine who would need to be called, include as
  the contact list.

34
Sample Recovery Strategy

• Department has three locations
• 1234 Headquarters St., Sacto, 95814
• 5678 Anywhere St., Sacto, 95825
• 9876 SomePlace St., LA 90210
• Critical operations would be restored at an
  unaffected site (identify priority and equipment
  needed).
• Contact
• J Resto at [phone] for Headquarters
• R Quick at [phone] for Anywhere
• M Pia at [phone] for SomePlace

35
Backup and Offsite Storage

• The backup and offsite storage procedures should
  include
• Retention schedule
• Procedures
• List of authorized staff
• Account information
• Contacts of offsite storage

36
What would you do?

• The data on one of your critical applications was
  corrupted and its MAO is 4 hours. It is 530 pm
  on Friday and Monday is a holiday. The business
  area have staff scheduled to work Saturday on
  this system. Technical staff has gone home, and
  several are out of town for the weekend.
• What would your backup and offsite storage
  procedures need to include?

37
Details Backup and Offsite Storage

• Document
• Retention schedule
• Detailed procedures
• Hardware and software (include version)
• Offsite storage details (location, acct )
• Retrieval of backups (contacts (24x7) and
  personnel authorized to retrieve)
• Process to identify data to be restored

38
Operational Recovery Procedures

• These procedures systematically detail the
  operational procedures for recovery in a timely
  and orderly way, they should include
• Detailed procedures that the backup or other IT
  professional could follow
• High-level network diagram that includes all
  critical applications

39
Data Center Services

• This section should include a
• Description of service to be provided.
• Interagency agreements, memorandums of
  understanding, or contracts.
• Specific coordination efforts with the data
  center critical to the recovery efforts.

40
Example Minor Impact

• Single site, minor impact. Your Web server
  providing access to one of your critical
  applications located at DTS has been compromised.
  You have contacted DTS and DTS is working to get
  the server back online within the hour.
• What would your need to include?

41
What would you do?

• Multiple site, major impact. There was a fire in
  a facility adjoining DTS facility where the
  servers are housed. The sprinkler system was
  activated and the servers had to be powered down.
  There is significant water damage. There is an
  estimate that it will take 14 to 21 days to
  reestablish services.
• What would your plan need to include?

42
Details - Data Center Services

• Expectations
• Meet with Data Center to identify
• Hardware/Software requirements
• Services required
• Timeframe for services
• Document Agreement Before its needed
• Create a Service Level Agreement (SLA) or
  Memorandum of Understanding (MOU)
• Develop Recovery Procedures

43
Resource Requirements

• This is a comprehensive list of
• Equipment
• Software
• Telecommunication needs
• Data
• Hard copy manuals
• Personnel essential for recovery

44
Assignment of Responsibility

• Designation of responsibilities and assignments
  should be listed. Procedures should include job
  title, and not individual names, for the recovery
  process.
• Individuals names can be placed in a single
  location for ease of maintenance.

45
Contact Information

• There are two types of contact information to be
  collected
• Employees, including management.
• Resource List including contractors, Major
  Service providers, vendors, other government
  entities, and outside resources critical to the
  recovery process.

46
Contact List

• Employee contact information should be designated
  as sensitive, and provided to authorized
  individuals.
• Resource lists typically have business contact
  information. This information can be provided
  more widely.

47
Testing

• Annual testing of the ORP is essential to
• Ensure for training the management and recovery
  teams.
• Validate that the procedures have the appropriate
  level of detail.
• Verify Call Back lists are current.
• Confirm that Recovery strategies are appropriate
  for your environment.

48
Governors Office of Emergency Services

• Introduction
• Mission and Goals of OES
• SEMS/NIMS
• Disaster Service Worker

49
Planning

• Be Smart, Be Responsible. Be Prepared. Get Ready
  Campaign
• Your Intranets and Emergency Preparedness
• Executive Order S-04-06
• State Emergency Plan /
• COOP-COG/ORP

50
Training and Testing

• Emergency Management Training Requirements for
  Public Employees
• The California Specialized Training Institute
  (CSTI)/OES Training Branch
• How to develop a Table Top Exercise (TTex)
• Definition of a TTex
• The 8 Step Process Used to Design a TTex
• After Action/ Corrective Action Process
• California Master Exercise Calendar (CMEX)

51
State IT Strategic Plan Action Item

• To align the ORP and COOP/COG, a work group has
  been established to
• review processes
• define terminology
• evaluate reporting requirements

52
Resources

• SISO web site http//www.infosecurity.ca.gov/ORP/
  
• Budget Letter 07-03 ORP Policy Changes
• http//www.dof.ca.gov/OTROS/StatewideIT/IT_BdgtLtt
  rs.asp
• ORP Policy in the State Administrative Manual
  (SAM)
• Operational Recovery Planning
  http//sam.dgs.ca.gov/TOC/4800/4843.htm
• Operational Recovery Plan http//sam.dgs.ca.gov/T
  OC/4800/4843.1.htm
• ORP SIMM 65A http//www.infosecurity.ca.gov/Pol
  icy/

53
Contact Us

• Rosa.Umbach_at_dof.ca.gov
• [phone] ext. 3242
• Colleen.Pedroza_at_dof.ca.gov
• [phone] ext. 3224
• SISO Office
• email security_at_dof.ca.gov
• Telephone [phone]
• www.infosecurity.ca.gov