Realizing the Need

1
(No Transcript)
2
Realizing the Need
In June, 2004, the Department of State (CDOS)
experienced a significant interruption in its
data center services. Following an extensive
outage, testing revealed the presence of zinc
whiskers, zinc fibers that can be extruded over
time by electroplated metal coatings, such as the
coating applied on the electrostatic-dissipating
raised floor tiles used in the CDOS data center.
The facility was rendered unusable for computer
operations. The Departments equipment had been
internally contaminated and as a result was
subject to additional disruptions. This
catastrophic event spotlighted a paramount need
for an achievable business continuity and
disaster recovery capability.
3
Disaster Recovery Survey

• In March of 2005 the Colorado Information
  Management Commission (IMC) through the Governors
  Office of Innovation and Technology (OIT)
  conducted a survey of the State Agency disaster
  recovery preparedness. The survey found that
• Of all 23 state agencies surveyed, none reported
  complete IT disaster recovery capability.
• Risk was high for all State agencies because
  disaster recovery capabilities were low.
• Threats were rising the risk profile was
  getting worse.
• Majority of agencies were planning and/or seeking
  individualized rather than collaborative disaster
  recovery investments.
• An overwhelmingly common misconception about
  disaster recovery planning existed where it was
  thought that having a backup strategy, doing
  backups everyday and running anti-virus software
  was the same as having a Disaster Recovery Plan.

4
Strategic Solution

• The Colorado Legislature approved a 3.6 million
  appropriation with 2.3 million in continuation
  funds from the CDOS cash fund to build and
  maintain a disaster recovery facility.
• Colorado Department of State, with the support
  of the Governors Office of Innovation and
  Technology successfully implemented a strategic
  solution for disaster recovery known as the
  enterprise Facility for Operational
  Recovery/Readiness/Response and Transition
  services, ( e-FOR3T.)
• The e-FOR3T initiative established a statewide
  disaster recovery facility for use by state
  agencies and government entities. The facility is
  operated under the provisions of a cooperative
  agreement between the Colorado Secretary of
  State's Office and participating agencies. The
  facility allows agencies to recover critical
  functionality in the event of a disaster or an
  event resulting in a significant interruption of
  service by
• Providing a tier 3 IT facility for operational
  recovery
• Promoting economies of scale through uniformity
  of service infrastructure
• Providing participating member agencies with a
  cost effective reliable level of expected
  service

5
E-For3t Disaster Recovery Facility
Addresses the following strategic objectives by
providing

• Enterprise Solution Uniform application of best
  practices for operational recovery, readiness
  response
• Funding Strategy Colorado Department of State
  utilizing its cash funds to maintain portions of
  the disaster recovery infrastructure that are
  common in facility infrastructure operations
• Participating Member Agency Flexibility Allows
  the flexibility to architect specific required
  recovery solutions
• All State of Colorado Government Entities Invited
  Participation is voluntary for any of these
  entities
• Facility Infrastructure Components Power,
  connectivity, and 12,167 sq ft of space to house
  equipment and personnel

Enterprise Facility for     Operational
Recovery Readiness
Response          Transition services
6

• e-FOR3T Specifications
• Data Center Space
• 6,549 sq ft raised floor
• Individually locking (front and rear) cabinets
• Private cages as required

• Migration Transition Support
• Staffing and resources for physical migration
• Documented processes and procedures
• Relocation and standup of state agency hardware
• Review post-transition analysis internally and
  externally
• Off-site tape vaulting

7

• e-FOR3T Specifications
• Office Space
• 5,618 sq ft fully furnished (desks, workstations,
  phones, etc)
• Security
• 24 x 7 magnetic card key access with secondary
  pin code
• Digital motion activated security cameras and
  intercoms

• Cable Management
• Provide all cabling for each agency deployment
• Cabling tagged and documented

8

• e-FOR3T Specifications
• Network Operations Center
• Telecom Network Support
• MNT network services provided on dual path system
  with redundant IP router LAN switches
• 20 remote access internet accounts with static IP
  addresses for each participating member agency
  (10 DSL 10 dial up)

• Remote hands services allowing State to use
  facility personnel to perform basic tasks as
  instructed by State engineers
• Insurance coverage for State assets
• MNT network failover to redundant IP network
• Private Network Transit (PNT) via backbone
  connection between Denver and alternate site

9

• e-FOR3T Specifications
• Power
• Facility power, delivery, infrastructure,
  generator and fuel
• UPS systems and megawatt diesel generators

• Cooling
• HVAC service to support requirements
• Anti-static raised floor with cooled air delivery
  
• Fire Detection Suppression
• Certified incipient smoke detection (VESDA)
• Dual-Interlock Pre-Action dry pipe sprinkler
  system

10
(No Transcript)
11
Continuity of Operation Plan
The Governors Office of Information Technology
(OIT) provides the leadership in planning for
recovery of State of Colorado functions after a
man-made or natural disasters through their
Continuity of Operation/Continuity of Government
Program. The Program Manager assists State of
Colorado agencies in developing COOP plans using
Federal standards for content per FPC 65.
Internet-based LBL Contingency Planner software
is used to create and update COOP
documentation. The templates included in the
Contingency Planner software include suggested
procedures for carrying out a disaster recovery
plan. Following the federal model, these plans
will address identification and prioritization of
agency essential functions, succession,
delegation of authority, interoperable
communications, identification of alternate
facilities, identification of vital records and
databases, human capital aspects, reconstitution
of facilities and the testing, training and
exercising of these plans. Each program within
CDOS was tasked with providing specific details
regarding their operations. By creating
documents with possible scenarios agencies were
able to determine how specific needs would change
based on the circumstances.
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Scenario I SOS and Other Agencies
Affected/Internet Functional A. Assumptions 1. Ac
tivation of the COOP is the result of a disaster
that affects more than the Secretary of
State. 2. The Secretary of States office is
uninhabitable. 3. Other affected agencies will
want to file emergency rules with Secretary of
State. 4. Internet and SOS systems operational at
e-For3t within 36 to 48 hours after COOP
activation. 5. IT has implemented VPN so that
employees can work from home rather than at
e-For3t. B. Critical functions In a normal work
environment rulemaking agencies make all filings
electronically, paper filings are not accepted.
With the current procedures, all electronically
filed rules need to be reviewed and the
formatting needs to be modified by the
Administrative Rules Staff in order to be
published online. This process requires
extensive review and formatting changes by the
Administrative Rules Staff to ensure that
documents are compatible with the online
publishing system. Normally, the rulemaking
process is spread out over a number of months and
many agencies file infrequently. Only a handful
of agencies file every month. In an emergency
situation, many more agencies may be forced to
adopt rules to accommodate the situation. The
additional load on the system may create problems
and the total volume of rules will necessarily
increase but the amount is unknown.
17

• Scenario I Continued
• C. Minimum personnel
• 1. Program Mgr
• 2. Program Asst II
• 3. Administrative Asst III
• D. Resource Requirements
• 1. Personal desktops, laptops or laptops
  provided by SOS.
• 2. Access to high speed Internet lines (May
  require subsidization of additional costs to
  employee).
• 3. VPN connectivity.
• 4. Telephone.
• 5. Scanner.
• E. Space Requirements
• 1. None employees work from home.
• F. Travel costs
• 1. Program Mgr may be required to travel from
  home to eFort on occasional basis.

18

• Scenario II SOS and Other Agencies
  Affected/Internet Nonfunctional
• A. Assumptions
• 1. Activation of the COOP is the result of a
  disaster that affects more than the Secretary of
  State.
• 2. The Secretary of States office is
  uninhabitable.
• 3. Other affected agencies will want to file
  emergency rules with Secretary of State.
• 4. Internet is NOT functional but e-For3t is
  operational with an Intranet capability.
• 5. IT has NOT implemented VPN. Employees cannot
  work from home.
• B. Critical functions
• The normal process of electronic filing of
  administrative rules is disrupted. Agencies must
  default to the previous procedure of delivering
  rules directly to the Secretary of State.
  Although the Internet is not working, the
  Intranet will allow for rules to be entered into
  the database for use when the Internet becomes
  available. Emergency rules will be the major
  portion of the workload with normal rules making
  up the balance.
• In this scenario, mail or hand carried documents
  must be delivered to a single location. The
  Administrative Rules program will have to receive
  the documents (either in electronic or paper
  format, or both), date/time stamp them and then
  process them into the system. For an interim
  period the official register will be published in
  print rather than electronically. Materials will
  be sent electronically or mailed to the outside
  vendor that produces the Colorado Register in
  print.

19

• Scenario II Continued
• C. Minimum Personnel
• 1. Program Mgr.
• 2. Program Asst. I
• 3. Admin. Asst. III
• D. Resource Requirements
• 1. Computer workstations with Intranet access to
  the database for three individuals.
• 2. Telephone (at least one).
• 3. Copy machine.
• 4. Fax machine.
• 5. Scanner.
• E. Space Requirements
• 1. Three workstations at eFort. One option
  would be to coordinate with the Arapahoe County
  Clerks office and use it as the mail reception
  point and initial processing site. That could
  eliminate the need for two of the three
  workstations at eFort.
• F. Travel Costs
• 1. Staff will be required to work from an
  alternate location.

20

• Scenario III Only SOS Affected/Internet
  Functional
• A. Assumptions
• 1. Activation of the COOP is the result of a
  disaster that only affects the Secretary of
  State.
• 2. The Secretary of States office is
  uninhabitable.
• 3. Agencies will only need to file rules in the
  normal course. These may be regular or
  emergency rules but the situation is no
  different than normal operations.
• 4. Internet and SOS systems operational at
  e-For3t within 36 to 48 hours after COOP
  activation.
• 5. IT has implemented VPN so that employees can
  work from home rather than at e-For3t.
• B. Critical functions
• In a normal work environment rulemaking agencies
  make all filings electronically, paper filings
  are not accepted. With the current procedures,
  all electronically filed rules need to be
  reviewed and the formatting needs to be modified
  by the Administrative Rules Staff in order to be
  published online. This process requires
  extensive review and formatting changes by the
  Administrative Rules Staff to ensure that
  documents are compatible with the online
  publishing system.
• In this circumstance, it will more or less be
  business as usual for the Administrative Rules
  Program. The only thing different will be that
  staff will either work from home or work at
  e-For3t.

21

• Scenario III Continued
• C. Minimum Personnel
• 1. Program Mgr.
• 2. Program Asst. I
• 3. Admin. Asst. III
• D. Resource Requirements
• 1. Personal desktops, laptops or laptops
  provided by SOS.
• 2. Access to high speed Internet lines (May
  require subsidization of additional costs to
  employee).
• 3. VPN connectivity.
• 4. Telephone.
• 5. Scanner.
• E. Space Requirements
• 1. Personnel could work from home per Scenario I
  or work at eFort per Scenario II.
• F. Travel Costs
• 1. Staff may need to work from an alternate
  location.

22
Links to COOP Info

• Continuity of Operation Planning/ Continuity of
  Government http//www.colorado.gov/cs/Satellite/O
  IT-New/OITX/1188204067666
• enterprise Facility for Operational Recovery
  Readiness Response Transition (e-FOR3T)
  http//www.sos.state.co.us/pubs/business/efort.htm
  
• LBL Technology Partners Contingency Planner
  softwarehttp//www.lbltech.com/ContingencyPlanner
  .htm