Title: Crisis Management The New Role of the CIO/CTOs
1Crisis Management The New Role of the CIO/CTOs
IT staff
2The City of New York
- Resident population of over 8 million daytime
population of 10 million - Over 350,000 City employees, 300,000 retirees
- New York City Government includes 5 counties
- The 1 million student school system reports to
the Mayor - Annual budget exceeds 54 billion dollars
- Trails only the Federal Government, and the
states of California New York - If New York City was a private sector
corporation, it would be in the Top 30 of the
Fortune 500 companies - Over 120 agencies, offices, and organizations
make up The City
3New York Citys Public Safety Record
- In December 2005, out of the nation's 10 and 25
largest cities, New York City ranked the safest
with the lowest overall crime rate. - Among cities with 100,000 residents or more, New
York City ranked 211th out of 227 cities.
4New York Citys Major Incidents - the past 5 years
- Terrorist Attack on World Trade Center
September 11th 2001 - Crash of Flight 587 - November 12th, 2001
- Northeastern U. S. Blackout August 14th 15th,
2003 - Staten Island Ferry crash October 15th , 2003
- One and a half hour partial 911 outage March
26th, 2004 - Major Transit Strike - December 20th 23rd 2005
- Blizzard, largest recorded snowfall in NYC-
February 12th 2006
5NYCs Major Incidents Effect on Telecomm (1)
- Land line cellular phone utilization peaks,
exceeding network capacities. - Phone service spotty and extremely limited.
- Challenge for critical communication for command
and control. - 911 may not be accessible.
- Government Emergency Telephone Service (GETS) may
be inaccessible via cell phone.
6NYCs Major Incidents Effect on Telecomm (2)
- A prolonged power outage, further erosion of
phone service occurs. - Cell sites lose power and fail over time 3-4
hours (100UPS, 50Generator Couplings,
10Generators) - Public Safety Radio infrastructure fails (unless
supported by UPS) - Non-copper based telephone infrastructure fails.
Fiber optics premise based multiplexers - Cable television service fails. Radio stations
w/o UPS go off the air. - VoIP service dependent upon operating non-copper
based infrastructure fails - Premise equipment w/o battery backup fails
wireless phones, cable modems, routers.
7The role of the CIO/CTO during power outages and
events impacting infrastructure
- What is the impact on public safety
communications? - What is the impact on public telecommunications?
- What do the citys web site, 311, 911 need to
do? - What IT support needs to be provided for the
emergency response? - What needs to be done to keep Municipal IT
functioning?
8Telecommunications - What is the impact, what is
the status, what type of volume are is being
handled, are there any issues/problems, any
developing?
- Are 911 systems operating normally?
- Are the telephone systems operating?
- Are cellular networks operating?
- Are our public safety radio systems operating
normally? - Have any critical Public Safety sites been
effected? - Police/Fire/EMS sites, Command and Control
Cityhall, OEM, Hospitals, Key agency buildings
and operational facilities - Have any sensitive locations/installations been
effected? - Nuclear power plants, Chemical plants,
refineries.. - Have any critical commercial sites been effected?
- Exchanges, strategic industries, banks, alarms,
surveillance, monitoring and security systems - Are cable systems up and operating?
9Public Facing IT Utilities What needs to be
done?
- 911, 311, 211
- How high is call volume, do we have enough call
takers, is the next shift going to be able to get
in to work? Supplement call takers? - What is the message, What do we tell people?
- What are people telling us? Who needs to know?
- Are there any new 311 functions that need to be
brought up quickly? How quickly? What
information do we need from callers, what is the
process? - Web site
- How is the web site performing?
- What is the message/content?
- Do we post an emergency home page (OEM)
- Municipal Television
- Is there information to go out on our crawls.
- Do we want to facilitate press coverage by
providing a pool feeds, where are the press
conferences going to be held?
10Municipal IT infrastructure
- What is the impact, what is the effect, how big
a problem is it, should DR procedures be
implemented? - Are data centers effected? Is there potential for
integrity to be compromised? - Are networks operational? Is there a security
threat? - Are any non-data center based systems adversely
effected? - Are systems critical to support emergency
operations up and running properly? - Web sites, GIS, E-Teams, CRM (311), Email, BES
servers - Have other technologies been adversely effected?
- PBXs, Traffic Pattern Control systems, Building
Control Systems
11Providing IT support for the emergency - Wireless
- Wireless
- Does anyone need radios, satellite phones, cell
phones, blackberries, EVDO? - Do we need to deploy additional batteries,
chargers, accessories? - Is the radio infrastructure stable, deploy back
up systems? - Are there Interoperability requirements? Deploy
ACU 1000s, system level interoperability? - Do we need to deploy Cells on Wheels (COWS),
Cells on Light Trucks (COLTS)? - Do we need to deploy temporary public pay phone
banks? - How is the infrastructure holding up, any systems
failing, Batteries, UPSs, Generators Operating,
is fuel required?
12Providing IT support for the emergency General
Technology
- Is there a need for new office environments to be
established, an EOC, field command centers,
Family Assistance Centers? - Is there a need for credential systems?
- Does anyone need laptops?
- Is there mapping and GIS support required for
press announcements, traffic control and
logistical support? - Are there new applications that need to be
brought up quickly? - Are there IT support staff in EOCs, public
safety agencies that need to be relieved?
13Aftermath of the Attack on the World Trade Center
September 11th 2001
- Needed to Coordinate IT equipment requests,
triage, stage, inventory, distribute, manage FEMA
reimbursement applications, pay vendors.
(coordinate and track donations) Lesson Learned
Develop procedures, establish Emergency IT
Contract - Needed to build facilities rapidly
- OEM Emergency Operation Center Family
Assistance Center - Backup Emergency Operation Center Executive
Offices - Lessons Learned - LAN in a box. (Network
equipment and Notebooks ready to deploy),
Emergency IT Contract, pre-configure - Needed to coordinate the restoration of telephone
service - Lessons Learned Strengthened Mutual Aide
Restoration Consortium Agreements - Developed a GIS capability at OEM Emergency
Operation Center - Implemented Emergency Resource Management
Software - Developed Agency by agency business /continuity
recovery plans - Centralize - City data centers and network fared
very well, agency servers and networks failed
14Crash of Flight 587 - November 12th, 2001
- Developed Mobile GIS capability
- Rapid deployment of Family Assistance Center
15Northeastern U. S. Blackout August 14th 15th,
2003
- Needed to have better control over the
Maintenance of UPSs and Motor Generators at some
of our sites - Needed to strengthen our protocols with the
Telcos, needed insight into their Motor
Generator maintenance and testing - 311 Handled 172,000 calls in a single day.
Harden 311. - 311 provided feedback from public
- 311 helped to relieve 911, 911 experienced normal
volume - Needed additional battery capacity for Public
Safety Radio repeaters spares, portable
generators - Needed satellite phones and radios for executives
- Needed to harden city hall
- Data centers network available, no users
16Staten Island Ferry crash October 15th , 2003
- Needed a system on hand at 311 to track missing,
casualties / next of kin.
17One and a half hour partial 911 outage March
26th, 2004
- 311 received and routed 911 emergencies to
precincts via non-emergency CRM workstations - Lessons Learned strengthen procedure to route
911 calls thru 311, ring down phones between
NYPD/FDNY/EMS - NYPD had work around procedures for 911 outage
- Painstaking change control procedures with
Verizon - The highest levels of redundancy need to be
implemented for 911
18Major Transit Strike - December 20th 23rd 2005
- Needed to develop telecommuting policies and
capabilities - Needed a plan and the capacity to transport 911
and 311 call takers to work and back. - May have needed uniformed NYPD trained to handle
911 calls if 911 call taker attendance was too
low. Needed to have NYPD officers trained. - The benefit of Call center capacity outside of
the region began to be apparent - NYC 311 handled 243,000 calls in one day, normal
911 call volume
19Blizzard, largest recorded snowfall in NYC-
February 12th 2006
- The need for call center capacity outside of the
region was reaffirmed.
20New York City 3-1-1 - Invaluable Everyday
Essential during Emergencies
New Yorkers only need to remember two numbers to
contact City government 911 for emergencies and
3-1-1 for everything else.
- All calls are answered by a live operator, 24
hours a day/7 days a week/365 days a year - Provides immediate access to language translation
services in over 170 languages - Allows callers to quickly and conveniently
- Be directed to a specific City, State or Federal
agency or program - Request detailed information about services and
programs - File a request for City services
- Professional, courteous, knowledgeable,
accessible
21311 Providing Access to Govt. Increasing call
volume
- 3-1-1 has increased the publics access to
non-emergency government services. - Since March 9, 2003, 3-1-1 serviced over 30
million calls, 15 Million/year - 3-1-1 averages approximately 47,000 calls per
day. - Since its inception, the average number of calls
per day has steadily increased - 3-1-1 has set of a service level of answering
over 98 of calls in less than 30 seconds, no
calls on hold beyond 3 minutes.
22New York City --- Why 3-1-1 is necessary