September 11

1
September 11

• What Worked, What Didnt
• Sean Donelan
• Donelan.COM
• Critical Infrastructure Design

2
Introduction

• Impact on the Internet
• Rumors
• Causes
• What worked
• What didnt work
• Duct tape solutions
• Recommendations

3
Names Omitted

• Individual company names omitted, unless there is
  only one company
• Building addresses used if well-known location
• General description of problems or vulnerabilities

4
Killed and Missing

• World Trade Center 445 people confirmed killed,
  4500 to 5000 people missing
• Pentagon 125 killed
• American Flight 11 92 killed
• United Flight 175 65 killed
• American Flight 77 64 killed
• United Flight 93 44 killed
• Estimated 2,600 citizens from 80 countries
  included in above numbers

5
Impact on the Internet

• The Internet wasnt a target
• You arent a Tier-1 provider if you werent
  affected by something
• Limited network partitioning US/Europe
• Local impact ranged from complete destruction to
  no impact
• Most network disruptions happened hours after the
  initial attack
• Most service disruptions due to problems in edge
  networks

6
Rumors

• 60 Hudson structurally unsound
• FBI seizing ISP equipment supporting terrorist
  web sites
• Military taking over satellite transponders
  shutting down ISPs
• Carrier/Ryder trucks missing/stolen
• Carnivore slowing down the Internet
• Terrorists knew the code name for Air Force One

7
Yogi Berra

• It aint over, till its over.

8
Causes

• Normal disruptions like maintenance, fiber
  cuts, tropical storms, and crackers continue
• Loss of third-party infrastructure
• Operator errors omissions
• Exceeded environmental design
• Direct damage due to the attack
• Software bugs/Hardware failures
• Lack of coordination/planning/information
• Lack of auto-start/auto-boot

9
Gross Performance
10
What WorkedInternet

• Undamaged portions of the Internet continued to
  function (mostly)
• TCP/IP worked (best-effort delivery)
• BGP routing worked
• Multicast routing worked
• Core application protocols (DNS, E-mail) worked
• VOIP (excess capacity, NMC bypass)
• Packet wireless, Blackberry, Richochet, 802.11b
• Carrier Hotels/Colos

11
What WorkedContent

• IRC used to feed live news captions
• Instant Messenger usage increased by and
  estimated 20
• Mirroring/Local caches
• Corporate web sites distributed updated
  information. Non-Internet companies seemed to
  use the web more effectively immediately after
  attack
• Charity fundraising from web sites with help from
  some e-commerce sites
• SPAM, SPAM, SPAM

12
OTooles Commentary on Murphys Law

• Murphy was an optimist.

13
What Didnt WorkComplex Services

• Load-balancing products replaced with DNS
  round-robin
• Generated web pages replaced with direct load
  pages
• Software disk mirroring product didnt
  automatically recover after power failures
• Analog lines repaired first

14
What Didnt WorkSecurity Authentication

• Dialup authentication problems
• Connect, but couldnt login
• Central authentication servers were located in
  other regions
• Several register/pay news web sites suspended
  authentication checks (public service, improved
  performance)
• Difficulties verifying authenticity of requests
  from the government (possible social
  engineering or just FUD)

15
What Didnt WorkCongestion

• Its so crowded, no one goes there anymore
• Well-known news web sites initially overloaded
  (cached by other sources)
• Government web site overloaded (FBI tip site)
• NANOG and other mailing lists posting delays, but
  did deliver
• Unicast (distributed and single source) streaming
  news sources overloaded
• Generally a point-source problem
• Not a backbone capacity issue (yet)

16
What Didnt WorkPOTS/Voice

• Worked but did calls get through?
• Carrier 1-800 call problems
• Cell sites depend on landlines
• ILEC versus CLEC access
• ISPs established new dialup numbers replacing out
  of service numbers
• Call centers were evacuated, who answered the
  phones

17
What Didnt WorkNew York City

• Network-wide effects
• Physical damage in New York City
• Network problems in New York City
• Pentagon and Western Pennsylvania are not major
  public Internet hubs

18
NYC Damage Map
19
What Didnt WorkThe net needs electricity

• Electric substations and grid damaged
• Outside plant carrier equipment not connected to
  the best available backup power source
• Batteries dont last a week
• Generator failures
• Operator turned off generator to save fuel
• Fuel delivery problems
• Lack of maintenance
• Environment exceeded design conditions
• Cooling (HVAC) equipment power supply

20
What Didnt WorkRedundancy Spares

• If only a single circuit exists and it is
  destroyed, no IP traffic
• Most end-users connected by a single circuit
• Multi-homing versus a second circuit
• Limited spare parts stored locally, rely on
  overnight couriers for replacement parts from
  central parts depots
• Non-revenue generating equipment

21
What Didnt WorkDiversity Avoidance

• Equipment in the World Trade Center primarily
  served tenants in complex (shared fate)
• SONET ring through WTC tower 1 and alternate path
  through WTC tower 2
• Damage to 140 West Street central office and
  surrounding underground infrastructure
• Backup circuit routed through same facility
• Advanced data circuits (ISDN/DSL) concentrated
  in a few central offices

22
Duct Tape Solutions

• Cables out windows and manholes and along streets
• Carriers shared working facilities in telco
  hotels to restore service, more carriers
  generally means more facilities
• Carrier provided emergency transit to ISPs in
  Europe to heal breaks in NYC
• ConEd organized generators and fuel truck route
  for many buildings
• Lots of offers of assistance

23
Blaise Pascal

• People are generally better persuaded by the
  reasons which they have themselves discovered
  than by those which have come in to the mind of
  others.

24
Recommendations

• Rumors will happen, must actively share
  information to combat it
• Update government response plans to include the
  Internet and post-1982 telecommunication carriers
• Automatic/Remote operation of backup systems in
  case of evacuation
• Plan for customer service during evacuation of
  call centers

25
More Recommendations

• Pre-plan emergency access with authorities,
  building owner, etc
• Pre-plan load shedding procedures to prevent
  shutting off critical equipment (Note specify
  critical equipment)
• Outside plant network transport equipment
  should be connected to building generator(s)

26
Net Recommendations

• Operators are dangerous, do nothing?
• Weakest link, know your circuits
• Centralized login can create a denial of service
  vulnerability during a crisis
• Using ISDN for out-of-band access may delay
  recovery
• Simple services work best in a crisis
• Diversity, Diversity, Diversity

27
What WorkedWhat Didnt Work

• Questions???
• Sean Donelan