How Can Downtime Ruin Your Budget

1
How Can Downtime Ruin Your Budget

• Carrie Higbie, The Siemon Company

2
Small Catastrophe Downtime

• Hardware failure
• Power outages
• Infrastructure failures
• Noise
• Bandwidth issues
• Poor convergence planning
• Hackers
• Virus attacks
• Bubba stuff
• User error

3
BIG Catastrophe Downtime

• Fire
• _at_ Hackers
• Virus attacks
• Power outages with inadequate UPS/generators
• Cut WAN links or WAN failures
• Flood
• Natural disaster
• Funding cuts
• Sabotage
• User error

4
Biggest Keys to Reliability

• Planning
• Redundant sites
• Geographically dispersed
• TEST
• Did I say TEST?
• Proper Provisioning
• Todays latest buzzword Compartmentalization
• Look back to look forward
• Think outside the box check outside the box

5
What does downtime cost?
6
Fortune 1000 Published Data
Annual revenue / 2080 hour per year revenue
assumptions per hour
7
Employee Costs (National Avg. Wage 33,252.09
1.4 / 2080 weighted hour rate)
of Employees Weighted average Wage / 2080
Hours per year
Average hourly wage based on figures supplied by
the National Bureau of Labor Statistics
8
Revenue Per Hour Per Employee
Annual Rev / 2080 / Number of Employees
Revenue Per hour Per Employee
9
Salary and Revenues Combined 15 of Workforce
Down for One Hour
Average hourly wage based on figures supplied by
the National Bureau of Labor Statistics
10
Formulas

• Revenue per hour
• Total revenue / 2080 hour work week
• Revenue per employee per hour
• Total Revenue / Number of Employees / 2080
• Salary expense per hour (weighted)
• Average hourly wage 1.4 (to include overhead) /
  2080
• Salary expense plus lost revenue
• Total revenue per hour weighted salary expense
  per hour of workforce down at any given time
  (we used 15)

11
How Much Uptime?
12
(No Transcript)
13
Downtime Costs Per Component
14
Cost Over Life of System
15
System Lifetime Graph
16
The Cost of a Slow Network Partially Down

• Examples
• Company A
• Number of Employees 500
• Average Hourly Wage 15.00
• Hours of Productivity Lost per Year 30
• Network Slow Cost 225,000.00
• Company B
• Number of Employees 1,000
• Average Hourly Wage 18.00
• Hours of Productivity Lost per Year 52
• Network Slow Cost 936,000.00
• Company C
• Number of Employees 5,000
• Average Hourly Wage 20.00
• Hours of Productivity Lost per Year 20
• Network Slow Cost 2,000,000.00

Calculate network slow cost Cost P x W x E P
Total Number of hours lost Productivity per
year (weekly minutes/60 x 52) W Average hourly
Wage E Number of Employees on the network
17
What causes slow response?

• Environmental conditions
• Temperature and humidity variations
• EM and RF interference
• High network traffic
• Outdated, slow PCs, NICs
• Poor installation
• Inferior network cabling

18
Other Costs

• Recuperation
• Verification
• Overtime
• Redesign
• Re-qualification
• Second guessing and justifications
• Rework of disaster recovery plan
• Rework of business continuity plan
• Mind altering medications

19
1998
20
Hidden Costs

• Freight
• Package insurance
• Costs to undo workarounds
• Costs to re-route cables / links
• Heavy equipment
• Overtime for security personnel
• Payment for personnel tracking downtime and
  creating downtime reports
• Fees/penalties
• Lost return business
• Employee sabotage
• Notification procedures and costs

21
Infonetics 2003 Greatest Threats

• Network products
• Security products
• Cables and connectors
• Servers
• Applications
• WAN and Internet connectivity
• E-commerce

22
How to Prevent Downtime

• Preventative maintenance
• www.cert.org - security subscriptions
• Planning
• TESTING
• Logging activities
• Security measures
• Contingency and business continuity planning
• Documentation
• REVISIONS
• Pay attention to codes

23
Example 1 Catastrophic Failure

• FIRE

24
Most Companies are Able to See the Obvious Costs
of a Fire
The Tip of the Iceberg Obvious Costs of Fires

• Cost of downtime
• Data loss
• Equipment loss

• Hourly Cost of Downtime and Lost Data
• Energy 2.8 mil
• Telecommunications 2.0 mil
• Manufacturing 1.6 mil
• Financial Institution 1.4 mil
• Information Technology 1.3 mil
• Insurance 1.2 mil
• Retail 1.1 mil
• Pharmaceuticals 1.0 mil
• Banking 1.0 mil

Source IT Performance Engineering Measurement
Strategies Quantifying Performance Loss, Meta
Group, October 2000.
Source IT Performance Engineering Measurement
Strategies, Meta Group, 2000
25
however, the hidden costs of fires that are not
seen can be significant.
The Bulk of the IcebergHidden Costs of Fires

• Productivity/efficiency Loss
• Employee absenteeism
• Clean up
• Medical costs
• Damage to customer relationships
• Loss of competitive advantage
• Litigation
• Damage to public image
• Regulatory requirements
• Contractual obligations

26
According to

• Each day, 323 non-residential structure fires
  will occur . . . .
• Every 5 days, 1 person will die
• Each day, 4 people will be injured
• Each day, 7.4 million in property damage occurs

Fire Loss in the US during 2002, Michael Kartner,
Jr., NFPA Fire Analysis and Research Division,
August 2003
27
NFPA codes and standards represent a set of
minimum fire safety requirements for the
protection of buildings.
28
NFPA 90A is responsible for plenum spaces in
buildings.

• NFPA 90A - Standard for Air Conditioning and
  Ventilation Equipment
• Sets requirements for flame, smoke and fuel load
• 4.3.10.2.6 -- All materials exposed to the
  airflow shall be non-combustible or limited
  combustible and have a maximum smoke developed
  index of 50
• Combustible cables allowed as exception (CMP,
  etc.)
• Requires listing of limited combustible cable

29
NPFA 70 is responsible for plenum cable products
and applications.

• NFPA 70 - National Electrical Code (NEC)
• Does not currently include LCC
• Should correlate with NFPA 90A requirements
• Article 645 permits type CM cable under raised
  floors in computer rooms under certain conditions
• Requires removal of abandoned cable

30
NPFA 13 sets the requirements for sprinklered
buildings.

• NFPA 13 Standard for the Installation of
  Sprinkler Systems
• In sprinklered buildings, use of combustible
  cables in concealed spaces, including plenums,
  requires installation of sprinklers in these
  spaces.
• Use of limited combustible cable does not require
  sprinklers in these spaces.
• Options
• 1. Sprinklers in concealed space
• 2. Cable in conduit
• 3. LCC cable (cost effective)

31
However, in the years that followed, many high
profile cable fires still occurred.
The fire spread horizontally on the tenth floor
and traveled through non-fire stopped
penetrations for communications wiring
October 1986
heat and smoke escaped through the
communications wiring openings that had not been
fire stopped. -- Fire Journal,
January/February 1988
Fire spread rapidly to 6 floors 11 fatalities
occurred due to smoke inhalation
Alexis Nihon Plaza - Montreal, ON
32
However, in the years that followed, many high
profile cable fires still occurred. (Contd.)
Fueled by the insulation, the fire quickly
spread into the groups of cables in the cable
tray and eventually emerged at the top of the
cables. The fire was able to travel horizontally
both in the confined spaces between cables in the
trays and in an open space between the top layer
of cables and the ceiling. -- NFPA Fire
Investigation
May 1988
The smoke and combustion by-products from
burning cable insulation penetrated the entire
building . . . . irreparably contaminating much
of the communications equipment. -- From report
of Illinois Office of the State Fire
Marshall and Illinois Commerce Commission
30 by 40 Fire area 90 Million in damages 17
Day outage Calls disrupted for 4 weeks
Bell Central Office - Hinsdale, IL
33
However, in the years that followed, many high
profile cable fires still occurred. (Contd.)
The fire, reported at 359 AM, jumped
helter-skelter from floor to floor following the
snaking utility products, feeding on insulation
and thick cable-housing and breaking out into
office here and there, fire officials said. --
New York Times
Rockefeller Center - New York, NY
October 1996
34
However, in the years that followed, many high
profile cable fires still occurred. (Contd.)
United Airlines Operations Control Center - Elk
Grove, IL
August 1999
35
Years of investigation and research has yielded
key lessons learned about the link between fires
and cabling. . . .
Fire Investigations Lessons Learned

• Ignition source is often electrical failure
• Cables can add significant fuel load and spread
  fire rapidly
• Mechanism of fire spread follows cable pathways
  and spaces
• Small fires can create extensive smoke and
  non-thermal damage
• Passive fire protection essential
• New cabling technologies can significantly
• reduce vulnerability to fire

36
. . . . and has highlighted the leading source of
fatalities and damage in a fire -- SMOKE.
The Effects of Smoke

• Low visibility in an evacuation
• Dangerous carbon monoxide
• Smoke permeation results in
• Fine particulate carbon and moisture
• Current leakage (shorts) which damage or destroy
  sensitive electronic equipment
• Data loss
• Early component failure

95 of the fire damage . . . . is attributed to
the smoke products and only 5 is caused by the
thermal effects of fire. -- Network
Reliability A Report to the Nation, FCC 1993
37
(No Transcript)
38
Recuperation Costs

• Hardware replacement
• Notification costs (internal and external)
• Data restoration
• Hot site utilization fees
• Overtime
• Emergency maintenance call fees
• Replacement of onsite mirrored components
• Smoke damage
• ETC.

39
Whats in a Contingency Plan?
40
Contingency Planning

• Detection and reaction
• Identification of the problem and notification of
  authorities
• Emergency services
• Environmental agencies
• Physical security
• Reduction of exposure
• HVAC failures
• Fire alarms procedures
• Electrical failure procedures
• Flood and water damage
• Evacuation
• Emergency management team notification
• Flow of procedures

41
Back-up Site Procedures

• Establish control center
• Begin disaster recovery
• LOG EVERYTHING
• Timed events
• Set up things to happen within time blocks
• Communications
• Verify all connectivity
• Disaster recovery checklists

42
Maintain the Following

• Up to date IS organizational charts with contact
  info
• Disaster planning coordinator
• Emergency management team
• Operations team
• Computer operations
• Facilities and replacements
• Cold site preparation
• Misc. support equipment and supplies
• Data entry team
• Data input and control

43
Maintain

• Special projects team
• Technical support personnel
• Data restoration integrity and verification team
• Programming assurance team
• Insurance team
• Internal audit team

44
Requirements Should be Updated Quarterly

• Communications links
• Cable pathways, labeling and terminations
• Power and HVAC requirements
• Computer equipment, vendors, and serial numbers
• Teleprocessing systems and requirements
• Terminal configuration charts

45
Supplier Lists

• Offsite paper copies should be maintained
• Hardware
• Software
• Additional systems
• Custom form suppliers
• Utility vendors

46
Prioritize Applications

• Assign risk assessment number from 1-5
• Assign responsibilities for each
• Update data dictionaries
• Update software revision levels
• Custom programming should be updated
• Source code should be backed up each time there
  is a revision
• Dont forget gateways and other applications that
  use any one application

47
Offsite Storage

• Virtualization
• Live redundant offsite storage
• Cold-site
• Hot-site
• Personnel backup files
• Custom forms at offsite

48
Other Costs

• Recuperation
• Verification
• Overtime
• Redesign
• Re-qualification
• Second Guessing and Justifications
• Rework of Disaster Recovery Plan
• Rework of Business Continuity Plan
• Mind altering medications

49
Hidden Costs

• Freight
• Package Insurance
• Costs to undo workarounds
• Costs to retroute cables / links
• Heavy equipment
• Overtime for Security personnel
• Payment for personnel tracking downtime and
  creating downtime reports
• Fees/penalties
• Lost Return Business
• Employee Sabotage
• Notification Procedures and costs

50
Document all Procedures

• IPL or power up
• Power down
• Schedules
• Operations run procedures
• Disk drive and file layouts
• Security databases
• Listing of all purchased software and serial
  numbers

51
Physical Security and Access Control

• Staff
• Service personnel
• Consultants and outside personnel
• Access control
• Vault access
• Non office hours and after hours contacts
• Security personnel
• Office security

52
Backup Facilities

• Layout
• Hardware
• Software
• Communications
• Contracts
• Testing
• Mock disasters
• Program compilations and complications

53
Most Important

• TEST
• RETEST
• REVISIT
• UPDATE
• AUDIT
• Practice your notification procedures

54
TIA 942 Data Center Standard

• Sets up hot zones for equipment
• All horizontal cables should be run and
  terminated accommodating growth so that it does
  not have to be revisited
• Fire, Life, Safety, Power and Lighting
  considerations
• Distribution areas and Telecommunications Rooms
• Equipment Placement
• Cabling Systems, Cabling Pathways and spaces
• Security and other included systems

55
TR 942 Design Considerations
56
TIA 942 Data Center Standard

• Redundancy
• N Base requirement
• N1 Redundancy
• N2 Redundancy
• 2N
• 2(N1)
• Tiers
• Tier 1 Basic data center
• Tier 2 Redundant components
• Tier 3 Concurrently maintainable
• Tier 4 Fault tolerant

57
Tiers What do they Mean?
58
10 Point Inspection

• Adherence to Standards
• Closet Clean-up
• Documentation
• SNMP or other monitoring and testing
• Identification of weak links
• Re-certify questionable links
• Labeling
• Speed reports
• IP Address listings
• Monitor reports (bandwidth and throughput)
• Replace home-made patch cables

59
Top Offenders

• Improperly terminated cables
• Improperly terminated patch cords
• Lengths exceeded specified maximum
• Cabling was improperly or not labeled
  (troubleshooting problems)
• Cables run over fluorescent lighting causing
  interference
• Electronics and closets in poor locations
  (humidity, EM, RF)
• Cables bunched to tightly causing the pairs to be
  flattened
• Cables tied to electrical conduits or run too
  close to Power panels
• Cabling that did not pass testing due to various
  issues
• Closet spaghetti
• CAT3 cables terminated to CAT 5 100M switches
• Bent fiber exceeding bend radius
• Cables not to spec
• Racks not grounded

60
Top Protocol Issues

• Extra protocols/all protocols being activated on
  the workstations
• Ports forced to 10Mb half duplex no longer needed
  
• Packet over-runs
• Unknown protocols
• Retransmissions
• Routing loops or VLAN errors
• Mystery devices or devices that should not be
  there
• High bit error rates

61
Data Center Solutions

• Blade patch
• Grid ready cabling
• XGLO
• TERA
• 10G6
• Wire management
• Specialty trunk cable assemblies
• IP video and CCTP products
• Dont be DENSE!

62
Shared Media Applications

• WiFi/WLAN
• Shared channel to switch
• Supports 10-20 users per access point
• Network Jack
• Provides additional connectivity where another
  cable cant be run
• Shared channel from switch to switch
• Adds some management features
• VoIP or IP Telephony
• Some implementations use switch inside phone
• Shared media to network switch
• HIGH bandwidth demands due to quality needed for
  voice

63
Powered (PoE) Applications (IEEE 802.3af)

• Power provided over cabling channel
• Can be mid-span (injected power within channel)
  or end-span (in switch)
• Gigabit / end span applications allow for power
  to transmit on data pairs
• 10/100 mid span allow power to be provided on
  non-data pairs
• 500mA absolute limit, peak allowable current
  450mA with a 50mA safety margin (guardbands)
• Port voltage of 44V Maximum 48V
• Most resistive allowable cable (20 ohms round
  trip)
• Cable drops an additional 7V when maximum current
  is flowing and arrives at powered equipment as
  37V
• 37V350mA 12.95W (maximum power to a powered
  device (PD)
• PSE (Power Source Equipment) must detect if
  attached device is standard or powered and drop
  power if not needed

64
802.3af (continued)

• Endspan PSE (at Switch) can operate at
  10/100/1000 alternative A or B or Both
• Midspan 10/100 only alternative B Only

65
WiFi (Wireless Fidelity) WLAN (Wireless LANs)
Ultrawide broadband (802.15.3a) 480Mb/s up to 30
(still in development)
66
10GBT Objectives

• Support operation over 4-connector structured
  4-pair, twisted-pair copper cabling for all
  supported distances and classes
• Define a single 10 Gb/s PHY that would support
  links of
• At least 100 m on four-pair Class F balanced
  copper cabling
• At least 55 m to 100 m on four-pair Class E
  balanced copper cabling
• Support a BER of 10-12 on all supported distances
  and Classes

The equipment technology is there. Is the cabling?
67
Other 10G Copper Standard

• 10GBASE-CX4
• Infiniband connector
• 10G over Twinax to 15m
• Already doubled with TERA
• Same chip with no modifications
• Second generation chips will go full 100meters

68
10Gig Ethernet Over Fiber

• IEEE 802.3ae published June 2002
• www.10gea.org
• Multiple interfaces designed to support MM SM
  media
• Allows limited use of existing TIA/ISO based MMF
• New 50/125µm MMF created for 10GbE
• Estimated 10x cost increase over 1GbE
• No auto-negotiation
• New application for legacy MMF in process

69
Optical GbE Distances
70
Considerations

• Gartner says..
• 20 of all IT purchases are for things that DONT
  work
• Most companies UNDERestimate labor by 50