Agenda

1
(No Transcript)
2
Agenda

• Backbone Background
• Architecture Goals
• Backbone Core Layers 1, 2, 3
• Backbone Distribution and Access
• IP Core
• Layer-2 VPNs/VLANs
• Enhancement Options
• Summary

3
Existing Backbone

• ATM LAN Emulation (LANE)
• Core links are ATM OC12 518 megabits/sec
• Node links are OC12 or OC3 129 megabits/sec
• Radial links are 10/100/1000 megabit ethernet
• Marconi/Fore ATM switches
• 6 Cisco IP routers, 400 IP customer subnets
• Cisco Catalyst switches
• Juniper border router

4
Existing Backbone Topology
5
Existing Fiber Infrastructure
6
Goals

• High Availability, Resiliency
• Single device or link failure doesnt
  significantly disrupt customers service
• High Capacity
• High Performance
• Hardware-based wire rate forwarding
• IP Multicast Enabled
• IPv6 Enabled
• QoS Enabled

7
Goals

• Centralized security, abuse mitigation
• Centralized out-of-band management
• Centralized measurement
• Campus-wide layer-2 (L2) Virtual Private Networks
  (VPNs) or Virtual Local Area Networks (VLANs)

8
Optical Core Introduction

• Optical meaning Dense Wave-Division Multiplexing
  (DWDM) on existing single-mode fiber
• Get more "light pipes" (16-32 lambdas) out of
  each existing, nearly exhausted, single-mode
  fiber pair
• Propose utilizing Cisco's flagship optical
  platform 15540 ESPX
• Layer 1 resiliency amongst core super-nodes
• Provide paths for non-IP DoIT services, e.g.
  ESCON, Fiber-Channel for existing Storage Area
  Network (SAN)

9
Optical Core Resiliency
10
Optical Router Connectivity
11
Layer-2/Layer-3 Core

• Utilizing Cisco's 6513 with MSFC-2
• 256 gigabit/sec switch fabric
• 10 gigabit ethernet external links on existing
  dedicated single-mode fiber
• DWDM-based GbE links to isolate VLAN traffic if
  required, for management, specific campus-wide or
  customer VLANs, etc.

12
Layer-2/Layer-3 Core Topology
VLAN trunks 10 GigE
DWDM Lambda GigE
13
IP Core/Border Router
User takes shortest path
14
Layer-2 Distribution Access
15
Why Layer 2 for Distribution?

• Does not require IP renumbering of all campus
  machines
• Enables DoIT to introduce an IPv6 gateway
  immediately to each VLAN without router software
  and hardware upgrades throughout the distribution
  and access layers
• Enables cross campus ethernet VLAN technology,
  which essentially provides wire-rate Virtual
  Private intra-campus Networks (VPNs),
  accommodating our existing VLAN-based services
  and enabling us to add more centrally-managed
  DoIT services.

16
Spanning Tree Protocol101

• Spanning Tree Protocol (STP), standard specified
  by IEEE 802.1d
• Used by virtually every ethernet bridge or switch
• STP discovers ethernet network topology and
  creates loop-free paths which also prevents
  undesired packet replication.
• For reliability, STP detects network failures and
  unblocks alternate paths when necessary.

17
Layer-2 Distribution Access
18
Why Campus-wide L2 VPNs?

• Multi-building departments can share a subnet and
  a common administrative domain.
• Middleboxes can be located in DoIT's 7x24 HA
  facility
• Firewalls, intrusion detection services
• Traffic management devices, e.g. Packeteer
  PacketShaper
• Measurement instrumentation
• Enables us to reconfigure traffic flow beneath
  the IP layer, across the over-engineered core,
  for traffic engineering or Quality-of-Service
  (QoS)

19
Campus-wide VLAN existing Wireless WiscWorld
20
Large Layer-2 Fear, Uncertainty, Doubt?

• No reason to fear Spanning Tree Protocol
• STP is mature, widely-deployed, and we have
  experience with it.
• Presence of potential loops is as much a feature
  as potential problem.
• Without loops, the topology has no redundancy.
  If anything breaks, connectivity is lost. Loops
  should not be viewed as misconfiguration but
  rather as a good design strategy.
• from Radia Perlman's book, Interconnections
  Bridges, Routers, Switches, and Internetworking
  Protocols

21
Disadvantages of Layer 2 Distribution STP?

• Spanning Tree unpredictabilities?
• Yes, traditional Spanning Tree Protocols latency
  would be a problem, as would non-deterministic
  spanning trees resulting from misconfiguration.
• Rapid Spanning Tree Protocol (RSTP, IEEE 802.1w)
  will be employed. Root link priorities will be
  configured to determine tree structure.
• The IEEE 802.1w specification provides for
  sub-second reconvergence after failure of one
  of the uplinks in a bridged environment. Cisco
  Catalyst OS Software Product Bulletin
• Configuring RSTP is not necessarily harder than
  configuring resilient IP routing.

22
L-2 Disadvantage Inefficient?

• Some backup uplinks will not be utilized because
  spanning tree will cause them to be unused until
  a failure occurs.
• Yes, but link-based load balancing is unnecessary
  because our single links are over-engineered to
  have excess capacity.
• However, both uplinks will be utilized between
  core and distribution layer devices.

23
L-2 Disadvantage Backhaul?

• Traffic between campus customers is sometimes
  unnecessarily backhauled to the super-node.
• True, however the campus topography, with
  relatively short distances to buildings,
  liberates us to backhaul traffic to central
  locations when convenient or economical.
• We propose that very little backbone traffic is
  exchanged between unrelated campus customers.
• Link utilization is not a problem in our current
  ATM network, which often backhauls all traffic
  onto existing ATM links.
• Campus measurement has shown that 50-90 of
  customer traffic is extra-campus.

24
L-2 Disadvantage Multicast?

• IP multicast traffic will sometimes be
  unnecessarily replicated on distribution links.
• Yes, this will happen whenever users in two
  different departments, which happen to be
  geographically close, join the same multicast
  group.
• There is an upper bound to the number of
  replications one per customer VLAN.
• High quality multicast video streams have rates
  of about 1Mb/s. If replicated to ten customers
  might reach 1 utilization of a Gigabit ethernet
  link.
• We propose that over-engineered High Capacity
  links and IGMP snooping will be sufficient to
  deal with IP multicast replication.

25
L-2 Disadvantage Manage?

• Network structure is hidden from end users,
  support staff and, to some degree, network
  engineers.
• True, in large layer-2 Networks the end user
  cant easily determine, nor report, which network
  devices are reachable.
• Our documentation, existing tools, and perhaps
  new network management tools will help.
• CiscoWorks Campus Manager advertises
• Intelligent discovery and display of large Layer
  2 networks
• Diagnostic tools for connectivity problems
• Layer 2 and Layer 3 path trace between source
  and destination
• Export of topology maps to Visio

26
L-2 Disadvantage Table sizes? Forwarding Table
Utilization

• Existing campus backbone routers
• Layer-2 (MAC) addresses seen since 2001 40,895
• since 1998 75,990
• Layer-3 IP addresses seen since 1998 65,063

• Catalyst 6500
• Supervisor Engine 2 including PFC supports
  128,000 entries (same for L-2 or L-3)
• Catalyst 3500
• Supports 8K-12K entries

27
Layer-3 vs. Layer-2 Distribution
28
Layer-3 vs. Layer-2 Distribution
29
Layer-3 vs. Layer-2 Distribution
30
Layer-3 vs. Layer-2 Distribution
31
Large Customer IP Connectivity

• Layer-3 Dual Route Connection
• OSPF convergence characteristics

32
Small Customer IP Connectivity

• Layer-2 Switched Ethernet Connection
• HSRP/VRRP Fail-over
• OSPF route option
• Host/server-based resiliency
• Authenticated OSPF

33
Small Customer IP Resiliency
Backup Gateway
Primary Gateway
34
Estimated Equipment Costs
35
Enhancement Options More Access

• Enable routing on super-node switches for
  additional customer access capability
• Mesh routers using DWDM

36
Enhancement Options More Core

• Add routers as IP-only core
• Build an IPv6 core

37
Enhancement OptionsDoIT Platform as Large
Customer
L-2 Switches
Routers
38
Design Enhancement Options Lab

• Complete lab environment
• Emulate entire working network
• Load testing
• Fail-over testing
• Hands-on training
• Experiment with new topology ideas, services
• Test new software builds

39
(No Transcript)
40
Single-mode Installations by June 2003
Future Backbone
41
Summary Goals

• Centralized IP Traffic Measurement
• NetFlow v6 collection from routers potentially
  available for Catalyst 65xx switches as well.
• Centralized security, abuse mitigation
• DoIT can host centrally located transparent
  firewall equipment for LANs.
• The Layer-2 MAC addresses of individual user
  machines will be visible to us in the core. This
  information has proven to be invaluable in
  identifying and mitigating network abuse.

42
Goals

• Layer-2 VLANs
• Existing VLAN-based services such as Wireless
  WiscWorld, Residence Halls Packeteer PacketShaper
  (rate limiter), and customer VLANs, such as
  School of Education, can migrate over as-is.
• Quality of Service (QoS)
• While we haven't yet had the need to implement
  multiple grades of intra-campus network service,
  modern equipment offers ethernet 802.1p Class of
  Service (CoS) at layer 2 and IPv4 Diff-Serv Code
  Points (DSCP) at layer 3, and the ability to map
  between them.

43
Goals

• IPv6
• Multicast
• Our border router is the Rendezvous Point (RP)
  and we use PIM Sparse Mode (PIM-SM) on 3 or more
  campus routers.
• High Capacity
• 10 Gigabit Ethernet core
• DWDM amongst super-nodes
• Gigabit Ethernet to distribution nodals
• Gigabit Ethernet to access radials where and when
  available

44
Campus Goals

• High Performance
• Layer-2 and Layer-3 wire rate forwarding
• Software-based VPNs not required for cross-campus
  VLANs
• Resiliency
• At Layer-2, link and equipment redundancy to the
  customer locations.
• At Layer-3, multiple gateway routers, HSRP/VRRP,
  and multiple paths between customers and campus
  core routers.

45
Closing Statement

• Our design meets campus needs by utilizing the
  best Cisco gear available.
• Reliability will be improved and a ten-fold
  increase in capacity and performance achieved.
• Ethernet switches rather than IP routers will
  limit ongoing maintenance time and costs.
• Campus-wide Virtual LANs effectively solve
  campus-specific problems.
• Retaining a large Layer-2 infrastructure enables
  us to benefit from the industrys recent ethernet
  transport improvements.

46
(No Transcript)