Avici TSR An overview True scalable routing

1
Avici TSR An overviewTrue scalable routing

• Ides Vanneuville
• Systems Engineer - EMEA

2
Agenda

• Introduction
• Next (best) generation networking
• Software features
• Hardware features
• Summary

3
Who is Avici ?

• Founded 1996 Billerica, MA
• Mission To market scalable core routers that
  delivers service providers reduced capital and
  operational expenditures and simplified
  operations without service disruptions
• Public Company (Nasdaq AVCI), 240 million
  raised
• 400 employees worldwide, 11 in Europe
• Gained market share (in last 12 months)
• 2 in core router market (1G links)
• 6.5 in core router market (10G links)

4
Explosive Demand for Data Services

• Peak-Hour Internet Bandwidth Demand

(terabits per second)
120 CAGR
Source Pioneer Consulting LLC
5
Connected Router Infrastructure

• Technical Problems
• Blocking POP
• Limited redundancy
• Complex redundancy
• Configuration complexity
• Routing complexity
• Routing Protocol Scaling
• Management System Scaling

Backbone Routers
AccessRouters

• Business Problems
• Floorspace
• Management
• Power
• Equipment Costs
• Upgrade Granularity

Solution?
6
Avicis Solution
Fabric
AccessRouters
Access Interfaces

• Blocking
• Limited hardware resilience
• Excessive software resilience
• Complex Failure Modes
• Coarse scalability
• Excessive network topology

• Non-blocking
• Rich hardware resilience
• Appropriate software resilience
• Simplified failure modes
• Granular scalability
• Efficient network topology

7
Next generation network
Avici
Avici
Avici
Avici
8
Why next generation network?

• Increase revenue deploying new services
• High-speed Internet access
• IP transit services IP bandwidth whole sale
• Virtual Private Networks (L2 L3)
• Voice over IP - Multimedia
• Application Service Provider connectivity
• Decrease cost converging networks

ATM services
IP services
ATM services
ATM services
ATM services
ATM services
ATM services
ATM services
Leased line services
Frame relay services
SDH
Optical
Voice
9
Building Blocks for CN

• The converged network must encompass the service
  levels of predecessor or legacy networks
• Composite links, QoS and MPLS is the enabling
  technology!

10
Virtual Private networks

• Layer 3 VPNs
• For IP based traffic
• Point-to-point
• Point-to-multipoint
• Can be combined with other services. E.g.
  Internet access.
• Layer 2 VPNs
• Transparent LAN services
• Transport over IP network
• Virtual Leased Line (VLL)
• IPSec
• Encrypted data
• Transport over IP network

• VPN requirements
• VPN edge equipment
• Scalable IP core
• Grow without pain
• Service differentiation (QoS)
• MPLS TE
• VPN provisioning tool

Source Telechoice 1999
11
More services

• Voice over IP (VoIP)
• Controlled bandwidth
• Low latency
• Controlled jitter
• Network resilience
• IP transit IP wholesale
• Scalable capacity
• Scalable connectivity
• VPN services
• Network resilience

• Multimedia
• Controlled bandwidth
• Low latency
• Controlled jitter
• Robust multicast environment
• Scalability

SP1
SP2
SP2
12
Role Differentiation Is Key
13
Carrier design principles

• Architected to suit to Carrier design principles
• Design network
• Establish and stabilize topology
• Acceptance tests
• Transition to operations
• Add customers, expand capacity cycle

14
Software features
15
About IPriori

• Unicast Routing
• Fully featured BGP-4 Implementation
• Confederates, Route Reflectors, damping, policy,
  Route Redistribution, Aggregation, dynamic
  configuration, peer security
• IGP Support
• IS-IS
• Level 1 and 2 support, Route Redistribution,
  Database overflow support, passive interface
  support
• OSPF
• Stub Area support, Virtual links, Router
  Summarization, Passive Interface support, Route
  Redistribution
• Multicast Routing
• PIM Sparse Intermediate Point
• PIM Dense/Sparse, Auto-RP, MSDP, MBGP for mcast
  NLRI

16
Easy Migration - Industry CLI

• server-id 2 upper
• hostname bay_TWO_UPPER_server
• system-password 7 S9bQQdb9Sd
• interface Loopback 0
• ip address 6.6.6.1 255.255.255.0
• no shutdown
• interface Ethernet 0
• ip address 10.5.222.1 255.255.0.0
• no shutdown
• module 1/15 4xoc3c
• no shutdown
• interface pos 1/15/1
• ip address 1.1.1.1 255.255.255.0
• crc 16
• no keepalive
• sonet scramble-atm
• no shutdown
• module 1/17 1xoc48c
• no shutdown

router bgp 4293 bgp router-id 1.1.1.1 bgp
dampening route-map 1 redistribute connected
route-map connected-bgp redistribute static
route-map static-bgp neighbor ebgp peer-group
neighbor ebgp send-community neighbor 3.1.1.2
remote-as 200 neighbor 3.1.1.2 peer-group ebgp
neighbor 3.1.1.2 timers 6 neighbor ibgp
peer-group neighbor ibgp remote-as 4293
neighbor ibgp send-community neighbor ibgp
route-map IBGPMAP out neighbor 1.1.1.2 remote-as
4293 neighbor 1.1.1.2 next-hop-self neighbor
1.1.1.2 route-reflector-client neighbor 1.1.1.2
timers 180 neighbor 2.1.1.2 remote-as 4293
neighbor 2.1.1.2 next-hop-self neighbor 2.1.1.2
route-reflector-client neighbor 2.1.1.2
send-community neighbor 2.1.1.2 timers 180
neighbor 2.1.1.2 route-map IBGPMAP out neighbor
10.10.10.2 remote-as 100 neighbor 10.10.10.2
timers 180

• ip route 5.5.5.0 255.255.255.0 null0
• ip community-list 1 deny 45940756
• ip community-list 1 deny 46071828
• ip community-list 1 deny 46006292
• ip as-path access-list 1 permit
• ip as-path access-list 100 deny _1_
• ip as-path access-list 101 permit _65010_
• ip as-path access-list 286 permit _286_
• ip as-path access-list 286 permit _3561_
• ip as-path access-list 286 permit _701_
• route-map EBGPMAP permit 10
• set local-preference 25
• set metric 150
• route-map IBGPMAP deny 10
• match as-path 100 286
• route-map IBGPMAP permit 20
• set community 45940837
• set local-preference 80
• set metric 10

17
Management and Security

• SNMP
• Standard and vendor-specific MIBS
• RMON Events and Alarms Group
• 64bit counters
• System event log for all events
• Protocol activity (tracing)
• Local and remote SYSLOG
• Log all CLI commands
• Standby Route Servers
• Dual Servers can be deployed for additional
  redundancy
• Traffic separation
• All ForMe traffic is classified into 24
  application-specific queues serviced by WRR to
  avoid starvation of control traffic and ensure
  against DoS attack
• Configurable for ForMe Traffic Filters
• Security
• TACACS
• Secure Shell

18
Network and Traffic Engineering

• Leverage emerging D-WDM Technology
• Get ahead of the serial bandwidth curve
• Stabilize topology
• Enable Rapid Provisioning
• Simplify the IP routing mesh
• Provide Protection options
• Provide TE network for the transport of Diff-serv
  traffic
• Enable Circuit Emulation/Private Line Service
• Enable 45ms restoration

19
Composite Links

• Combines up to 64 physical connections to a
  single logical connection
• Removes serial bandwidth limitations
• Granular bandwidth scaling
• 4-1 speed mismatch
• Preserves packet sequence
• Add/remove trunks dynamically
• Recovery in 45ms
• Compartmentalizes link failure
• Routing table stability

! interface pos 2/29/1 no ip address no
keepalive sonet scramble-atm no
shutdown ! interface composite-link TSR1_CL ip
address 100.100.100.1 255.255.255.0 peer default
ip address 100.100.100.2 member-link POS 1/22/1
member-link POS 1/37/1 member-link POS 1/39/1
member-link POS 2/29/1 no shutdown
20
Composite Links intelligent optics

• TSR monitors load on composite links
• If total premium traffic exceeds a limit, shifts
  ports from other links

Composite Link4 ?
Optical Switches

• TSR signals optical switch to reassign
  wavelength to high traffic link
• Dynamic reallocation of total network

Composite Link3 ?
Label Switch Router
21
SuperCOM 2001 demo
22
MPLS Traffic Engineering

• Ingress, Midpoint, and Egress
• Penultimate Hop Popping is NOT mandatory
• Ingress behavior allows unparalleled tunnel
  scaling
• Constraint Based Routing with Avici enhancements
• Enhanced RSVP signaling for Label Distribution
• Enhanced OSPF and IS-IS to support TE
• LDP
• End to end TE instrumentation for network mngt
• Tight Integration with Composite Links
• Hardware label switching on OC-3, OC-12, OC-48,
  OC-192 and Gigabit Ethernet

23
Traffic Engineering
EGRESS
LSR
K
Shortest Path to
ISPs 1, 2, 3, and 4
B
Short Cut Tunnel 1
INGRESS
E
LSR
C
A
EGRESS
LSR
M
Subscriber
Router G's Next Hop
ISP 4
For Routes in ISP 4
24
Optimized Path Layout

• Intelligent MPLS Reflood Timing
• Improve responsiveness of Ingress path layout
• Resilience Optimization
• Recover from MPLS failures with minimal impact to
  core topology
• Adaptivity Optimization
• Take full advantage of positive changes in the
  core topology
• Rebuild LSPs, ordered by required bandwidth

25
Intelligent Reflood Timing

• IGP Flooding for TE
• If midpoint cannot honor bandwidth reservation at
  the requested priority level the midpoint will
  immediately queue a BW TLV to IGP
• This keeps the ingress up-to-date
• Otherwise MPLS reflood timer is provided
• Immediate
• Pacing
• Piecewise Linear Model
• 99 3s 9510s 90 20s 50 180s 0600s
• Insures that higher bandwidth demands are flooded
  more frequently

26
Resilience Optimization

• Controls response to failures in LSP path
• Triggered by link loss, LSR loss, preemption
• Three methods to enact repairs
• Immediate Response
• Pacing
• Sorted-Pacing
• Sequence LSP creation based on their demand for
  bandwidth at an associated priority level
• Piecewise Linear Model
• Insures that LSPs with higher bandwidth demands
  are signaled first

27
Adaptivity Optimization

• Controls re-optimization of LSP layout when the
  topology changes
• Responsible for two activities
• Timing of new CR-SPF
• Timing and sequencing of LSP re-layout
• New SPFs might be necessitated based on IGP and
  IGP-TE flooding
• Adaptivity Methods
• Disabled
• Pinned
• Periodic
• Piecewise Linear
• Applies to both SPF and Reroute

28
Head-end Reroute

• IGP or RSVP signaling indicates that a
  reservation along the primary path can no longer
  be sustained
• Administrative Change to LSP entity
• Congestion along a given LSP
• Failed Link
• Failed Node
• Headend LSR constructs a new constrained SPF to
  Egress
• Utilizes a Make-Before-Break algorithm for new
  path selection
• Creates a new primary LSP
• Restoration time is a function of path
  complexity!
• Restoration is restricted to the entire path

29
Integration with Composite Trunks
Compose Trunk
Subnet 1
Subnet 1
Subnet 2
Subnet 2
Higher Pre-emption value affects tunnel
rerouteRe-route time affected by reservable
bandwidth attribute
30
MPLS Fast-Reroute Protection

• Router switches traffic to pre-configured backup
• Allows identification of which traffic to protect
• Traffic switched to backup and traffic normally
  on link contend for resources using QoS no
  traffic is discarded if link is not over-utilized
• Works for POS and gigabit Ethernet

31
MPLS Fast-Reroute Drafts

• draft-atlas-rsvp-local-protect-interop-01.txt
  (Avici)
• draft-swallow-rsvp-bypass-label-01.txt (Cisco)
• draft-gan-fast-reroute-00.txt (Juniper)

32
Future tunnel resilience options

• Composite Path
• Leverages same mechanism as composite trunks
• Establish multiple tunnels for same prefixes
• Unequal tunnel size support
• Fast headend re-route
• Fast Reroute
• Midpoint repair mechanism
• Failure affinity groups
• Backup bandwidth pre-reserved
• Midpoint reroute in 45ms

33
VPN Support RFC 2547

• VPN-P Support
• Support for Multi-Protocol Extensions to BGP-4 to
  support route-distinguisher
• Compatible with Cisco, Juniper, Unisphere,
  Riverstone, Alcatel, etc.
• Traffic Engineering/VPN tunnel integration
• Provide a TE core for Virtual Private Networks

34
QoS Functional Summary

• Combination of WFQ, WRR, WRED and GPRA (CBR)
• Programmable ASIC Pipeline
• Multiple hierarchy of Algorithms
• Many iterations per algorithm implemented
• Minimum packet data handling
• Functionality increased for Multiservice module
• Counters maintained per interface,VC,class
  traffic management and accounting information

35
QoS Treatment per node
36
MPLS TE/Composite Trunks/QoS
WRED discardstunnel two orhop-by-hop
trafficduring tunnel reroute
Compose Trunk
Subnet 1
Subnet 1
Subnet 2
Subnet 2
QoS operates on Diff-serv inferred MPLS exp
marking
37
Class-based traffic tunnels
Tunnel establishmentFiltering on next-hop and
DSCPCombined QoS packet treatmentAggregate or
per-class tunnels
Dynamic Tunnel mapping to QoS traffic
classesClasses configured on a per node
basisSignaling interactionLabel inferred QoS
packet treatment assignmentConfigurable packet
treatment
3
5
6
1
protected forwarding table
38
Traffic Engineering with QoS

• Enables no-loss, low delay variance, low latency
  services
• Virtual Leased Line
• Voice over IP
• Storage Area Networks
• Add new properties to Service Level Agreements
• Improve reliability at reduced cost
• Deliver protection appropriate for each service,
  up-to SONET APS levels
• Improve utilization in network backbone

39
Hardware features
40
The Avici Systems TSR

• Carrier class features
• Carrier class design
• Unprecedented scaling
• Future proof

41
Patent-pending Velociti Fabric

• Toroidal direct connect fabric
• Scales to 560 active modules
• Each element adds switching forwarding capacity
  
• Generation 1 - 60 Gbps
• Generation 2 - 240 Gbps
• Each module connects to6 other modules
• Path diversity through fabric
• Resilient
• Non-blocking

42
Velociti - Non-uniform 3-ary n-cube
Generation 1
Generation 2
Generation 3
AlternateMinimumPaths
Axis Wrapped
43
Fabric Forwarding
1. 6 Physical Axis 2. Virtual Channel
Overlay 3. 24 Path Diversity
6x10G Axis - Generation 16x40G Axis - Generation
2
Link State Fabric Routing Flow invariant path
selection
3636 byte fabric unit Wormhole forwarding
44
Backplane Mesh Topology
45
Growing the TSR System

• Expanding the interconnect
• Two physical backplanes in each bay
• Connected top-to-bottom
• Connected side-to-side
• Connected over-the-top
• Power Distribution
• -48VDC 60A per shelf redundant feed
• Environment Controllers
• Two per bay interconnected between bays

46
Avici Product Scalability
1 TSR 400 Gbps
6 TSR Bay 2.4 Tbps
4 TSR Bay 1600 Gbps
2 TSR 800 Gbps
SSR/TSR 600 Gbps
1 SSR 200 Gbps
2 SSR 400 Gbps
47
TSR Interface Modules

• Packet over SONET/SDH
• Concurrent IP and MPLS
• 1 port OC-192c/STM-64
• 1 port OC48c/STM-16
• 2 port OC48c/STM-16
• 4 port OC12c/STM-4
• 4 port OC3c/STM-1
• 16 port OC3c/STM-1
• Ethernet
• 2 port GbE
• 8 port GbE Q2/02
• 10GbE 2H/02

48
Module I/O Architecture
49
Software System Architecture
Routing and Configuration tasks
Forwarding and Fabric Routing tasks
Bay Environment Controller
Bay Environment Controller
50
System Assurance Testing

• Regression
• Fully Automated conformance testing
• Over 2000 scripts for link layer/protocol/function
  testing
• Stress and large topology
• Highly accelerated software live cycle testing
• Tests combination forwarding, IGP/EGP topology,
  features and operator interaction
• Topology Testing
• Fully connected TSRs
• Automated customer topology testing
• Custom inhouse testing tools
• Topology Simulation
• Packet Forwarding
• Industry leading QoS test tools
• Strategic Co-development relationship with
  Netcom/Spirent
• ADTECH AX4000

51
Field proven operation

• First systems deployed in NTON (DARPA
  Supercomputer research network)
• Operational in the Internet in May 1999 (BGP)
• Currently deployed in one of the worlds largest
  carrier IP networks
• Support OC-192 links
• Operating as BGP Route Reflector in one of the
  worlds largest RR configurations

52
ATT network
53
Summary
54
Summary - Carrier imperatives

• Cost
• Lower the cost of building and operating the
  network
• Velocity
• Decrease the time to add or change services
• Profit
• Deliver value added services and quality of
  service enhancements

55
Lower Operational Costs
Backbone Interfaces
Backbone Interfaces
VS.
High Speed(Optical) IP Transit
Low Speed (electrical) IP VPN / Low Speed Transit
Low Speed (electrical)
High Speed(Optical)
Customer Facing Interfaces
Customer Facing Interfaces
Current best practice
Avici alternative
56
Provisioning - Months to Minutes

• Backbone capacity increased without truck-rolls
• Market share not impacted by provisioning
• Immediate provisioning for all bandwidth products
  up-to physical network capacity

Immediate fulfillment of customer demands
57
Velocity Composite Links

• Avoid hitting the transmission capacity wall
• Backbone links with 64 times the capacity of our
  competitor on the same transmission network
• No impact to customer or network stability

Backbone Capacity added in days not months
58
Profit

• Use IP core for converging all services
• Scale services past competitors offerings
• Spend your time creating new services
• Make at the edge of the network
• Save at the core of the network

59
Conclusion

• Next generation public network switch
• Scalability
• Non-stop system and software architecture
• Quality of Service
• Network and Traffic Engineering
• Benefits
• Reduce capital costs
• Stable service velocity
• Increased equipment life cycle

Faster, stable network expansion dramatic
reduction in cost per bit delivered
60
Thank You