Design of a Diversified Router: Model and System Overview

1
Design of aDiversified RouterModel and System
Overview
Jon Turner, John DeHart, Fred Kuhnsjon.turner_at_wu
stl.edu, jdd_at_arl.wustl.edu, fredk_at_arl.wustl.edu ht
tp//www.arl.wustl.edu/arl
2
Outline

• What is NOT covered in these slides
• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

3
Not Covered

• Control
• Installation
• Configuration
• Initialization
• Monitoring
• End Host
• Substrate/Meta Model
• Details about how PlanetLab works.
• These are very important topics but for now we
  will talk about the Data Path in the Network.

4
Outline

• What is NOT covered in these slides
• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

5
System Architecture

• General purpose blades.
• shared blades run Plab OS
• no change to current apps
• also support dedicated blades
• use separate blade server to preserve ATCA slots
  for NPs
• NP blades.
• support dedicated PEs
• control from Vserver on PE/GP
• shared PE options
• shared NP for fast path
• shared NP with plugins
• 10 GE fabric switch
• VLANs used to isolate metarouters
• uplinks for connecting to multiple chasses
• Good ratio of PEs to LC 31

compute blade with disk
Radisys7010
Radisys 7010 with RTM
1 GE for control 10 Gb/s for data
6
Sharing NPs

• Motivation
• MRs with limited IO bandwidth cant make full
  use of NP blade
• extreme (but perhaps common) case 25 MRs with
  100 Mb/s IO
• useful to allow such MRs to share a PE
• Packet flow
• ingress LC terminates external tunnel and adds
  internal header
• MR fast path determines outgoing metalink and
  queues packet
• selected packets directed to Vserver on shared GP
  blade
• injected packets placed on metalink queues in PE
• Configuration
• single NP blade hosts two shared NPs
• each NP engineered for 5 Gb/s throughput
• divide TCAM between NPs

7
Shared PE Processing

• code constraints
• finite, acyclic flow graph
• bounded path length
• restricted memory accesspacket buffer, control
  block, output area
• thread-safe

• inputs (in regs)
• buffer pointer
• output MI
• ctl blk pntr
• lookup result pntr
• output
• buffer offset

use MR to find parser code and MR control block
header includes MRMI

• inputs (in regs)
• buffer pointer
• MI
• ctl blk pntr
• output pntr
• output
• 16B lookup key

per metalink queues with static rates

• input MRlookup key
• output
• output MI
• qid (supplied by substrate)
• lookup result
• stats index

• Stats module not shown inputs from DeMux,
  Lookup and QM.
• Potential for plugins using spare MEs
• one ME per MR, static SRAM regions,

8
Managing MR-Specific Data

• MR control software runs on separate PE.
• xScale provides interface through control daemon.
• enable/disable MR (input packets discarded when
  disabled)
• read/write MR control block
• block read/writes to from memory area using
  offsets
• read/write filters
• substrate fills in hidden fields (MR in key, qid
  in result)
• read statistics
• read/write code segments
• cryptographic hash used to verify
  compliance-checking
• only when MR is disabled

9
Line Card Functions

• Substrate functions only
• Ingress
• terminate tunnels (VLAN, plain IP, IP tunnel,
  MPLS)
• map physical interface tunnel id MLI to MRMI
• map MRMI to destination blade and qid
• queue packets in queues with static rates (paced)
• provision for xScale to respond to ARP requests
• Egress
• arriving packets include MRMI(next-hop-metanet-a
  dr)
• monitor rates for each MR/MI pair raise alarm
  if threshold exceeded
• if MI is designated as multipoint,
• map next-hop-metanet-adr to MAC adr and write
  into blank spot in buffer
• if no matching entry, pass buffer pointer to ARP
  handler in xScale
• xScale handles ARP, fields reply and enqueues
  packet when reply received
• if no reply, send exception notification MR
  exception interface
• place packet in outgoing per-MI queue

10
Possible Extensions

• Queueing in shared PE
• allow MR to define multiple queues per MI with
  queue weights, per queue space allocation and per
  MI space allocation
• substrate implements fair queueing scheduler
• WDRR or stratified round-robin
• packet discarded if queue using more than its
  space allocation and more than its share of the
  shared space
• implement when users demand it or spare time
  available
• Limited Plugins
• replace Header Formatter with parallel block of
  MEs, one per MR
• code restrictions (static and run-time checks)
• access to restricted address ranges
• restricted use of bus bandwidth resources
• instructions inserted to increment usage counters
• xScale (or perhaps dedicated ME) checks counters
  periodically
• MRs using excessive bus bandwidth are disabled

11
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

12
Schedule

• Two time horizons
• techX Project
• End of 2006
• Future Diversified Router Project
• Outstanding new grant request under which we do
  the rest of the work
• JDDs Proposal techX End of Year Target
• Data Path
• Default IPv4 MR
• We should be able to use N instances of this to
  show multiple MRs running in a Substrate Router.
• Legacy PlanetLab nodes in a Blade Server
• By Legacy we mean receiving/sending plain IP
  packets, no Meta/Substrate functionality present.
• Basic LC Substrate functionality
• IP Tunnel Substrate Link termination
• Plain IP handling
• MetaLink Loopback Block
• Control?
• Install an MR
• Configure an MR?
• Routing Protocols?

13
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

14
Substrate/MetaNet Model

• The Substrate should provide a view to Meta Nets
  that their Meta Routers are directly connected
  via Physical Links
• A Meta Interface emulates a Physical Interface
• A Meta Link emulates a Physical Link
• The Substrate should provide isolation between
  Meta Nets

Meta Net A
MR_A
MR_A
MR_A
MI
MI
MI
MI
MI
MI
Meta Net B
MR_B
MR_B
MI
MI
MI
MI
Meta Net C
MR_C
MR_C
MR_C
MI
MI
MI
MI
MI
MI
15
Substrate/MetaNet Model

• Substrate Links connect Substrate Router Peers
• A Substrate Link is terminated at a Physical
  Interface of a Substrate Router.
• Meta Links connect Meta Router Peers
• A Meta Link is terminated at a Meta Interface on
  a Meta Router

MetaLink 1
Substrate Link
MetaLink 2
MetaLink 3
MR_A
MR_A
MR_A
MI
MI
MI
MI
MI
MI
MR_B
MR_B
MI
MI
MI
MI
MR_C
MR_C
MR_C
MI
MI
MI
MI
MI
MI
Substrate Router X
Substrate Router Z
Substrate Router Y
16
Mapping the Model onto Hardware
Meta
Dedicated NP
Dedicated GP
Shared NP
Shared GP
Switch
LC
Loopback
Substrate
17
Mapping the Model onto Hardware

• Line Card
• Typically supports multiple Physical Interfaces
• Our Radisys blade when implementing a LC would
  have an RTM with 10 1-GE physical interfaces.
• Substrate Link Termination
• Including IP Tunnel Substrate Link termination
• Mux/Demux of Meta Links into/out of Substrate
  Links.
• Pass through Meta Links
• Plain IP
• Switch
• Substrate Switch
• Meta Routers on shared blades do not have a view
  of the switch.
• All switching functions are provided by the
  Substrate
• Traffic from all MRs is isolated from other MRs,
  including dedicated blade MRs, via the use of
  VLANs
• Meta Switch
• An MR on a dedicated blade has a view of a
  Meta-Switch including
• Port its blade is on
• Portions of LCs that it has access to.
• Loopback (next slide)
• NP Network Processor Blade

18
Mapping the Model onto Hardware

• Line Card
• Switch
• Loopback
• Provides MRxMIi connectivity to MRyMIj within
  Substrate Router
• More on this later when we provide details on the
  components
• NP Network Processor Blade
• Dedicated entire blade is dedicated to one MR.
• No Substrate functionality present.
• Shared blade is shared among several MRs.
• Substrate functionality provides resource sharing
  and isolation
• GP General Purpose Processing Blade
• May be realized by a blade in the ATCA chassis or
  in a blade server directly connected to switch
  blade which resides in ATCA chassis.
• The switch blade has front panel Ethernet
  interfaces (uplinks) that could be connected to
  the blade server for this purpose.
• Dedicated entire blade is dedicated to one MR.
• No Substrate functionality present.
• Shared blade is shared among several MRs.
• Substrate functionality provides resource sharing
  and isolation.

19
Model Layering and Abstractions

• Two Types of Traffic
• Legacy
• MAC layer
• Legacy Layer (i.e. non-Substrate, typically IPv4)
• Substrate
• MAC layer
• Substrate layer
• Meta layer
• Figures below are LAYERS, not Packet formats

20
Layer Attributes

• Attributes at the different layers
• Physical/MAC
• Type (Ethernet, ATM, )
• Wireless (1/0)
• Type (P2P, MA, )
• Substrate
• MNid
• MRid
• MLI
• Link Type
• Link Model (P2P, Multi-Access)
• ARP support
• Meta
• MetaLink
• MI
• MnAddr
• What is exposed to the MR?
• Type Ethernet, Wireless,
• P2P vs. Multi-Access

21
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

22
Traffic at a Substrate Router

• For now assume traffic is on wired Ethernet.
• Substrate
• EtherType Substrate
• Each frame has a Substrate Header
• MLI Meta Link Identifier
• Identifies MRMI to which this frame should be
  delivered
• Len
• Line Card
• sends Meta Frame to the Blade hosting the MR
  indicated by the MLI
• includes in the Meta Header the MRs MI that this
  frame is for.
• IP
• Other

23
Traffic at a Substrate Router

• Substrate
• IP
• EtherType IP
• Substrate Tunnels
• IP Proto Substrate
• Each frame has a Substrate Header immediately
  following IP Header
• MLI Meta Link Identifier
• Identifies MRMI to which this frame should be
  delivered
• Len
• Line Card sends Meta Frame to Blade hosting MR
  indicated by MLI
• Included in Meta Header is the MRs MI that this
  frame is for.
• Plain IP Control Protocols
• Contains no Substrate Header
• ICMP
• Substrate on LC may need to process some ICMP
  messages.
• ICMP or a subset of ICMP messages sent to XScale
  on LC.
• Depending on processing in Substrate/XScale, ICMP
  messages may be forwarded on to MR(s).
• BGP Border Gateway Protocol (inter-domain
  routing protocol)
• Treated like Plain IP Data described below Sent
  to default IP MR.

24
Traffic at a Substrate Router

• Substrate
• IP
• Substrate Tunnels
• Plain IP Control Protocols
• Plain IP Data
• Contains no Substrate Header
• Line Card forms Meta Frame and sends it to
  default IP MR
• A unique MI for the default IP MR is associated
  with each Physical Interface.
• Via the default IP MR, plain IP traffic may be
  destined for
• Other MRs on this Substrate Router via the
  Loopback
• Other MRs on other Substrate Routers via Meta
  Links carried on Substrate Links
• The Default IP MRs on peer Substrate Routers
  could exchange route information so they know how
  to get to other Substrate Routers.
• Legacy (Pure IP) Planet Lab Nodes on this
  Substrate Router.
• Substrate Control Entities on XScale?
• Egress as Plain IP?
• Policy based option
• Other?
• Other
• ARP

25
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

26
IP Address Assignment
Substrate Router(A.B.C.0 A.B.C.255)
PlanetLab Blades
X.Y.Z.1
1
U.V.W.1
2

• Consider a Substrate Router (SR) with the
  following characteristics
• 3 physical interfaces that connect to three
  Internet Routers
• Interface 1 ?? X.Y.Z.1
• Interface 2 ?? U.V.W.1
• Interface 3 ?? R.S.T.1
• Blade server that contains 6 Legacy (Plain IP)
  PlanetLab Nodes
• A Default IPv4 MR
• Block of IP Addresses A.B.C.0 A.B.C.255

R.S.T.1
3
27
IP Address Assignment
Substrate Router(A.B.C.0 A.B.C.255)
PlanetLab Blades
X.Y.Z.1
A.B.C.1
A.B.C.128
X.Y.Z.2
1
U.V.W.1
A.B.C.129
U.V.W.2
2

• Assign addresses to the SR as follows
• Each Physical Interface has a Substrate IP Tunnel
  Address
• Interface 1 ?? A.B.C.128 (A.B.C.0x80)
• Interface 2 ?? A.B.C.129 (A.B.C.0x81)
• Interface 3 ?? A.B.C.130 (A.B.C.0x82)
• Default IPv4 MR has 4 IP Addresses
• To X.Y.Z.1 out Interface 1 X.Y.Z.2
• To U.V.W.1 out Interface 2 U.V.W.2
• To R.S.T.1 out Interface 3 R.S.T.2
• To PlanetLab Nodes A.B.C.1

R.S.T.1
A.B.C.130
R.S.T.2
3
28
IP Address Assignment
Substrate Router(A.B.C.0 A.B.C.255)
PlanetLab Blades
X.Y.Z.1
A.B.C.1
A.B.C.128
X.Y.Z.2
1
U.V.W.1
A.B.C.129
U.V.W.2
2

• Route announcements
• Out Interface 1
• A.B.C.128/32
• A.B.C.0/25
• Out Interface 2
• A.B.C.129/32
• A.B.C.0/25
• Out Interface 3
• A.B.C.130/32
• A.B.C.0.25

R.S.T.1
A.B.C.130
R.S.T.2
3
29
BGP Notes

• RFC 4271 BGP-4
• BGP is defined as an inter-Autonomous System (AS)
  routing protocol.
• Autonomous System A set of routers under a
  single technical administration
• BGP Speaker A router that implements BGP
• BGP Identifier an IP Address assigned to a BGP
  Speaker. Same for every local interface and BGP
  peer.
• A router may also have separate IP Addresses for
  its interfaces.
• UPDATE message used to exchange route information
• NEXT_HOP attribute defines IP address of the
  router that SHOULD be used as the next hop to the
  destinations listed in the UPDATE message.
• Would this potentially be used in ARP requests?
• BGP uses TCP between peers.
• From RFC 4271, Section 3
• In the context of this document, we assume that a
  BGP speaker advertises to its peers only those
  routes that it uses itself (in this context, a
  BGP speaker is said to use a BGP route if it is
  the most preferred BGP route and is used in
  forwarding).

30
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• MetaLink Loopback Block
• Switch
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

31
Meta Link Loopback

• We would like to be able to route Plain IP pkts
  to an MR
• We can achieve this by using a default IPv4 MR
• But we dont want the IPv4 MR to have to know
  anything about other MRs and their
  Meta-Interfaces
• That is a Substrate function
• So, we need a Substrate function to translate
  from
• MRxMIi to MRyMIj
• We would also like to be able to route Plain IP
  pkts to Legacy PlanetLab Nodes residing on Blades
  in a Substrate Router
• Legacy PlanetLab Nodes have no Substrate.
• Again we can use the default IPv4 MR to achieve
  this.
• But the packets cannot have a Substrate Header or
  Meta Header when they arrive at the PlanetLab
  Nodes.
• So, we need a Substrate function to strip these
  headers.
• We would also like to be able to do similar
  things with IP Pkts in Substrate Frames on
  MetaLinks which terminate at the default IPv4 MR
• We will implement a Meta Link Loopback block
• Provides Internal Meta Link Loopbacks at the
  Substrate level.
• Probably will reside on one or more Line Cards.

32
Meta Link Loopback
Physical interface n on LC A
Substrate Router
Def IPv4 MR
LC A
n
m
Physical interface m on LC A
MR Y
n
LC B
Physical interface n on LC B
MR X
Loopback
PLab
Switch
33
Meta Link Loopback Plain IP to/from MR Y
Substrate Router
Def IPv4 MR
IPAm
VLAN a used.
IP
IPBn
a
IP
Ab MI b associated with MR A used
LC A
n
IPYk
IP
m
IPAm
Plain IP
IP
MR Y
Yk
n
IPBn
Y
Plain IP
LC B
IP
MR X
Loopback
IPYk
IP
Yk
Y
PLab
Switch
34
Meta Link Loopback Plain IP to/from PLab
Substrate Router
Def IPv4 MR
IPAm
VLAN a used.
IP
IPBn
a
IP
IPPi
IP
Ab MI b associated with MR A used
LC A
n
m
Plain IP
IP
IPAm
MR Y
n
IPBn
Plain IP
LC B
IP
MR X
IPPi
Loopback
PLab
Plain IP
P
Plain IP
P
Switch
35
Meta Link Loopback General
IPAn
Substrate Router
Def IPv4 MR
IP
IPAm
VLAN a used.
IP
IPBn
a
IP
IPPi
IP
IPXj
Ab MI b associated with MR A used
LC A
n
Substrate
IPAn
IP
IPYk
IP
IP
m
Plain IP
IP
IPAm
MR Y
Yk
n
IPBn
Y
Plain IP
LC B
IP
MR X
Xj
X
IPPi
IP
Loopback
IPXj
IPYk
Yk
Y
Xj
X
PLab
Plain IP
P
Plain IP
P
Switch
36
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• MetaLink Loopback Block
• Switch
• NP Blades
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

37
Switch

• Switch Blade Specs
• Promentum ATCA-2210
• http//www.radisys.com/products/ds-page.cfm?produc
  tdatasheetsid1191
• 20-port 10GE fabric switch
• 14 10GE links to user slots
• 4 10GE links for external connections (up/cross
  links) on front panel
• 24-port 1GE Base switch
• 14 1GE links to users lots
• 1GE link to redundant switch blade
• 1 10GE and 4 1GE links for external connections
  (up/cross links) on front panel
• Wire-speed L2 and L3 switching
• 4K IEEE 802.1Q VLANs
• Etc
• Traversing the Switch
• Switching is based on Ethernet Destination
  Address
• Isolation is based on VLAN.
• One VLAN will be assigned to each MetaNet present
  on a Substrate Router.
• All switch traffic for a MetaNet will be required
  to use its assigned VLAN.
• Frames from a MetaNet will only be transmitted to
  a port which is allowed to receive the specified
  VLAN.

38
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• MetaLink Loopback Block
• Switch
• NP Blades
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

39
NP Blades Radisys 7010
40
NP Blades Radisys 7010
10Gb Per Port Ethernet Switch Blade
ATCA Backplane
16 /
FIC
NPUA
SPI-4 Switch
/ 16
1 /
10G MAC
10Gb Per Port Ethernet Switch Blade
/ 1
16 /
NPUB
/ 16

• SPI-4 Switch supports 6 Interfaces
• Each Interface can support up to 16 SPI-4 Ports
• Each Port on an Interface can be switched to any
  other Port on any other Interface
• Radisys 7010 uses 4 of the 6 Interfaces
• 10G MAC Chip (IXF18105) supports and uses only 1
  SPI-4 Port on its SPI-4 Interface
• Therefore, all traffic from 10G MAC Chip must go
  to one NPU, either NPUA or NPUB.
• This should be fine for our LC in the future
• One NPU will be for Ingress and one for Egress
• For our NPE Blades, this will be a bit of pain.
• We can direct all traffic to NPUA and then have a
  block right after RX that splits the traffic for
  NPUA and NPUB
• Traffic for NPUA continues on to the processing
  blocks on NPUA
• Traffic for NPUB would go directly to a TX block
  to be transmitted to NPUB via the SPI Switch.
  (Does not go to 10G MAC chip)
• Still need to work out whether this will actually
  work or not.

41
NP Blades Radisys 7010

• Radisys 7010
• Two IXP2850 NPs
• 1.4 GHz Core
• 700 MHz Xscale
• Each with
• 3x256MB RDRAM, 533 MHz
• 3 Channels of QDR II SRAM, 8MB each, 200 MHz
• 16KB of Scratch Memory
• 16 Microengines
• 8KB instruction store
• 640 32-bit words of local memory
• Network Search Engine on 4th QDR channel
• Shared between NPs
• IDT75K72234
• 18Mb TCAM
• SPI-4 Switch for interconnections

42
Line Cards Radisys 7010 with RTM

• Radisys 7010 with RTM
• Same board as NP blade.
• Two IXP2850 NPs
• One used for Ingress
• One used for Egress
• Rear Transition Module (RTM)
• Connects via ATCA Zone 3
• 10 1GE Physical Interfaces
• Supports Fiber or Copper interfaces.

43
Radisys 7010 Blade
44
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

45
Substrate Links

• 2 Types of Substrate Links
• Point-to-Point (P2P)
• Connects two peer Substrate Routers
• May be realized via
• Directly Connected (P2P-DC)
• One physical link connects an Interface on one
  Substrate Router to an Interface on another
  Substrate Router
• Can use VLANs or not.
• Tunnel (P2P-Tunnel)
• Two Substrate Routers have a Tunnel defined that
  connects an interface on each router.
• Such a tunnel comprises exactly one Substrate
  Link.
• Can use VLANs or not.
• Multi-Access Network (P2P-VLAN0)
• Substrate Routers with interfaces connected to a
  Multi-Access Network all use a predefined VLAN
  (VLAN0).
• VLAN0 is defined for each Multi-Access Network
  and my be different on each one.
• Each of these Substrate Routers is pre-configured
  with the Ethernet addresses of the other
  Substrate Routers connected to the Multi-Access
  Network.
• By using the Ethernet address of peers, pairs of
  Substrate Routers have defined Substrate Links.
• With N Substrate Routers, there are at most
  (N(N-1))/2 Substrate Links
• Multi-Access

46
Substrate Links

• 2 Types of Substrate Links
• Point-to-Point (P2P)
• Multi-Access
• Supports Unicast, Multicast and Broadcast traffic
  from Meta Routers.
• Interfaces on Substrate Routers connected to a
  Multi-Access network are NOT necessarily
  pre-configured with each others Ethernet
  addresses.
• ARP may be used over the Multi-Access network to
  do the translation from MetaNet Address to
  Ethernet Address.
• ARP protocol and tables are managed by XScale.
• Can use VLANs or not, but cannot use VLAN0.

47
Substrate Links

• Substrate Link Restrictions
• P2P-DC does not co-exist on a physical interface
  with any of the other types of SL.
• P2P-Tunnel, P2P-VLAN0 and Multi-Access may all
  co-exist on the same physical interface.
• A physical interface may have
• At most one P2P-DC SL
• XOR
• One or more P2P-Tunnel SLs
• One or more P2P-VLAN0 SLs
• At most one Multi-Access SLs

48
IP Tunnel Substrate Links

• Each IP Tunnel Substrate Link is terminated at a
  single Physical Interface on a Substrate Router.
• The only route TO this end of the tunnel is from
  the outside into the associated physical
  interface.
• Nothing inside the Substrate Router routes TO
  this end of the Tunnel
• From inside the Substrate Router packets are just
  sent INTO the tunnel so they come out the other
  end.
• Included in the configuration of an IP Tunnel
  Substrate Link is the Address (IP or MAC?) of the
  next/first hop for the route to the other end of
  the tunnel.
• What happens if the routes in the Internet change
  and this next/first hop is no longer the way to
  go?
• Physical Interface will receive ICMP No Route to
  Host or ICMP Redirect messages.
• ICMP messages should be sent to the XScale on the
  Line Card.
• XScale should then nullify (mark as down, ) the
  IP Tunnel Substrate Link and all
  MetaLinks/MetaInterfaces associated with that
  Substrate Link.

49
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• Focus is on wired Ethernet
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

50
Point-to-Point Directly Connected (P2P-DC)
LC
LC
MR
MR
MetaLinks
MR
MR

• P2P should not need the TypeSubstrate since it
  is a point-to-point physical link and should only
  be carrying Substrate traffic.
• Configuration/Policy/Implementation option?

51
Point-to-Point via Tunnel (P2P-Tunnel)
Tunnel Substrate Link
MR
MR
LC
LC
MR
MR
MetaLinks
P2P-Tunnel SL Packet With VLANs
P2P-Tunnel SL Packet Without VLANs

• This example shows a Substrate Link in an IP
  tunnel over Ethernet.
• We could also envision other tunnel types
• MPLS
• ATM
• Sonet?
• Etc
• Ether/MAC layer encap ? Tunnel Protocol
• Tunnel layer encap ? Substrate

52
MPLS Tunnels

• Weve looked at MPLS just enough to convince
  ourselves that we can do something, but not
  enough to understand all the implications.
• Some of the MPLS RFCs
• 3031 Multiprotocol Label Switching Architecture
• 3032 MPLS Label Stack Encoding
• 3037 LDP Applicability
• LDP Label Distribution Protocol
• 4221 Multiprotocol Label Switching (MPLS)
  Management Overview
• MPLS uses a 32-bit field which contains
• Label (20b)
• Experimental (3b)
• S-Bit (1b) 1 indicates bottom of label stack
• TTL (8b)

53
Point-to-Point via Multi-Access Network
(P2P-VLAN0)

• For this example, on each LC pictured, one
  interface is being used.

Ethernet Switch
LC
MR-1
MR-2
LC
MR-2
LC
MR-1
MR-4
MR-2
M3
LC
MR-4
MR-2
MR-3
54
P2P-VLAN0 Packet Format
P2P-VLAN0 SL Packet (VLAN0)

• VLANVLAN0 provides isolation from other traffic
  on the Multi-Access network.

55
Multi-Access

• One MLI used per MetaNet on a Multi-Access
  Substrate Link.
• We may want to use the MnID for the MLI for each.

Ethernet Switch
LC
MR-1
MR-3
LC
MR-3
LC
MR-1
MR-2
MR-2
LC
MR-1
MR-2
56
Multi-Access Packet Formats
Multi-Access SL Packet with VLANs (VLAN?0)
Multi-Access SL Packet without VLANs
57
ARP for IP Packet Format

• Ethernet Header
• Dst Addr
• Src Addr
• Type 0x0806 (ARP)
• Hard Type Type of hardware address
• 1 ? Ethernet
• Proto Type Type of protocol address being mapped
• E.g. for IP 0x0800 (same as EtherType for IP)
• Hard/Proto Size size in bytes of address fields
• OP Operation
• 1 ARP Request
• 2 ARP Reply
• 3 RARP Request
• 4 RARP Reply
• Sender Eth Address
• Sender IP Address
• Target Eth Address
• Target IP Address

58
ARP for MetaNet Multi-Access Packet Format

• Ethernet Header
• Dst Addr
• Src Addr
• Type 0x0806 (ARP)
• Hard Type Type of hardware address
• 1 ? Ethernet
• Proto Type Type of protocol address being mapped
• Hard/Proto Size size in bytes of address fields
• OP Operation
• 1 ARP Request
• 2 ARP Reply
• 3 RARP Request
• 4 RARP Reply
• Sender Eth Address
• Sender MetaNet Address (MNidMnAddr)
• Target Eth Address
• Target MetaNet Address (MNidMnAddr)

59
Other Access Technologies

• ATM?
• Sonet?
• Wireless
• How would we characterize a wireless access
  network?
• How would we provide substrate links on a
  wireless network?
• Multi-Access network without any mechanism for
  isolation and differentiation of P2P and
  Multi-Access Links
• Two possibilities
• We do not support P2P on such a network
• We use one bit in the MLI (or one bit somewhere
  in some header) to indicate P2P vs. Multi-Access.

60
Summary Ethernet Pkt Formats with VLANs
Ethernet Hdr
Tunnel Hdr (IPv4)
Substr. Hdr
Meta
Ethernet Trailer
61
Summary Ethernet Pkt Formats without VLANs
P2P-Tunnel SL Pkt Without VLANs
P2P-VLAN0 SL Pkt
Multi-Access SL Pkt without VLANs
P2P-DC SL Pkt Without VLANs
DstAddr (6B)
DstAddr (6B)
DstAddr (6B)
Ethernet Hdr
SrcAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
TypeIP (2B)
TypeSubstrate (2B)
TypeSubstrate (2B)
MLI (2B)
MLI (2B)
LEN (2B)
LEN (2B)
Tunnel Hdr (IPv4)
Meta Frame
Meta Frame
MLI (2B)
Substr. Hdr
LEN (2B)
Meta Frame
Meta
PAD (nB)
PAD (nB)
PAD (nB)
Ethernet Trailer
CRC (4B)
CRC (4B)
CRC (4B)
62
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation (Next set of
  slides)
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

63
Extra

• The next set of slides are for templates or extra
  information if needed

64
Text Slide Template
65
Image Slide Template
66
Multicast Alternatives

• Provide support in CRF for Multicast
• Allow MRs to provide code to interpret Lookup
  Result.
• Define a default Unicast format for a Lookup
  Result
• A pure Unicast MR could use this exclusively and
  would not need to provide any code to interpret
  Lookup Result.
• Type bit indicates Default format
• CRF Specific Data
• QID
• Stats Index
• MR Related Data Used by CRF
• MI
• MR Specific Data Opaque to CRF
• Whatever the MR developer wants
• Define an MR-specific format
• Type bit indicates MR-Specific
• MR supplies a block of code to interpret result
  (next slide)

67
Core Components

• Functionality
• Initialization
• Table management
• Route/Filter tables
• L2 Tables
• MR State Table (in Demux)
• MetaRouter Installation/Removal

68
Core Components (sample App)
69
Core Components
MetaRouter Management Core
MR Specific Core Components ???
MR Code Management
GP PE?
MR Lookup Tbl Mgmt
MR State Tbl Mgmt
MR L2 Tbl Mgmt
RX Core
DeMux Core
Parse Core
Lookup Core
Hdr Fmt Core
QM Core
TX Core
Portability Framework Core Components
Infrastructure
Portability Framework Resource Manager
XScale
Micro- Engines
DeMux
Parse
Lookup
HeaderFormat
Rx
Tx
70
System Configuration
NumGP
NumLC
NumPE
Blade Server
e.g. 10 GE Interfaces Per LC
Switch Blades (2)

• NumLC NumPE NumGP 2 (switch) lt 14
• 14 of slots in a full ATCA Chassis
• Good ratio of PEs to LCs is probably 31
• Blade Server may take the place of the GP Blades
• Connected through Uplinks on Switch Blade(s)

71
MetaLinks

• Types of MetaLinks
• May be directional and asymmetric.
• Normal
• Frames enter with a Substrate Header(MLILEN) and
  Meta Frame
• Frames are sent to a MetaRouter on a Processing
  Engine
• Frames leave with a Substrate Header(MLILEN) and
  Meta Frame
• Gateway
• Frames either
• Enter an interface as Legacy and get delivered to
  an MR as Meta Frames
• OR
• Leave an MR as Meta Frames and get sent out an
  interface as Legacy
• Pass-Through
• Frames enter with a Substrate Header and Meta
  Frame and leave with a Substrate Header and Meta
  Frame without going to a MetaRouter.
• i.e. they go LC to LC without visiting a
  Processing Engine.
• No need for Legacy Pass-Through

72
MPLS Tunnels

• Weve looked at MPLS just enough to convince
  ourselves that we can do something, but not
  enough to understand all the implications.
• Some of the MPLS RFCs
• 3031 Multiprotocol Label Switching Architecture
• 3032 MPLS Label Stack Encoding
• 3037 LDP Applicability
• LDP Label Distribution Protocol
• 4221 Multiprotocol Label Switching (MPLS)
  Management Overview
• MPLS uses a 32-bit field which contains
• Label (20b)
• Experimental (3b)
• S-Bit (1b) 1 indicates bottom of label stack
• TTL (8b)
• There are some special labels defined, for
  instance
• IPv4 NULL Label
• when at bottom of stack indicates that the
  encapsulated packet is IPv4 and should be routed
  as such.
• How would we use it?
• We could define a similar special label for
  Substrate.
• Our Substrate Routers would need to participate
  in the MPLS label distribution protocols and
  such.
• The sending Substrate Router would push the
  special Substrate label onto the bottom of the
  stack and then push the first routing label onto
  the stack.

73
OLD

• The rest of these are old slides that should be
  deleted at some point.