Design of a Diversified Router: Line Card

1
Design of aDiversified Router Line Card
Jon Turner, John DeHart, Fred Kuhnsjon.turner_at_wu
stl.edu, jdd_at_arl.wustl.edu, fredk_at_arl.wustl.edu ht
tp//www.arl.wustl.edu/arl
2
Outline

• What is NOT covered in these slides
• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

3
Not Covered

• Control
• Installation
• Configuration
• Initialization
• Monitoring
• End Host
• Substrate/Meta Model
• Details about how PlanetLab works.
• These are very important topics but for now we
  will talk about the Data Path in the Network.

4
Outline

• JSTs original slides
• Schedule
• Model
• Traffic Types
• IP Addressing
• Components
• Switch
• MetaLink Loopback Block
• LC
• Substrate Link Types
• Packet Formats
• LC Rx/Tx Design Implementation
• Common Router Framework (CRF)
• Functional Blocks for implementing a Router

5
Design Implementation

• We will now look at how our model might be
  implemented.
• Again, the primary focus will be on wired
  Ethernet as the access technology.
• We will look at RX and TX separately.
• First we will focus on the Substrate Router
  functionality in the LC.
• As we do we will also show the packet formats as
  they traverse other parts of the Substrate
  Router.
• Some of this will apply only to the Shared NP
  case.
• For the sake of simpler diagrams we will reduce
  the IXP PE functional blocks as shown below
• Then we will look at the implementation of a
  common router framework in the IXP PE.

6
Design Implementation
Packet arriving On Port N
Packet leaving On Port M
LC
Txsubstrate
MR
Rxsubstrate
Switch
Switch

IXP PE (Shared)
RX
TX
7
Substrate Link Configuration

• RX Rules for determining type of Frame/Substrate
  Link
• P2P-DC is pre-configured on an interface by
  interface basis.
• ((EtherType Substrate) AND (VLANVLAN0)) ?
  P2P-VLAN0 SL
• VLAN0 is a predefined VLAN Id for use by the
  Substrate Network to connect peer substrate
  routers on a Multi-Access network
• SL id Senders Ethernet Address
• Lookup MLI in SL Specific Table ? MRMI
• ((EtherType Substrate) (VLAN ? VLAN0)) OR
• ((EtherType ARP) and (ProtoType Substrate))
  ? Multi-Access
• MLI is unique across a Multi-Access Substrate
  Link
• MLI Lookup in Multi-Access-SL Table ? MRMI
• ((EtherType ARP) and (ProtoType ? Substrate))
  OR
• ((EtherType ? Substrate) AND
• (EtherType ? ARP)) ? Tunnel or Legacy
• Substrate Link in a Tunnel (e.g. IPv4)
• ((EtherType IP) AND (IP_Proto Substrate)) ?
  Substrate Tunnel in IPv4
• Legacy (e.g. IPv4, ARP)
• ((EtherType IP) AND (IP_Proto TCP)) ? Legacy
  IPv4
• ((EtherType ARP) AND (ProtoType ? Substrate)) ?
  Legacy ARP
• Etc.

8
Substrate Link Configuration

• TX
• Substrate Link uniquely identified by MRMI tuple
• MRMI ? SL
• MetaLink uniquely identified by MRMI tuple
• MRMI ? MLI
• Hence
• MRMI ? SLMLI
• Different Header formats may be defined per SL
• i.e. IPv4 Tunnel header vs. P2P-VLAN0 header
• Located with lookup based on MLI
• P2P-VLAN0
• One or more MLI per MetaNet on the Multi-Access
  network
• Multi-Access
• One MLI per MetaNet on the Multi-Access network
• Meta-Destination Address, to get Ethernet Address
  use ARP
• Legacy
• Substrate Link in a Tunnel
• IPv4
• Gateway

9
Tables and Lookups

• The following slides have some tables and
  lookups.
• They are not meant as the only way to do things,
  just one way.
• Some if not all of the tables and lookups will
  almost certainly be implemented using the TCAM.
• For different types of frames we might need
  different types of headers.
• For this a lookup result will probably indicate
  what format/template to use and some/all of the
  info to use.
• The Point
• These slides are meant to show that we have the
  right information at the right place to do the
  needed lookup.
• There is still some design work for the actual
  implementer(s) to do to get it all right and make
  it efficient.

10
LC Packet TX All SL Types

• MnFlags define what type of Next Hop Address
  (NhAddr) is being given
• or perhaps what type is needed (maybe Substrate
  provides it for broadcast)
• Type (2b)
• NhMnAddr (01b)
• We may need to use ARP to translate to
  MAC/Ethernet Address
• MAC Address(10b)
• Could be used for Broadcast/Multicast
• NULL (00b) No Next Hop Address given or needed.
• Size (6b) Length of NhAddr field in Bytes

MR
Txsubstrate
Switch

MI (2B)
Blade-to-Blade Ethernet Header (MR specific VLAN)
MnFlags(1B)
NhAddr (nB)
LEN (2B)
Meta Frame
TxMI (2B)
MnFlags (1B)
NhAddr (nB)
LEN (2B)
Meta Frame
PAD (nB)
CRC (4B)
11
LC Packet TX All SL Types
MR
Txsubstrate
Switch

MI (2B)
MnFlags(1B)
NhAddr (nB)
LEN (2B)
Meta Frame
If SL_TYPEMulti-Access Lookup(MnID, MnAddr)
VLANTxMI ? SLMLI
12
LC Packet TX P2P-DC

• P2P-DC does not use ARP Cache.
• DAddr is configured in the Hdr field of Per SL ML
  Tables

• P2P-DC should not need to give NhMnAddr
• MnFlags would indicate NULL

Recalculate
13
LC Packet TX P2P-Tunnel (IPv4)

• P2P-Tunnel does not use ARP Cache.
• DAddr is configured in the Hdr field of Per SL ML
  Tables
• This may be an over-simplification.
• The packet needs to be routed toward the other
  end of the Tunnel

• This seems to assume that we have a static first
  hop for the IP Tunnel.
• Is that reasonable?
• Probably, Yes.

Recalculate
14
LC Packet TX P2P-VLAN0

• Do we want the P2P-VLAN0 to use the ARP Cache?
• Or is the DAddr configured in the Hdr fields of
  Per SL ML Tables?

Recalculate
15
LC Packet TX Multi-Access

• Multi-Access does use ARP Cache.
• If DAddr entry for MnAddr is missing, send packet
  up to XScale for it to perform ARP Request.

Recalculate
16
LC Packet RX All SL Types
P2P-VLAN0
Multi-Access
P2P-DC
P2P-Tunnel
17
LC Packet RX P2P-DC
Per SL ML Tables

• There is one P2P-DC SL Table per Port (Physical
  Interface).
• Blade info in tables provides info so we can fill
  in templates for appropriate header(s) to get
  frame to necessary blade.
• QID is used on LC

VLANMR
RxMI
Blade
QID
Packet arriving On Port N
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
VLANMR
(Port N P2P-DC Table)
RxMI
Blade
QID
VLANMR

• Do we want to include Senders MnAddr in Frame
  going to MR?
• No.
• But we may need to give it the MAC address.

Recalculate
Packet arriving On Port N
LC
MR
Rxsubstrate
Switch

18
LC Packet RX P2P-Tunnel
Packet arriving On Port N
Per SL ML Tables

• There is one P2P-Tunnel SL Table per tunnel SL.

VLANMR
RxMI
Blade
QID
RxMI
Blade
QID
VLANMR
Fct(PortN, IPv4, IP Src_Addr)
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
VLANMR
(Port N IPv4 Tunnel(i) Table)
RxMI
Blade
QID
VLANMR

• Do we want to include Senders MnAddr in Frame
  going to MR?
• Probably not
• IP Routers dont

Recalculate
Packet arriving On Port N
LC
MR
Rxsubstrate
Switch

19
LC Packet RX P2P-VLAN0
Per SL ML Tables
Per SL ML Tables

• There is one P2P-VLAN0 SL Table per SL.
• Located based on SrcAddr of sender

VLANMR
RxMI
Blade
QID
RxMI
Blade
QID
VLANMR
Packet arriving On Port N
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
VLANMR
(Port N P2P-VLAN0 Table)
RxMI
Blade
QID
VLANMR

• Do we want to include Senders MnAddr in Frame
  going to MR?

Recalculate
Packet arriving On Port N
LC
MR
Rxsubstrate
Switch

20
LC Packet RX Multi-Access
Per SL ML Tables

• There is one Multi-Access SL Table per port.
• Contains the MLI entries for the Multi-Access SL

VLANMR
RxMI
Blade
QID
Packet arriving On Port N
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
VLANMR
(Port N Multi-Access Table)
RxMI
Blade
QID
VLANMR

• Do we want to include Senders MnAddr in Frame
  going to MR?

Recalculate
Packet arriving On Port N
LC
MR
Rxsubstrate
Switch

21
Pure Legacy Traffic
IXP PE Blade
IXP PE Blade
GP Blade
GP Blade
. . .
IPv4 MR
Switch Blade

• LC
• If ((EtherTypeIPv4) AND (IP Proto ? Substrate))
• then non-Tunnel Legacy
• Send frame to pre-configured default IPv4 MR

22
LC Packet RX Legacy IPv4
Packet arriving On Port N
Per SL ML Tables
Fct(PortN)

• There is one Legacy MR Table per Port.
• Entries point to Legacy MR for the specified
  Protocol.
• There is at most one Legacy MR for each legacy
  protocol

VLANMR
RxMI
Blade
QID
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
VLANMR
Fct(IPv4)
RxMI
Blade
QID
VLANMR
(Port N Legacy MR Table)
RxMI
Blade
QID
VLANMR
Recalculate
Recalculate
Packet arriving On Port N
LC
MR
Rxsubstrate
Switch

23
LC Packet TX All SL Types
MR
Txsubstrate
Switch

MI (2B)
MnFlags(1B)
NhAddr (nB)
LEN (2B)
Meta Frame
If SL_TYPEMulti-Access Lookup(MnID, MnAddr)
VLANTxMI ? SLMLI
24
LC Packet TX Legacy IPv4

• Legacy IPv4 does use ARP Cache.
• If DAddr entry for MnAddr is missing, send packet
  up to XScale for it to perform ARP Request.

Recalculate
25
Pure Legacy Traffic
IXP PE Blade
IXP PE Blade
GP Blade
GP Blade
. . .
IPv4 MR
Switch Blade

26
Substrate/MetaNet Model

• To the Substrate, some Meta Links Pass Through
• Pass through a Substrate Router without visiting
  a Meta Router

MetaLink 1
Substrate Link
MetaLink 2
MetaLink 3
MR_A
MR_A
MR_A
MI
MI
MI
MI
MI
MI
MR_B
MR_B
Pass Through Meta Link
MI
MI
MI
MI
MR_C
MR_C
MR_C
MI
MI
MI
MI
MI
MI
Substrate Router X
Substrate Router Z
Substrate Router Y
27
Pass-Through MetaLink

• For Pass-Through ML, we need to include MnFlags
  and NhMnAddr so Tx can handle Multi-Access case
• Allocate special set of MR/MI for use by
  Substrate when handling Pass-Through MetaLinks

Packet arriving On Port N
Packet arriving At LC Y
Packet arriving On Port N Of LC X
Ethernet Switch
LC Y
LC X

28
Rx Pass-Through MetaLink
Packet arriving On Port N
Recalculate
Packet arriving On Port N Of LC X
Ethernet Switch
LC Y
LC X

29
Tx Pass-Through MetaLink
Ethernet Switch
LC Y
LC X

If SL_TYPEMulti-Access Lookup(MnID, MnAddr)
VLANTxMI ? SLMLI

• This is exactly like the common case shown for
  all SL Types.
• After this things proceed as previously shown for
  the type of SL that is being used for the
  Transmit.

30
LC Functional Blocks
Lookup (1 ME)
Switch Tx (3 ME)
QM/Schd (1 ME)
Hdr Format (1 ME)
S W I T C H
Phy Int Rx (2 ME)
Key Extract (1 ME)
Lookup (1 ME)
Phy Int Tx (3 ME)
QM/Schd (1 ME)
Hdr Format (1 ME)
Key Extract (1 ME)
Switch Rx (2 ME)
Rate Monitor (1 ME)

• ME counts are my first guesses for number needed.
• For each block Ill show
• Function
• What this block does.
• Memory Accesses
• SRAM
• DRAM
• Buffer Descriptor Accesses
• Which fields in Buffer Descriptor the block needs
  to read and/or write.
• Notes
• Additional thoughts about this block.
• Next
• Analyze the SRAM Accesses and try to map them on
  to the available SRAM Channels.
• Analyze the SRAM and DRAM accesses and calculate
  packet processing rates.

31
LC Functional Blocks

• Ingress (Physical Interface ? Switch)
• PhyInt Rx
• KeyExtractor
• Lookup
• Hdr Format
• QM/Scheduler
• Switch Tx
• Egress (Switch ? Physical Interface)
• Switch Rx
• KeyExtractor
• This is only extracting the VLAN and the MI.
  Combine with previous or following block?
• But it involves a DRAM Read so we probably want
  to leave it separate and use all 8 threads.
• RateMonitor
• Before or after the Lookup?
• Is this different than what QM/Scheduler
  will/could do?
• Lookup
• Hdr Format
• QM/Scheduler
• PhyInt Tx

32
LC Buffer Descriptor

• Hopefully we can use the same buffer descriptor
  for the LC and the CRF Processing Engine.
• There might be some fields that are used on one
  and not on the other but thats ok (MR_ID, TxMI,
  VLAN not needed on LC)
• PE Buffer Descriptor
• LW0 buffer_next 32 bits Next Buffer Pointer
  (in a chain of buffers)
• LW1 offset 16 bits Offset to start of data
  in bytes
• LW1 BufferSize 16 bits Length of data in the
  current buffer in bytes
• LW2 reserved 8 bits reserved/unused
• LW2 reserved 4 bits reserved/unused
• LW2 free_list 4 bits Freelist ID
• LW2 packet_size 16 bits (Total packet size
  across multiple buffers)
• LW3 MR_ID 16 bits Meta Router ID
• LW3 TxMI 16 bits Transmit Meta Interface
• LW4 VLAN 16 bits VLAN
• LW4 reserved 16 bits reserved/unused
• LW5 reserved 32 bits reserved/unused
• LW6 reserved 32 bits reserved/unused
• LW7 packet_next 32 bits pointer to next packet
  (unused in cell mode)
• Leave multi-buffer fields there as a template for
  the dedicated blade implementation of a
  jumbo-frame MR.

33
LC RX Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
RBUF
Port(8b)

• Rx
• Function
• Coordinate transfer of packets from RBUF to DRAM
• Memory Accesses
• SRAM
• Write Buffer Descriptor
• Free List?
• DRAM Transfer from RBUF
• Buffer Descriptor Accesses
• Write/Initialize Buffer_Next, Buffer_Size,
  Offset, Free_List, Packet_Size
• Notes
• Buffer Handle
• contains the SRAM address of the buffer
  descriptor.
• from the SRAM address of the descriptor we can
  calculate the DRAM address of the buffer data.
• Offset of where packet starts should be a
  constant.

34
LC RX Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
Buf Handle(32b)
Lookup Key(16B)
Port(8b)
Really only needs to be max of 10B (see next
slide for view of Keys)

• Key_Extract
• Function
• Extracts lookup key based on type of frame that
  was received.
• Memory Accesses
• DRAM
• Read as much of header as is necessary to extract
  key
• May vary depending on type of Substrate Link
• SRAM None
• Buffer Descriptor Accesses None
• Notes
• Calculates DRAM Address based on SRAM descriptor
  address in buffer handle and the Offset passed to
  it by RX.
• Frame offset in buffer is a constant and does not
  need to be read from Buffer Descriptor

SL Type (4b)
35
LC RX Functional Blocks

• Substrate Link Type
• Will be used as Database ID by Lookup Block
• Lookup Keys

SL Type (SL Type ID) (size)
Port(8b)
MLI(16b)
P2P-DC(0x0) (28 bits)
Port (8b)
MLI (16b)
IP SAddr (32b)
EtherType (16b) 0x0800
P2P-Tunnel - IPv4(0x1) (76 bits)
Port (8b)
MLI (16b)
Ethernet SAddr (48b)
P2P-VLAN0(0x2) (76 bits)
Port(8b)
MLI(16b)
MA(0x3) (28 bits)
Port (8b)
EtherType (16b) 0x0800
Legacy IPv4(0x4) (28 bits)
Physical Interface
36
LC RX Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
Buf Handle(32b)
Lookup Result (16B)
Lookup Key(16B)

• Lookup
• Function
• Performs Lookup and passes result on to Hdr
  Format.
• Memory Accesses
• DRAM None
• SRAM
• TCAM Lookup
• Write Lookup command
• Read Lookup Result (1-5 words from TCAM SRAM
  Controller or other SRAM Controller)
• Buffer Descriptor Accesses None
• Notes
• Lookup does no processing on the lookup result.
• Need to decide how lookup result will be stored
  and retrieved.
• See notes on TCAM for information about the
  issues involved.

SL Type (4b)
37
LC RX Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract

• Hdr Format
• Function
• From lookup result
• re-writes headers in DRAM to make frame ready to
  transmit.
• Extract QID to pass on to QM/Scheduler
• Memory Accesses
• DRAM
• SRAM
• Read Descriptor and Re-Write Descriptor
• OR
• Atomic Increment/Decrement some fields in
  Descriptor
• Buffer Descriptor Accesses
• Update Size and Offset fields
• Notes
• Pass Size on to QM/Scheduler so it does not have
  to read buffer descriptor for Enqueue to update Q
  Length.
• Buffer Offset should be a constant and should not
  need to be read from Buffer Descriptor

38
LC RX Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)

• QM/Scheduler (See Saileshs slides for more
  details)
• Function
• Enqueue and Dequeue from queues
• Scheduling algorithm
• Drop Policy
• Memory Accesses
• DRAM None
• SRAM
• Q-Array Reads and Writes
• Scheduling Data Structure Reads and Writes
• QLength Data Structure Reads and Writes
• Dequeue Read Buffer Descriptor to retrieve
  Packet Size
• Buffer Descriptor Accesses Read packet size
• Notes

39
LC RX Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
TBUF

• Switch TX
• Function
• Coordinate transfer of packets from DRAM to TBUF
• Memory Accesses
• SRAM Read Buffer Descriptor
• DRAM Transfer to TBUF
• Buffer Descriptor Accesses
• Read Size and Offset
• Notes
• Calculate DRAM address based on SRAM Descriptor
  address in buffer handle

40
LC TX Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor
RBUF
Buf Handle(32b)
Offset (16b)

• Rx
• Function
• Coordinate transfer of packets from RBUF to DRAM
• Memory Accesses
• SRAM Write Buffer Descriptor
• DRAM Transfer from RBUF
• Buffer Descriptor Accesses
• Write/Initialize Buffer_Next, Buffer_Size,
  Offset, Free_List, Packet_Size
• Notes
• Buffer Handle
• contains the SRAM address of the buffer
  descriptor.
• from the SRAM address of the descriptor we can
  calculate the DRAM address of the buffer data.
• Passing the offset of where the packet starts in
  memory will save the next block from having to
  read the buffer descriptor.
• Perhaps we should just pass the actual DRAM
  Buffer Pointer?

41
LC TX Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor
Buf Handle (32b)
Lookup Key

• Key_Extract
• Function
• Extracts lookup key based on type of frame that
  was received.
• Memory Accesses
• DRAM
• Read VLAN and TxMI from Frame
• SRAM None
• Buffer Descriptor Accesses None
• Notes
• Calculates DRAM Address based on SRAM descriptor
  address in buffer handle and the Offset passed to
  it by RX.

42
LC TX Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor

• Rate Monitor
• Function
• Ensures that MR/MIs behave according to their
  Rate Specs.
• Does this need to be a separate function from the
  QM/Scheduler?
• Memory Accesses Unknown at this point
• DRAM
• SRAM
• Buffer Descriptor Accesses Unknown at this point
• Notes

43
LC TX Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor

• Lookup
• Function
• Performs Lookup and passes result on to Hdr
  Format.
• Memory Accesses
• DRAM None
• SRAM
• TCAM Lookup
• Write Lookup command
• Read Lookup Result (1-5 words from TCAM SRAM
  Controller or other SRAM Controller)
• Buffer Descriptor Accesses None
• Notes
• Lookup does no processing on the lookup result.
• Need to decide how lookup result will be stored
  and retrieved.
• See notes on TCAM for information about the
  issues involved.

44
LC TX Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor

• Hdr Format
• Function
• From lookup result
• re-writes headers in DRAM to make frame ready to
  transmit.
• Extract QID to pass on to QM/Scheduler
• Memory Accesses
• DRAM
• SRAM
• Read Descriptor and Re-Write Descriptor
• OR
• Atomic Increment/Decrement some fields in
  Descriptor
• Buffer Descriptor Accesses
• Update Size and Offset fields
• Notes
• Pass Size on to QM/Scheduler so it does not have
  to read buffer descriptor for Enqueue to update Q
  Length.

45
LC TX Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor
Buf Handle(32b)

• QM/Scheduler (See Saileshs slides for more
  details)
• Function
• Enqueue and Dequeue from queues
• Scheduling algorithm
• Drop Policy
• Memory Accesses
• DRAM None
• SRAM
• Q-Array Reads and Writes
• Scheduling Data Structure Reads and Writes
• QLength Data Structure Reads and Writes
• Dequeue Read Buffer Descriptor to retrieve
  Packet Size
• Buffer Descriptor Accesses Read packet size
• Notes

46
LC TX Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor
TBUF
Buf Handle(32b)

• Switch TX
• Function
• Coordinate transfer of packets from DRAM to TBUF
• Memory Accesses
• SRAM Read Buffer Descriptor
• DRAM Transfer to TBUF
• Buffer Descriptor Accesses
• Read Size and Offset
• Notes
• Calculate DRAM address based on SRAM Descriptor
  address in buffer handle

47
ML Loopback Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Loopback Hdr Re-Format
Lookup
Phy Int Tx
QM/Schd
Hdr Format
Key Extract
Switch Rx
Rate Monitor

• Loopback path should be able to re-use some of
  the blocks implemented for the LC
• Loopback Hdr Re-Format
• Needs to be able to strip off the previous MRs
  Header.
• For Plain-IP ? IPv4 MR ? MR Y
• When frame arrives at Loopback (between IPv4 MR
  and MR Y) it will still have IP Header which
  should be stripped off before frame is sent to MR
  Y.
• Lookup Result should probably include a length
  field or a Buffer offset field that indicates
  where new Meta Frame should start.

48
LC Notes on TCAM Lookups

• See techX_Design_TCAM_Usage.ppt slides for notes
  on how the TCAM will be used by the Lookup Block

49
Extra

• The next set of slides are for templates or extra
  information if needed

50
Text Slide Template
51
Image Slide Template
52
LC SRAM Accesses
SRAM ChA
SRAM ChA
SRAM ChF
SRAM ChH
SRAM ChA
SRAM ChC
SRAM ChD
SRAM ChA
Lookup (1 ME)
Switch Tx (2 ME)
QM/Schd (1 ME)
Hdr Format (1 ME)
Phy Int Rx (2 ME)
Key Extract (1 ME)
S W I T C H

• SRAM Ch A SRAM Buffer Descriptors for Ingress
  (Phy Int ? Switch)
• SRAM Ch B SRAM Buffer Descriptors for Egress
  (Switch ? Phy Int)
• SRAM Ch C TCAM Access
• SRAM Ch D Lookup Associated Data for Ingress
  (Phy Int ? Switch)
• SRAM Ch E Lookup Associated Data for Egress
  (Switch ? Phy Int)
• SRAM Ch F Q-Array for Ingress (Phy Int ?
  Switch) QM
• SRAM Ch G Q-Array for Egress (Switch ? Phy Int)
  QM
• SRAM Ch H QM Scheduling/Dequeue Data Structure
• SRAM Ch I QM Scheduling/Dequeue Data Structure

Lookup (1 ME)
Phy Int Tx (1 ME)
QM/Schd (1 ME)
Hdr Format (1 ME)
Key Extract (1 ME)
Switch Rx (2 ME)
Rate Monitor (1 ME)
SRAM ChB
SRAM ChB
SRAM ChG
SRAM ChI
SRAM ChC
SRAM ChE
SRAM ChB
SRAM ChB
53
LC Number of Per Pkt SRAM Accesses

• Rx
• 1 SRAM Buffer Descriptor WRITE (A,B)
• Lookup
• TCAM
• 1 SRAM WRITE of TCAM Lookup Command (C, C)
• 1-8 SRAM READs of Results Mailbox (C, C)
• 1 READ if we are just retrieving an Index or
  Pointer
• Up to 8 locations to read if we are retrieving
  the actual result data from Results Mailbox
• Associated Data
• Read of up to 8 32-bit words in consectutive SRAM
  locations (D, E)
• Hdr Format
• 1 SRAM Buffer Descriptor WRITE (A,B)
• QM/Sched
• Q-Array
• Enqueue lt 1 SRAM Read Queue Descriptor (F,G)
• Dequeue lt 1 SRAM Read Queue Descriptor (F,G)
• SRAM Buffer Descriptors
• Enqueue
• 1 Buffer Descriptor Read (A,B)

54
LC SRAM Accesses

• Buffer Descriptor Accesses (A,B) 3 Writes, 3
  Reads
• Rx Write initial descriptor
• Hdr Format Write descriptor to update sizes,
  offsets,
• QM/Scheduler
• Enqueue Read Descriptor to get pkt size to do
  drop policy check
• Enqueue Write Descriptor to update next pkt
  pointer (how does Q-Array do this for us?)
• Dequeue Read Descriptor to get pkt size to do
  credit check?
• Tx Read Descriptor

55
LC Number of Per Pkt SRAM Accesses

• Channel A
• Ingress W, R, W, R, R
• Channel B
• Egress W, R, W, R, R
• Channel C
• Ingress W, 1-8 R,
• Egress W, 1-8 R,
• Channel D
• Ingress lt8 R
• Channel E
• Egress lt 8 R
• Channel F
• Ingress R, W, R, W
• Channel G
• Egress R, W, R, W
• Channel H
• Ingress W, R, W
• Channel I
• Egress W, R, W

56
LC Functional Blocks
Output Hlpr (1 ME)
QM/Schd (1 ME)
Input Hlpr (1 ME)
Lookup (1 ME)
Switch Tx (2 ME)
Hdr Format (1 ME)
S W I T C H
Phy Int Rx (2 ME)
Key Extract (1 ME)
. . .
QM/Schd (1 ME)
Scratch Rings
NN Ring
NN Ring
QM/ Sch Hlpr (1 ME)
QM/Schd (1 ME)
QM/ Sch Hlpr (1 ME)
Lookup (1 ME)
Phy Int Tx (1 ME)
Hdr Format (1 ME)
Key Extract (1 ME)
Switch Rx (2 ME)
Rate Monitor (1 ME)
. . .
QM/Schd (1 ME)

• Multiple QM/Scheds
• MetaLinks assigned to QM/Sched to balance load
  based on their allocated BW
• Input Helper ME demuxes from NN ring from Hdr
  Format to multiple Scratch Rings
• Output Helper ME muxes from multiple Scratch
  Rings to NN to TX

Scratch Rings
NN Ring
NN Ring
57
OLD

• The rest of these are old slides that should be
  deleted at some point.

58
LC Notes on TCAM Lookups

• The following slides show a way to use the TCAM
  for the lookups.
• Slight adjustments might be desirable depending
  on
• Ease of doing operations on non-byte length bit
  fields
• What we learn about methods for using the TCAM.
• Field and Identifier sizes
• MN id 32 bits
• MI id 16 bits (64K Meta Interfaces per Meta
  Router)
• MLI 16 bits (64K Meta Links per Substrate
  Link)
• Port 8 bits (256 Physical Interfaces per
  Line Card)
• QID 20 bits (1M Queues per Queue Manager)
• QM ID 4 bits (16 Queue Managers per LC or PE.)
• We probably can only support 4 QMs (2 bits)
• (64 Q-Array Entries) / (16 CAM entries) ? 4 QMs
  per SRAM Controller.

59
LC Notes on TCAM Lookups

• Lookup Key size options
• 32/36, 64/72, 128/144, 256/288, 512/576 (all in
  bits)
• Lookup Result options
• Absolute Index relative to beginning of TCAM
  array
• Database Relative Index relative to beginning of
  selected database
• Memory Pointer Index points to SRAM location of
  result data
• Associated Data 32, 64 or 128 bits of data
  associated with the lookup result.
• Associated Data is stored in ZBT SRAM attached to
  TCAM.
• TCAM Databases
• How many to use?
• 1 for TX and 1 for RX?
• 1 for TX and 1 for each of the SL Types on Rx (5
  types)?
• Other

60
LC TCAM Lookup Keys on RX

• P2P-DC(28b) SL_Type(4)/Port(8)/MLI(16)
• P2P-Tunnel(IPv4)(76b) SL_Type(4)/Port(8)/EtherTyp
  e(16)/IPSrc(32)/MLI(16)
• P2P-VLAN0(76b) SL_Type(4)/Port(8)/EthSAddr(48)/ML
  I(16)
• MA(28b) SL_Type(4)/Port(8)/MLI(16)
• Legacy(28b) SL_Type(4)/Port(8)/EType(16)
• Fields
• SL_Type (4b) Substrate Link Type
• 0000 DC
• 0001 IPv4 Tunnel
• 0010 VLAN0
• 0011 MA
• 0100 Legacy (non-substrate) without VLAN
• 0101 Legacy (non-substrate) with VLAN
• Port(8b) Physical Interface number
• MLI(16b) Meta Link Identifier
• Ethertype(16b) Ethernet Header Type field
• IPSrc(32b) IP Source Address
• EthSAddr(48b) Ethernet Header Source Address

61
LC TCAM Lookup Keys on RX
62
LC TCAM Lookup Results on RX

• Standard Fields (116b)
• Type (4b)
• 0000 Default, not Pass Through, ignore MnFlags
• 1000 Pass Through with no extra lookup needed
  for NhAddr(nB), MnFlags(8b) should be 0x00
• 1001 Pass Through with extra lookup needed for
  NhAddr(nB)
• VLAN (16b)
• MI (16b)
• Blade Eth Hdr (48b)
• Only needs to have the DAddr.
• SAddr can be configured and constant per LC
• First EtherType can be constant 802.1Q
• Second EtherType can be constant Substrate
• QID (24b)
• MnFlags(8b)
• Can we say there will be no Pass Through Meta
  Links where one side will be on a Multi Access
  and hence might need a NhAddr field?
• Pass Through Meta Link Fields
• MnFlags(8b)
• NhAddr(nB)
• Even a 32b IP NhAddr would not fit in a 128b TCAM
  Result.

63
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• The Lookup Result for TX will consist of several
  parts
• Lookup Result
• Constant fields
• Calculated fields
• Fields that can be stored in Local Memory
• Some of these are common across all SL Types
• Other fields are specific to each SL Type
• Common across all SL Types (100b)
• From Result (44b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (24b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port

64
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (100b)
• From Result (52b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (24b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• P2P-DC Hdr (64b)
• Constant (16b)
• EtherType (16b) Substrate
• Calculated (0b)
• From Result (48b)
• Eth DA (48b)

65
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (100b)
• From Result (52b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (24b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• P2P-MA Hdr (64b)
• Constant (16b)
• EtherType (16b) Substrate
• Calculated (0b)
• From Result (48b)
• Eth DA (48b)

66
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (100b)
• From Result (52b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (24b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• P2P-VLAN0 Hdr (80b)
• Constant (16b)
• EtherType1 (16b) 802.1Q
• EtherType2 (16b) Substrate
• Calculated (0b)
• From Result (64b)

67
LC TCAM Lookups on TX

• Result (continued)
• Common across all SL Types (100b)
• From Result (52b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (24b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• P2P-Tunnel Hdr for IPv4 Tunnel (224b)
• Constant (64b)
• Eth Hdr EtherType (16b) 0x0800
• IPHdr Version(4b)/HLen(4b)/Tos(8b) (16b) All can
  be constant?
• IP Hdr Flags(3b)/FragOff(13b) (16b) what is our
  stance on Fragments? If never used, these are
  constants, if it is possible we will have to use
  them, then this has to be calculated. Either way,
  shouldnt be in Result
• IP Hdr TTL (8b) Constant
• IP Hdr Proto (8b) Substrate
• Calculated (48b)
• IP Hdr Len(16b) needs to be calculated for each
  packet sent, so shouldnt be in Result.

68
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (100b)
• From Result (52b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• Ignored for Legacy Traffic
• QID (24b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• Legacy (IPv4) with VLAN Hdr (96b)
• Constant (16b)
• EtherType1 (16b) 802.1Q
• Calculated (0b)
• ARP Lookup on NhAddr (48b)

69
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (100b)
• From Result (52b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• Ignored for Legacy Traffic
• QID (24b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• Legacy (IPv4) without VLAN Hdr (64b)
• Constant (0b)
• Calculated (0b)
• ARP Lookup on NhAddr (Is ARP cache another
  database in TCAM?) (48b)
• Eth DA (48b)

70
SUMMARY LC TCAM Lookups

• Rx Key Size 128 bits
• Rx Result Size 128 bits
• Tx Key Size 32 bits
• Tx Result Size 256 bits (we should try to
  squeeze this in to 128 bits!)
• What if QID went from 24 to 22 bits.
• 2 bits for QM id
• 20 bits for QID 1M queues per QM instance, 4M
  Queues total.
• And Port went from 8 bits to 6 bits (64 physical
  interfaces per Line Card)
• If we cant use the 128 bit Result size for Tx,
  we might as well include the Local Memory result
  in the TCAM lookup.
• The Local Memory result was going to be the data
  that was only dependent on the Physical Interface
  id and was going to go into Local memory to try
  to keep the TCAM result below 128 bits.