IXP2400 Protocol Offloading

1
IXP2400 Protocol Offloading

• Yan Luo (yluo_at_cs.ucr.edu)
• Chris Baron (cbaron_at_cs.ucr.edu

2
NP Overview
3
TCP/UDP/IP
4
VIA-UDP
5
IXP OS bypass
User
Kernel
BigPhys
SendBuf
RcvBuf
PCI
IXP
DRAM
Downbuf/Upbuf
6
Protocol Overview

• Customized protocol
• Contains src/dest MAC, IP, and port
• Unique packet ID and length of payload
• Padding for ease of reading in blocks
• Add own protocol header in the future

7
Socket Table

• Socked id
• Src/dstIP
• Src/dstPort
• User send/recv buff start
• Phys send/recv buff start (for DMA)
• Send/recv buffer size
• Recv pointer (where data last written)

8
Theory of Operation (host)

• Setup
• Client application calls open() system call,
  kernel adds new entry to lookup table, returns
  unique socket descriptor
• Client calls setup_connection(), a macro to ioctl
  calls, to add src/dst IP and requested ports for
  connection
• Client calls ixpmalloc(), a macro to mmap()
  kernel memory into userspace, used for send and
  receive buffers

9
Theory of Operation (host)

• Send
• Macro to write() function for driver
• If no space left on IXP, return error, else
  DMA/PIO to IXP, returning number of bytes copied
• Currently uses PIO, for less latency
• Receive
• If no data available, return error
• User must loop/poll for data

10
Theory of Operation (IXP)
11
(1) Host Send

• User calls send data
• Kernel assembles the packet header based on the
  socket table
• Lookup the Downbuf tail/num in the bridgeCSR
• Checks if downbuf is full
• DMA/PIO data
• Updates Downbuf tail in bridgeCSR

12
(2) Downbuf Manager

• Polls bridgeCSR to get new tail of DOWNBUF
• Examine protocol header
• Fill in meta data (packet size, id etc)
• Dispatch to worker ME
• Examine the head of DOWNBUF, advance head if it
  is processed.

tail
head
Just arrive
Finished
In-progress
13
(3) Downbuf Worker

• Get a packet from Downbuf manager
• Extract IP from packet
• Lookup MAC and output port in IP forwarding table
• Fill in MAC into packet
• Fill in meta data (packet offset, len, output
  port)
• Put the packet in TX queue

14
(4) XMIT ME

• Get a packet from TX queue
• Split packet into mpackets
• Move mpackets to TBUF of MSF
• Enable sending mpackets

15
(5) RCV ME

• Waked up by MSF for new packets
• Assemble and move mpackets to DRAM
• Fill in meta data (packet offset, size, input
  port etc)
• Put packet in RX queue

16
(6) Upbuf Manager

• Get a packet from RX queue
• Get meta data
• Examine packet header (dstIP, destPort etc)
• Look up socket table for up_rcv_ptr
• Prepare DMA descriptor
• Enable DMA
• Update up_tail in bridge CSR

17
(7) Host RCV

• User requests receive data
• Kernel looks up the bridgeCSR from socket table
• Returns the number of bytes arrived or error for
  no data available

18
Datapath Timing
19
Datapath Timing
20
Future additions

• Use Interrupts
• Use bridge registers more efficiently
• Own protocol header

21
END