Using the WUGS-20 GigE Line Card

1
Using the WUGS-20GigE Line Card
John DeHartApplied Research LaboratoryComputer
Science and Engineering Departmenthttp//www.arl.
wustl.edu/arl/projects/techX
2
Introduction

• Really understanding how the WUGS-20 GigE line
  cards work requires understanding how ethernet,
  ARP, Routers and ethernet switches all work.
• This set of instructions cant possibly teach all
  of that. If you require all the details you are
  going to have to do a lot of work/study on your
  own.
• These instructions will hopefully get you to the
  point where you can get the GigE line cards
  working for some useful experimentation.
• Questions are always welcome
• Please keep in mind that we are continuing to
  support Kits users without support. The Kits
  program is officially over.
• The number of Kits users who have GigE cards is
  very small (2).
• If you have questions about the GigE cards, you
  should probably send them to John DeHart directly
  (jdd_at_arl.wustl.edu) and CC the other users
• Ron Srodawa (srodawa_at_oakland.edu)
• Min Song (msong_at_odu.edu)
• Our standard configuration has migrated to use
• FPXs and SPCs on each port of an NSP (formerly
  called MSR)
• Linux for all end hosts used in our router
  testbeds
• NetBSD used ONLY for building kernels and plugins
• We will still try to help you with other
  configurations but it is becoming increasingly
  difficult to replicate other configurations

3
Configuring a GigE Line Card

• SUNI Config
• There is a SUNI chip on the GigE card which needs
  some configuration.
• We wont go into the details, just dont change
  the SUNI configuration commands in the
  configuration script. Dont even change the
  order.
• Ethernet Config
• MAC Address
• Mask
• IP Address
• Next Hop IP Address

4
Configuring a GigE Line Card

• Utilities
• readWriteRegs low level utility for configuring
  the GigE card(s)
• Src Directory
• wu_arl/wugs/gige/src/utilities/ReadWriteRegisters
• Binary
• Linux/readWriteRegs
• configGigE.sh script for configuring the GigE
  cards
• uses readWriteRegs
• Src Directory
• wu_arl/wugs/gige/src/utilities/configScripts
• requires that WUARL environment variable be set
  to point to the root of your wu_arl tree.
• for me, I set gt setenv WUARL /d/jdd/wu_arl
• iperf IP traffic generator
• useful for experimentation
• version 1.6.1 exists in CVS tree
• wu_arl/utilities/IP/iperf
• website http//dast.nlanr.net/Projects/Iperf/
• tcpdump Useful for debugging
• run as root

5
configGigE.sh utility Usage Message

• Running configGigE.sh with no arguments will give
  the Usage message
• gt ./configGigE.sh
• ./configGigE.sh -vci ltVCIgt -mac ltMAC_H16gt
  ltMAC_L32gt -0,1,2 ltMYIPgt ltMASKgt ltNHIPgt
• -vci VCI VCI on which to send
  GigE control packets from the CP
• -mac MAC_H16 MAC_L32 hex MAC address to
  assign to GigE card, high 16 bits first then low
  32 bits
• -0 MYIP0 MASK0 NHIP0 hex values for MYIP0
  MASK0 NHIP0
• -1 MYIP1 MASK1 NHIP1 hex values for MYIP1
  MASK1 NHIP1
• -2 MYIP2 MASK2 NHIP2 hex values for MYIP2
  MASK2 NHIP2
• Examplegt ./configGigE.sh -vci 100 -mac 0x4400
  0x5E040001 -0 0xC0A82801 0xFFFFFF00 0xC0A82805 -1
  0 0 0 -2 0 0 0
• this would assign
• MAC address 44 00 5e 04 00 01
• MYIP0 192.168.50.1
• MASK0 255.255.255.0 (24 bits)
• NHIP0 192.168.50.5
• MYIP1 0
• MASK1 0
• NHIP1 0

6
Control Connection for GigE Configuration
CP
VCIN
VCIN
VCI30
P0GLink
P1GigE
APIC
VCIN1
VCIN1
Commands to read/write registers in GigE card
VCI30
P2GigE

• Each GigE card needs a separate bidirectional
  control connection between it and the CP.
• At the GigE end it must use VCI30
• In the example above we have two GigE cards and
  we are using VCI N and VCI N1 at the CP end.
• There are two connections on P0 mapping
• VCI N ? P1, VCI 30
• VCI N1 ? P2 VCI 30
• And one connection on each of P1 and P2
• P1 VCI 30 ? P0 VCI N
• P2 VCI 30 ? P0 VCI N1
• Remember You still need at least one ATM
  interface on your switch so you can connect a CP
  that can send ATM control cells!

7
A Simple Configuration
Host1
VCI128
GigE-1

• IP 192.168.40.2
• 24 bit netmask
• MAC ltas defined by hwgt
• route 192.168.50.0/24 gw 192.168.40.1

GE Eth

• MYIP0 192.168.40.1
• MYMAC 00005E040001
• MASK0 (Dont care)
• NHIP0 (Dont care)

Host2
VCI128
GigE-2

• IP 192.168.50.2
• 24 bit netmask
• MAC ltas defined by hwgt
• route 192.168.40.0/24 gw 192.168.50.1

GE Eth

• MYIP0 192.168.50.1
• MYMAC 00005E040002
• MASK0 (Dont care)
• NHIP0 (Dont care)

8
Simple Configuration Ping

• When Host1 tries to ping Host2 for the first
  time
• Host1 has a route that tells it to use
  192.168.40.1 as the next hop to get to Host 2
  (192.168.50.2)
• Host1 has no ARP entry for 192.168.40.1
• Host1 generates an ARP request asking for MAC
  address of 192.168.40.1 and sends it out its GE
  Ethernet interface
• GigE-1 receives the ARP request which matches its
  MYIP0 address and responds with its MAC address
• Host1 receives the ARP reply
• Host1 then sends out the ping packet.
• GigE-1 receives the ping packet and formulates an
  AAL5 frame and sends the cells of the frame out
  on VPI/VCI 0/128.
• WUGS-20 is configured to send VPI/VCI 0/128 to
  port of GigE-2
• GigE-2 receives first ping packet destined for
  192.168.50.2
• GigE-2 has no ARP entry for 192.168.50.2
• GigE-2 generates an ARP request asking for the
  MAC address of 192.168.50.2 and sends it out its
  ethernet interface.
• Host2 receives the ARP request which matches its
  IP address and responds with its MAC address
• GigE-2 receives the ARP reply and puts an entry
  in its ARP table.

9
Simple Configuration Ping

• When Host1 tries to ping Host2 for the second
  time
• Host1 has a route that tells it to use
  192.168.40.1 as the next hop to get to Host 2
  (192.168.50.2)
• Host1 has an ARP entry for 192.168.40.1
• Host1 sends out an ethernet frame for the ping
  packet with a next hop MAC address of GigE-1.
• GigE-1 receives the ethernet frame and formulates
  an AAL5 frame and sends the cells of the frame
  out on VPI/VCI 0/128
• WUGS-20 is configured to send VPI/VCI 0/128 to
  the port of GigE-2
• GigE-2 receives first ping packet destined for
  192.168.50.2
• GigE-2 has an ARP entry for 192.168.50.2
• GigE-2 sends the ping packet out its ethernet
  interface
• ETC (similar stuff happens for the ping reply
  except some ARP information was gleaned on the
  forward path)

10
A Configuration with a Switch
IP 192.163.204.10 MAC 0800207CE325
Host1
IP 192.163.204.2 MAC 0800207CE325
IP 192.163.204.3 MAC 0800207CF245
Host2
Host3
P3
Ethernet Switch
MSR
P1
Port 1 MAC 00005E040001 MYIP0
192.163.204.1 MASK0 (Dont care) NHIP0
MYIP1 MASK1 NHIP1 MYIP2 MASK2
NHIP2
Host4
IP 192.163.204.4 MAC 0001037C2303
11
A Configuration with a Switch Ping

• When Host1 tries to ping Host2
• The MSR Port 3 is configured to route packets
  destined for Host2 to Port 1 on the base VPI/VCI
  of 0/128.
• Port 1 has a GigE interface.
• Port 1 initially does not have an ARP entry for
  Host2.
• Port 1 sends an ARP request looking for Host2 and
  drops this first ping packet.
• The Ethernet switch sends this packet to Host2
  which sends an ARP reply.
• The Ethernet switch sends the ARP reply back to
  the MSR Port 3.
• The next ping from Host1 to Host2 will go
  through.
• Similar for when Host1 pings Host3 or Host4.
• This configuration still does not need to use the
  Next Hop information in the GigE card.

12
A Configuration with Routers
Host 1 192.168.100.10
192.168.202.0/24 Hosts
192.168.210.0/24 Hosts
192.168.201.0/24 Hosts
P3
Ethernet Switch2
Router2
P0
P1
Ethernet Switch1
MSR
P0
192.168.200.0/24 Hosts
Ethernet Switch3
Router3
P0
P1
192.168.220.0/24 Hosts
13
A Configuration with Routers

• In this configuration we have Ethernet Switch1
  supporting three subnets
• 192.168.200.0/24
• 192.168.201.0/24
• 192.168.202.0/24
• Each of the Routers (Router1, Router2, MSR) has a
  presence in each of the subnets (200, 201, 202)
  on their P0 interfaces
• MSR P0 (limited to three)
• 192.168.200.1
• 192.168.201.1
• 192.168.202.1
• Router2 P0
• 192.168.200.2
• 192.168.201.2
• 192.168.202.2
• Router3 P0
• 192.168.200.3
• 192.168.201.3
• 192.168.202.3
• MSR also supports hosts on the 192.168.100.0/24
  subnet
• such as Host1 192.168.100.10

14
A Configuration with Routers (cont)

• MSR P0 GigE Card Configuration
• MAC 44 00 5E 04 00 01
• MYIP0 192.168.200.1
• MASK0 0xffffff00
• NHIP0 192.168.200.2
• used for Egress VCI 129
• MYIP1 192.168.201.1
• MASK1 0xffffff00
• NHIP1192.168.201.3
• used for Egress VCI 130
• MYIP2 192.168.202.1
• MASK2 0
• NHIP20
• used for Egress VCI 131 (unused in this example)

15
Example Commands to Configure P0 GigE

• Assume that VCI 101 is set up in the MSR switch
  to go from the CP port to P0 VCI 30.
• cd WUARL/wugs/gige/src/configScripts
• ./configGigE.sh -vci 101 -mac 0x4400 0x5E040001
• -0 0xc0A8C801
  0xffffff00 0xc0A8C802
• -1 0xc0A8C901
  0xffffff00 0xc0A8C903
• -2 0xc0A8CA01 0 0

16
A Configuration with Routers Ping

• When Host1 tries to ping any host in the (200,
  201, 202) subnets
• Just like our earlier example where there were no
  routers.

17
A Configuration with Routers Ping

• When Host1 tries to ping a host in the 210
  subnet
• The MSR P3 is configured to route packets
  destined for the 210 subnet to P0 on the VPI/VCI
  of 0/129. This will cause P0 to use MYIP0, MASK0
  and NHIP0.
• NHIP0 is the IP address of Router2
• A function applied to (MYIP0, MASK0, NHIP0) is
  used to generate an index into the ARP table in
  the GE card to store the MAC address of Router2
• details are not important at this point.
• P0 has a GigE interface.
• P0 GigE initially does not have an ARP entry for
  Router2.
• P0 GigE sends an ARP request looking for Router2
  and drops this first ping packet.
• Router2 sends an ARP reply with the MAC address
  from its P0.
• The next ping from Host1 to the 210 subnet will
  go through.

18
A Configuration with Routers Ping

• When Host1 tries to ping a host in the 220
  subnet
• The MSR P3 is configured to route packets
  destined for the 220 subnet to P0 on the VPI/VCI
  of 0/130. This will cause P0 to use MYIP1, MASK1
  and NHIP1.
• NHIP1 is the IP address of Router3
• A function applied to (MYIP1, MASK1, NHIP1) is
  used to generate an index into the ARP table in
  the GE card to store the MAC address of Router3
• details are not important at this point.
• P0 has a GigE interface.
• P0 GigE initially does not have an ARP entry for
  Router3.
• P0 GigE sends an ARP request looking for Router3
  and drops this first ping packet.
• Router3 sends an ARP reply with the MAC address
  from its P0.
• The next ping from Host1 to the 220 subnet will
  go through.

19
Performance

• In our tests we have seen the following
• Using two 2.4 GHz Athlon machines
• Using iperf
• UDP
• packet size 512 bytes
• sending rate 720 Mb/s
• receiving rate 720 Mb/s with about 0.2 drops
  at receiver
• receiving host couldnt keep up
• TCP 700 Mb/s - 770 Mb/s
• using the iperf defaults
• 8K buffer size
• MSS set by TCP based on MTU