APIC

1

• APIC
• Tutorial
• ---
• Software
• Drivers, Libraries and Utilities
• John DeHart
• Washington University
• jdd_at_arl.wustl.edu
• http//www.arl.wustl.edu/jdd

2
Overview

• Drivers
• NetBSD
• Linux
• We will NOT talk about the MSR APIC Driver!!!
• User Mode Library
• APICCell library
• Utilities
• atm_ifconfig
• Traffic Generators

3
Driver Overview

• The Big Picture
• IP
• RATM

4
Driver Overview (Especially NetBSD)
User
Kernel
5
IPv4 Forwarding in the NetBSD Kernel (APIC)
TRANSPORT
socket layer
tcp/udp_input
tcp/udp_output
N E T W O R K
ip6_forward
ip6_output
ip6_input
ip_forward
ip_output
ipintr
U PPER
D A T A
atmc_input
L OWER
L I N K
apic_isr
apic_intr
input packets
interrupts
output packets
6
User IP in the NetBSD Kernel
TRANSPORT
socket layer
tcp/udp_input
tcp/udp_output
N E T W O R K
ip_forward
ip_output
ipintr
U PPER
D A T A
atmc_input
L OWER
L I N K
apic_isr
apic_intr
input packets
interrupts
output packets
7
Two Sets of Receive Descriptors/Buffers
AAL5 Receive Pool Chain
Cluster mbuf
Cluster mbuf
Cluster mbuf
...
D
D
D
Host Memory
AAL5 buffer
AAL0 buffer
Cluster mbuf
- 48 gt
- 56 gt
2048
2016
2016
Currently the same size, but not necessarily so
8
Interrupt on Done
Driver Reads Which Channel Caused Intr.
AAL5 Frame
IP Q
EOF
Interrupt
Chan N
Done
Done
Done
Host Memory
Channel N
Notify Register
9
Interrupt on Descriptor Read
Driver Reads Which Channel Caused Intr.
Receive Pool Chain
RATM Q
Cluster mbuf
Cluster mbuf
APIC Reads Next Descriptor Makes it Current
Descriptor for Chan N
Interrupt
Done
Chan N
Host Memory
...
D
D
Host Memory
Channel N
Current Descriptor
10
Receive Pools Two Per-Frame, One Per-Cell

• Two Per-Frame Pools
• AAL5
• Interrupt on DONE
• size lt requested system buffer (mbuf) size
• size 48 0
• Used for IP and RATM AAL5 applications
• AAL0
• Interrupt on Descriptor Read
• size lt requested system buffer (mbuf) size
• size 56 0
• Used for continuous AAL0 data applications
• For current cluster mbuf
• AAL0_size AAL5_size 2016
• Fairly large number of buffers (100s to 1000s)

11
Receive Pools (cont)

• AAL0 per-cell Pool
• buffer size 56 bytes
• Interrupt on Descriptor Read
• Each time APIC reads a new Descriptor, the
  previous one is available for the Driver to
  process
• Used for per-cell AAL0 applications like GBNSC
• Small number of buffers (10s)
• CAVEAT Deadly if used for high bandwidth channel
• Interrupt on EVERY Cell
• We are using a cluster mbuf (2K Bytes) for each
  cell!!
• Can we use regular mbuf (200 Bytes)?
• Physical address alignment implications?

12
atm_ifconfig

• NetBSD ATM VC configuration utility
• gt atm_ifconfig
• Usage atm_ifconfig ltinterfacegt closeall
• Usage atm_ifconfig ltinterfacegt getall
• Usage atm_ifconfig ltinterfacegt ltvpivcigt
  openclosemodify options...
• Options are
• -aal0 -aal0_cell -aal5
• -llc -nollc ltprotocolgt use LLC/SNAP?
• protocol is inet ratm value
  of ethernet type
• -loopback
• -inport FIBER,RIBBON,SWITCH,LINK
• -outports FIBER,RIBBON,SWITCH,LINK,BOTH
• -lowdelay -paced ltrategt -besteffort
• -maxqueue ltmax_descriptors_in_tx_queuegt
• rate is in Kbits/sec

13
NetBSD Device, VC and Route Configuration

• ifconfig apic0 inet 192.168.208.2 netmask
  0xffffff00 up
• Configure VCI 32 for newGBNSC to use (NetBSD)
• atm_ifconfig apic0 0x20 open -aal0_cell -nollc
  ratm -inport FIBER -outports FIBER
• Configure VCI 100 for IP usage (NetBSD)
• atm_ifconfig apic0 0x64 open aal5 -nollc inet
  -inport FIBER -outports FIBER
• route add iface 192.168.208.1 -link
  apic00.0.0.64

Error on handouts
14
Linux Device, VC and Route Configuration

• /etc/sysconfig/network-scripts/ifcfg-atm0
• /etc/init.d/atm start
• atmsigd
• ilmid
• atmarp
• route add

15
Linux Device, VC and Route Configuration (cont)

• /etc/sysconfig/network-scripts/ifcfg-atm
• DEVICEatm0 device name wuapic0
• DEVICE_ADDRdemand4-apic0 host's hardware
  address (made up)
• IPADDR192.168.10.3 hosts's IP address on the
  ATM LIS
• NETMASK255.255.255.0 netmask (e.g. arl uses
  8 host bits)
• NETWORK192.168.10.0 network (e.g. arl's
  experimental ATM LIS)
• BROADCAST192.168.10.255 broadcast address (not
  really useful)
• ARPSRVlrcsuns-a ATM address of ATMARP
  server omit if local
• ARPSRVQOS"ubrmax_pcr100kbps" QOS of
  the ATMARP server VC
• ARPDEFQOS"ubrmax_pcr700kbps" default
  QOS for IP over ATM VCs
• ONBOOTno needs to be loaded by
  ATM-specific procedure
• MTU32768 larger packets ok on tx

16
Linux Device, VC and Route Configuration (cont)

• gt /etc/init.d/atm start
• Starting ATM demons OK
• gt ps -auxwww grep atmsigd
• root 973 0.0 0.1 1708 628 pts/1 S
  0956 000 atmsigd -b -l syslog -D /var/tmp -t
  20
• root 1019 0.0 0.1 1708 600 pts/1 S
  0956 000 grep atmsigd
• gt ps -auxwww grep ilmid
• root 984 0.0 0.0 1428 384 pts/1 S
  0956 000 ilmid -b -l syslog
• root 1021 0.0 0.1 1708 600 pts/1 S
  0956 000 grep ilmid
• YOU MAY WANT TO KILL THESE
• kill 973 984
• atmsigd and ilmid are started in /etc/init.d/atm
• If you are SURE you dont want them on your
  machines back home, that is the place to edit
  them out. DO NOT do that here!!!

17
Linux Device, VC and Route Configuration (cont)

• gt ifconfig atm0
• atm0 Link encapUNSPEC HWaddr
  00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
• inet addr192.168.10.3
  Mask255.255.255.0
• UP RUNNING MTU32768 Metric1
• RX packets0 errors0 dropped0
  overruns0 frame0
• TX packets0 errors0 dropped0
  overruns0 carrier0
• collisions0 txqueuelen100

18
Linux Device, VC and Route Configuration (cont)

• gt atmarp -s 192.168.10.1 0.0.50 null
• for a network route
• gt route add -net 192.168.10.0 netmask
  255.255.255.0 gw 192.168.10.1
• for a host route
• gt route add host 192.168.10.2 gw 192.168.10.1
• gt atmarp -a
• ----- Itf 0 (192.168.10.3, netmask 255.255.255.0)
  -----
• Default QOS ubr,aal5max_sdu9188
• IP 192.168.10.1, state VALID, addr ltnonegt, flags
  0x1004ltPERM,NULLgt
• 0.0.50
• Send buffer 65535
• ----- Unknown incoming connections -----
• ----- Incoming unidirectional connections -----
• ----- End of dump -----

19
Linux Device, VC and Route Configuration (cont)

• gt netstat -nr
• Kernel IP routing table
• Destination Gateway Genmask
  Flags MSS Window irtt Iface
• 128.252.153.0 0.0.0.0 255.255.255.0
  U 40 0 0 eth0
• 192.168.10.0 192.168.10.1 255.255.255.0
  UG 40 0 0 atm0
• 192.168.10.0 0.0.0.0
  255.255.255.0 U 40 0 0 atm0
• 192.168.10.0 0.0.0.0 255.255.255.0
  U 40 0 0 atm0
• 127.0.0.0 0.0.0.0 255.0.0.0
  U 40 0 0 lo
• 0.0.0.0 128.252.153.249 0.0.0.0
  UG 40 0 0 eth0

20
User Mode Library

• User Mode library is Driver functionality exposed
  to User Space programs.
• You get direct manipulation of Descriptors and
  Buffers
• Access to APIC Registers
• It has many of the complexities of a Driver.
• It has many of the DANGERS of a Driver.
• Source wu_arl/utilities/APIC/ApicLibrary
• Binary wu_arl/utilities/lib/(OSTYPE)/libAPIC.a
• Installed include files wu_arl/utilities/include/

21
User Mode Library

• Following slides contain code from one of our
  utility programs wu_arl/utilities/APIC/FileTransf
  er/FileRead.c
• Best way to learn how to use the User Mode
  library is to start from existing code.
• If you want to use the User Mode library
• YOU MUST INVEST SOME TIME!!!
• Start by using the FileTransfer utility.
• Get it to work for you.
• Add your own debug statements to it
• Figure out what it is doing.
• Then use it as an example to do what you want.

22
User Mode library example

• / From wu_arl/utilities/APIC/FileTransfer/FileRea
  d.c /
• / Initialize /
• i InitApicLibrary(false) / try for an
  apic /
• if (i)
• fprintf(stdout, "Failed to grab an apic -
  is the user mode code enabled?\n")
• exit(-1)
• / map the Data (main data pool) /
• ApicHandle ApicLibraryMap(0, / use default
  apic id /
• APIC_LIBRARY_READ5, / open in read
  mode /
• i, / initial descr /
• FramesReady, / buffer count /
• 4096, / size of frames /
• rvpi, / VPI connection /
• rvci, / VCI connection /
• ENDIANNESS_LITTLE, / endian
  (little) /

23
User Mode library example

• / From wu_arl/utilities/APIC/FileTransfer/FileRea
  d.c /
• / Start reading /
• AckHandle ApicHandle / save output vci
  data /
• if (ApicHandle-gtudescr lt 0)
• fprintf(stdout, "No AckPool descriptors
  allocated - see /var/log/messages\n")
• exit(-1)
• AckFirstDescr AckHandle-gtudescr / obtain
  first used descriptor /
• AckCount AckHandle-gtudescc
• AckLastDescr AckFirstDescr AckCount - 1
• bzero(AckHandle-gtubuf, 2 page_size)
• i ApicWriteDescriptor0(AckFirstDescr 1,
  NotReady, AckHandle)
• NextDescr DataFirstDescr / init circular
  chain /
• i ApicStartAAL5Read(DataHandle) / always
  read first /
• fprintf(stdout, "Looking for FileSend AAL5
  data\n")
• SendAckNak(-1, ACKNAK_IDLE1) / send an I'm
  Alive /

24
User Mode library example

• / From wu_arl/utilities/APIC/FileTransfer/FileRea
  d.c /
• / Processing loop. Process36K() does the work /
• while (!kern_start_addr)
• int what Process36K()
• if (what lt 0)
• if (Debug)
• fprintf(stdout, "failure code d\n",
  what)
• if (LastSequence -1)
• // we still haven't received our first frame.
  Continuing sending pings
• SendAckNak(-1, ACKNAK_IDLE1) / send
  an I'm Alive /
• else
• SendAckNak(LastSequence, what) /
  ping the other side /
• continue
• SendAckNak(LastSequence, ACKNAK_OK) /
  say ok to the other side /

25
User Mode library example

• / From wu_arl/utilities/APIC/FileTransfer/FileRea
  d.c /
• / In Process36K() Check for a DONE_VALID
  descriptor /
• / patrol the descriptors for this i/o /
• while (!CheckReady(NextDescr)) / (136K,
  14k) X 8 segments /
• w0 ApicReadDescriptor0(NextDescr,
  DataHandle)
• w1 ApicReadDescriptor1(NextDescr,
  DataHandle)
• if (apic_fval(w0, DESC_SYNC) !
  DESC_SYNC_DONE_VALID_LINK)
• if (Debug)
• fprintf(stdout,
• "Not Valid Next Descr0x04x,
  W00x08x, W10x08x\n",
• NextDescr, w0, w1)
• return retcode

26
User Mode library example

• / From wu_arl/utilities/APIC/FileTransfer/FileRea
  d.c /
• / In Process36K() Examine data /
• / we have a valid well aligned frame /
• Trailer (unsigned int ) ((unsigned char
  ) DataHandle-gtubuf offset 4096) / get
  real address /
• LT (struct load_trailer ) Trailer /
  make a struct of it /
• Bytes LT-gtDataSize0
• Swap LT-gtByteSwap0 / get byte swap code
  /
• Dest (unsigned int ) LT-gtPAddress0
• LastSequence LT-gtSequence0

27
User Mode library example

• / From wu_arl/utilities/APIC/FileTransfer/FileRea
  d.c /
• / In Process36K() Write Data to a file /
• if (fd)
• int s1 (int ) ((unsigned char )
  DataHandle-gtubuf
• ((offset - APIC_AAL5_PAYLOAD_PAGES)
  4096))
• int s2 s1
• int q 0
• for (q 0 q lt 512 8 q) / swap words
  for output /
• int x s2
• s2 (s2 1)
• (s2 1) x
• s2 2
• out write(fd, s1, LT-gtDataSize0)

28
AAL5Generator

• Uses APIC User Mode Library
• Configures one APIC descriptor/buffer pair
• Sets descriptor to point to itself and be
  Read-Only
• INFINITE LOOP!! (but its a good infinite loop)
• APIC will transmit the buffer until descriptor is
  turned off
• After a given length of time (approximate!!)
• Turns descriptor off
• Uses APIC global pacer and per VC pacing
• controls bit/sec rate and packet/sec rate
• Intended for high performance test and analysis
• Multiple instances can run at same time on same
  machine
• Companion sendCommand utility
• to change characteristics on the fly
• Need to add control for APIC Batch size
• APIC batches cells transmitted. Controlled by an
  APIC register
• easy to add control, just has not been done yet.

29
sendCommand

• (Different then the MSR sendcmd you will see
  tomorrow)
• Provides remote control of AAL5Generator
• changes the charateristics of a running
  AAL5Generator
• What can be changed
• Packet Rate
• Kbit Rate
• Destination Address
• Run State
• Quit
• Stop
• Start

30
sendCommand

• Sample Commands
• Sends the command to AAL5Generator on tabby which
  is listening on port 4444 for incoming commands
• Change the packet per second rate to 10000
• sendCommand p 4444 tabby Packet Rate 10000
• Change the Kbit per second rate to 10000
• sendCommand p 4444 tabby Kbit Rate 10000
• Change the Destination IP Address to
  192.168.212.2
• sendCommand p 4444 tabby DestAddr
  192.168.212.2
• Stop sending packets
• sendCommand p 4444 tabby Stop
• Restart sending packets
• sendCommand p 4444 tabby Start
• Quit causes AAL5Generator to Stop sending and
  exit
• sendCommand p 4444 tabby Quit

31
Source and Executables

• source location
• wu_arl/utilities/APIC/AAL5Generator
• binary location
• wu_arl/utilities/bin/OSTYPE/AAL5Generator
• wu_arl/utilities/bin/OSTYPE/sendCommand

32
AAL5Generator Usage

• Most Used Command Line Options
• -svci (VCI for local sending of AAL5 Frames)
• -kbits (xmit rate using paced link rate)
• -frate (Frames/sec)
• -seconds (running time, 0 means run forever)
• -packet (20 lt X lt1912) pkt size (bytes)
  includes IP hdr
• -port (UDP port on which to receive ctrl
  commands)
• -src (IP Source address to put in packet IP
  Header)
• -dst (IP Destination address to put in packet
  IP Hdr)

33
AAL5Generator Usage (contd)

• Less Used Command Line Options
• -pacer (088Mb/S lt X lt 2553.5Mb/S pacer
  clock)
• -svpi (VPI for local sending of AAL5 Frames
  (0))
• -rate (16 bit divisor whole)
• -frac (8 bit divisor fractional)
• -fsize (0ltXlt7) (40,136,232,520,760,1192,1480
  ,1912)
• -descr (0 lt X lt 64K initial descr)
• -debug (print debug info)
• -swap (htonl() byte swap output)
• -apic (Linux Apic Id (0 lt X lt 3))

34
APICCell Library API

• Provides a cell manager for sending and receiving
  APIC control cells
• Uses the GBNCELL_CellManager
• for now it assumes we are talking to a WUGS
• thus it is used to talk to SPCs
• Provides classes for APIC Registers
• Provides member functions for reading and
  writing
• External Memory
• Global Registers
• Transmit Registers
• Receive Registers

35
APICCell Library Files

• Source location
• wu_arl/wugs/src/common_code/APICCELL
• Binary
• wu_arl/wugs/lib/(OSTYPE)/libAPICCELL.a
• Sample test code
• wu_arl/wugs/src/common_code/APICCELL/test_code

36
APICCell Library Example

• // Class defn from wu_arl/wugs/src/common_code/API
  CCELL/test_code/RWPacingParam.h
• class RWPacingParameter public
  APIC_OperationBase
• private
• APIC_CellManager cellManager
• Boolean done
• Boolean isRead
• public
• RWPacingParameter(APIC_CellManager cm)
  APIC_OperationBase()
• cellManager cm
• done FALSE
• isRead TRUE
• void readParameter()
• void writeParameter(int p)
• virtual void failed(APIC_ControlCell c)
• virtual void receiveCell(APIC_ControlCell c)
• Boolean isDone() return done

37
APICCell Library Example

• // Snippet of code from
• // wu_arl/./APICCELL/test_code/RWPacingParam.C
• void
• RWPacingParameterwriteParameter(int p)
• isRead FALSE
• done FALSE
• APIC_ControlCell cell
• APIC_RegisterData tmp_data
  cell.writeGlobalRegister (
• APIC_RegID_PacingParameter, // write this
  global apic register
• TRUE, //cell should
  return
• NULL, //data to send
  NULL causes one to be a data
• // structure to be created
  and returned to us
• getCMData()) //and this CMDATA value
  
• tmp_data-gtsetPacingParameter(p)
• cellManager-gtsend(cell)

38
APICCell Library Example

• // Snippet of code from
• // wu_arl/./APICCELL/test_code/RWPacingParam.C
• void
• RWPacingParameterreceiveCell(APIC_ControlCell
  cell)
• APIC_RegisterData data cell.getData()
• cout ltlt endl ltlt "\t" ltlt data-gtgetName()
• if (isRead)
• cout ltlt "\t " ltlt data-gtgetPacingParameter(
  ) ltlt endl
• else cout ltlt " \t returned " ltlt endl
• done TRUE

39
Notes
40
Notes