Solaris IPoIB (ibd) Implementation

1
Solaris IPoIB (ibd) Implementation

• Kanoj Sarcar
• Sr Staff Engineer
• Sun Microsystems
• (presented by Bill Strahm)

2
Architecture Overview

• DLPI driver
• Only supported for GID0
• Interop (TopSpin, openib)
• Software structure
• IP SDP uDAPL
• ibd ...
• IBTF / IBMF
• HCA driver
• Performance

3
Tools changes (20 byte MAC)

• New SIOCXARP ioctls and struct xarpreq
• arp(1M) changes
• PPPD proxyarp etc
• Undocumented helper function ala ether_ntoa,
  ether_aton
• extern char _link_ntoa(const unsigned char ,
  char , int, int)
• extern unsigned char _link_aton(const char ,
  int )
• ifconfig(1M), snoop(1M)
• RARP
• /etc/ethers, NIS assume 6 bytes!

4
Tools changes (IPoIB specific)

• DHCP agent
• Usage of BROADCAST bit
• Setting IB hardware type in DHCP packet in htype
• Client id format
• Code Len Type lt----------------
  Client-Identifier --------------gt
• ---------------------------------------
  ---------------....----
• 61 21 00 00 (4 octets)
  16 octet GID
• ---------------------------------------
  ---------------....----
• DHCP Server
• No explicit ARP cache setting with DHCP chaddr
• Snoop(1M) parses IPoIB, disallows MAC filters

5
Tools changes (IPoIB specific)Part II

• ifconfig(1M)
• hme0 flags1000843ltUP,BROADCAST,RUNNING,MULTICAST
  ,IPv4gt mtu 1500 index 2
• inet 10.6.98.56 netmask ff000000
  broadcast 10.255.255.255
• ether 03ba244df
• ibd1 flags1000843ltUP,BROADCAST,RUNNING,MULTICAST
  ,IPv4gt mtu 2044 index 4
• inet 192.168.100.100 netmask ffffff00
  broadcast 192.168.100.255
• ipib 004a000000123402c919
  765711
• SNMP (net-snmp)
• ifSpeed consistent with draft-ietf-ipoib-ibif-mib-
  07.txt
• ifPhysAddress as MAC address (to match ARP)
• octet/packet counts only for ipoib network
  interface

6
IPoIB oddities

• DLPI receiver indicates src addr unavailable
  for unicast packets
• If transmitter guarantees GID0 usage, SM assisted
  reverse lookup possible
• RARP unsupported
• Non persistant MAC
• Server requires client's MAC address for 3rd
  party lookup

7
IPoIB Configuration

• One time discovery
• Reconfiguration boot or IO discovery command
• Create possible ipoib instance per port/Pkey
• SM must be up
• Interface initialization
• IP plumb triggers MCG probe as validity check
• SM must be up, fabric admin must create IP MCG
• Decisions
• End nodes do not create IP MCG
• End nodes do not have to know IB parameters
• Issues
• IP multicast group membership requests
• Can not wait, can not spool!
• Solaris IP stack joins 224.0.0.1 at interface init

8
Interesting Driver Details

• QPN/CQ/WQE/PathRecord
• Tx path copy vs registration
• GID/QPN to Pathrecord (hash/cache)
• Disallow (DLPI) change MAC address
• Async thread for (blocking) SM communication
• IB link up/down handling
• ULP notification for IPMP failover
• MCG create/delete trap handling
• 2 MCG membership lists (full/sendonly, non)
• MCG pathrecord and membership implication