ECE 6160: Advanced Computer Networks InfiniBand

1
ECE 6160 Advanced Computer NetworksInfiniBand

• Instructor Dr. Xubin (Ben) He
• Email Hexb_at_tntech.edu
• Tel [phone]

Note Some of these slides are adapted from
InfiniBand Trade Association (http//www.infiniban
dta.org).
2
Prev

• Networked Storage
• NAS, SAN
• iSCSI
• STICS

3
I/O Systems
interrupts
Processor
Cache
Memory - I/O Bus
Main Memory
I/O Controller
I/O Controller
I/O Controller
Graphics
Disk
Disk
Network
4
InfiniBand Architecture (IBA) Focus

• 0 Minimized cabling
• 0 Intra-box / Inter-box solution
• 0 Simplified Switches
• 0 Server application architecture support
• 0 RAS features necessary for 24x7 systems

5
Why InfiniBand?

• Ease of connect
• 0One fabric connection for all host I/O
• IPC, Storage I/O, Network I/O
• 0Improved upgrade/growth strategy
• Scalability
• 0Thousands of nodes per subnet / multiple
  subnets
• 0Cost-effective intra-subnet switches
• 0Multiple link widths for high bandwidth
• Performance
• 0CPU-offloaded hardware support for message
  queuing,memory protection, fabric protocol
  processing

6
Why InfiniBand? (Contd.)

• Reliability, Availability, Serviceability
• 0Redundant paths and/or fabrics
• 0In-band management
• Connector, backplane, baseboard, chassis, power
• 0Layered error management architecture
• Flexibility
• 0Topology independent
• 0Multiple link widths
• 0Multiple units of transfer
• 0Auto-negotiation
• 0Multiple form factors / standardized connectors

7
IB Architecture Model
8
IB Architecture Layers
9
Physical Layer Physical protocol overview
10
Link Characteristics
11
Work queue
12
Host Channel Adapter
13
Target Channel Adapter
14
Switch
15
Router
16
Addressing

• For subnet transactions within a cluster group,
  IB uses 64-bit addressing. (LID)
• For external transactions between IB subnets, it
  uses 128-bit IPv6 addressing. (GID)
• Easily integrate into mainstream IP networking
  via IP routers.

17
Link Layer

• Responsible for the orderly reception and
  transmission of packets including
• 0Link initialization and control
• 0Provide Virtual Lanes (VLs)
• 0Transmit packets
• Control flow
• Arbitration between Virtual Lanes (VLs)
• 0Receive packets
• Error checking

18
Virtual Lanes (VLs)

• Communications over the IB fabric are performed
  through VLs.
• A VL represents a pair of receive and transmit
  buffers on a port, with each VL providing its own
  flow control
• Each port/link may support up to 15 VLs for data
  and one for management.
• The data streams originating from an HCA are
  mapped to VLs and are muxed into a single stream
  over the link, and demuxed at the receiving side.

19
Virtual Lane
20
Class of service

• Class of Service is intended for use in traffic
  isolation or providing differentiated services
• IBA does not assign specific meaning to Class of
  Service
• Service Level (SL) is
• 0used to identify service class of a flow
• 0a property of the connection or datagram
• 0a 4-bit field of the packet
• SL defines VL to use on each hop

21
Link Layer Flow Control

• Each VL has separate buffering and flow control
• Credit based flow control is used
• 0Credits sent on per VL basis
• 0Receiver issues credit based on packets taken
  from incoming VL buffer
• 0Credits sent to sender
• 0Sender manages flow control

22
Transport Layer

• The five Transport Services
• 0Reliable Connection
• 0Reliable Datagram
• 0Unreliable Connection
• 0Unreliable Datagram
• 0Raw Datagram
• The Transport Functions
• 0Send
• 0RDMA Read Write
• 0Atomic
• not all functions apply to all the services
• send WQE indicates the function type to use

23
Message
24
Data Packets
25
Path MTU
26
Transport services one purpose
27
IBA reliable transport
28
IBA unreliable service
29
Raw Datagrams
30
Service Reliable Connection
31
Service Unreliable Connection
32
Service Reliable Datagram
33
Service Unreliable Datagram
34
Error Detection
35
Summary

• InfiniBand replaces the traditional server bus
  with a high-speed switch.
• Communication paths over InfiniBand links are
  established by VLs
• Local IB grouping (subnet) communications are
  performed via data link switching.
• Intersubnet communication is based on IPv6
  addressing and network routing.
• Intelligent storage switches may provide
  any-to-any storage connectivity.
• Overall, IB provides a solution for both internal
  and external connection of server nodes.