ECE 6160: Advanced Computer Networks InfiniBand

1
ECE 6160 Advanced Computer NetworksInfiniBand

• Instructor Dr. Xubin (Ben) He
• Email Hexb_at_tntech.edu
• Tel [phone]

Note Some of these slides are adapted from
InfiniBand Trade Association (http//www.infiniban
dta.org).
2
Prev

• Networked Storage
• NAS, SAN
• iSCSI

3
I/O Systems
interrupts
Processor
Cache
Memory - I/O Bus
Main Memory
I/O Controller
I/O Controller
I/O Controller
Graphics
Disk
Disk
Network
4
InfiniBand Architecture (IBA) Focus

• 0 Minimized cabling
• 0 Intra-box / Inter-box solution
• 0 Simplified Switches
• 0 Server application architecture support
• 0 RAS features necessary for 24x7 systems

5
What is Infiniband?

• Infiniband Architecture(IBA) defines a System
  Area Network (SAN)
• IBA SAN is a communications and management
  infrastructure for I/O and IPC
• IBA defines a switched communications fabric
• high bandwidth and low latency
• protected, remotely managed environment.
• IBA hardware off-loads from the CPU much of the
  I/O communications operation.

6
Why InfiniBand?

• Ease of connect
• 0One fabric connection for all host I/O
• IPC, Storage I/O, Network I/O
• 0Improved upgrade/growth strategy
• Scalability
• 0Thousands of nodes per subnet / multiple
  subnets
• 0Cost-effective intra-subnet switches
• 0Multiple link widths for high bandwidth
• Performance
• 0CPU-offloaded hardware support for message
  queuing, memory protection, fabric protocol
  processing

7
Why InfiniBand? (Contd.)

• Reliability, Availability, Serviceability
• 0Redundant paths and/or fabrics
• 0In-band management
• Connector, backplane, baseboard, chassis, power
• 0Layered error management architecture
• Flexibility
• 0Topology independent
• 0Multiple link widths
• 0Multiple units of transfer
• 0Auto-negotiation
• 0Multiple form factors / standardized connectors

8
IB Architecture Model
9
IB Architecture Layers
10
Physical Layer Physical protocol overview
11
Link Characteristics
12
Work queue
13
Host Channel Adapter
14
Target Channel Adapter
15
Switch
16
Router
17
Addressing

• For subnet transactions within a cluster group,
  IB uses 64-bit addressing. (LID)
• For external transactions between IB subnets, it
  uses 128-bit IPv6 addressing. (GID)
• Easily integrate into mainstream IP networking
  via IP routers.

18
Link Layer

• Responsible for the orderly reception and
  transmission of packets including
• 0Link initialization and control
• 0Provide Virtual Lanes (VLs)
• 0Transmit packets
• Control flow
• Arbitration between Virtual Lanes (VLs)
• 0Receive packets
• Error checking

19
Virtual Lanes (VLs)

• Communications over the IB fabric are performed
  through VLs.
• A VL represents a pair of receive and transmit
  buffers on a port, with each VL providing its own
  flow control
• Each port/link may support up to 15 VLs for data
  and one for management.
• The data streams originating from an HCA are
  mapped to VLs and are muxed into a single stream
  over the link, and demuxed at the receiving side.

20
Virtual Lane
21
Class of service

• Class of Service is intended for use in traffic
  isolation or providing differentiated services
• IBA does not assign specific meaning to Class of
  Service
• Service Level (SL) is
• 0used to identify service class of a flow
• 0a property of the connection or datagram
• 0a 4-bit field of the packet
• SL defines VL to use on each hop

22
Link Layer Flow Control

• Each VL has separate buffering and flow control
• Credit based flow control is used
• 0Credits sent on per VL basis
• 0Receiver issues credit based on packets taken
  from incoming VL buffer
• 0Credits sent to sender
• 0Sender manages flow control

23
Transport Layer

• The five Transport Services
• 0Reliable Connection
• 0Reliable Datagram
• 0Unreliable Connection
• 0Unreliable Datagram
• 0Raw Datagram
• The Transport Functions
• 0Send
• 0RDMA Read Write
• 0Atomic
• not all functions apply to all the services
• send WQE indicates the function type to use

24
Message
25
Data Packets
26
Path MTU
27
Transport services one purpose
28
IBA reliable transport
29
IBA unreliable service
30
Raw Datagrams
31
Service Reliable Connection
32
Service Unreliable Connection
33
Service Reliable Datagram
34
Service Unreliable Datagram
35
Error Detection
36
Summary

• InfiniBand replaces the traditional server bus
  with a high-speed switch.
• Communication paths over InfiniBand links are
  established by VLs
• Local IB grouping (subnet) communications are
  performed via data link switching.
• Intersubnet communication is based on IPv6
  addressing and network routing.
• Intelligent storage switches may provide
  any-to-any storage connectivity.
• Overall, IB provides a solution for both internal
  and external connection of server nodes.