Storage area network and System area network (SAN)

1

• Storage area network and System area network
  (SAN)
• What are they?
• Network requirements
• Hardware/software issues
• References
• Ulf Troppens, Rainer Erkens, a nd Wolfgang
  Muller, Storage Networks Explained - basic and
  application of Fibre Channel SAN, NAS, iSCSI and
  Infiniband, John Wiley Sons, 2004.
• W. J. Dally and B. Towles, Principles and
  Practices of Interconnection Networks, Morgan
  Kaufmann, 2004.
• Ajay V. Bhatt, Creating a Third Generation I/O
  Interconnect, available at http//www.express-lan
  e.org

2

• Storage area network (SAN)
• Server-centric IT architecture storage devices
  exist only with servers

3

• Storage-centric IT architecture SCSI cables are
  replaced by a network (storage is now independent
  of servers).

4

• Storage area network (SAN) requirement
• Serial transmission for high speed and long
  distance
• Low transmission errors
• Low delay of transmitted data
• Needs to make it feel like using a local disk
• Low delay is a relative term
• The disk subsystem has around 1ms 10ms latency
  itself.
• The communication protocol should not use CPU.

5

• Current Storage area network (SAN) technology
  (IBM)
• Fibre Channel
• TCP/IP Gigabit Ethernet (iSCSI)
• InfiniBand

6

• System area network a network with a high
  bandwidth and a low lantency that serves as a
  connection between computers in a distributed
  computer system.

7

• Why system area network
• Historically, the system area network comes with
  a particular parallel machine (supercomputer,
  e.g. Cray T3D, Cray T3E, SGI origin 2000, IBM SP,
  Thinking machine CM5, Intel Polygon)
• The network is very expensive due to low volume
• CPU is two generations behind
• A more cost effective way to build these system
  is to decouple the processor technology from the
  networking technology.
• To form cheaper clusters of workstations with the
  off-the-shelf system area network technology
  (compared to traditional supercomputers).
• current system area networks
• Myrinet, Quadrics, Infiniband

8

• System area network requirement
• Low latency and high bandwidth at the application
  level.
• Not just at the hardware level
• Not just at the system level
• Implicitation
• Hardware, network interface, software messaging
  layer should work together to achieve the goal.
• Infiniband is designed as both storage area
  network and system area network.

9

• Hardware issues
• High speed links
• Infiniband 2.5Gbps 250MBps, 10Gbps1GBps, 30
  Gbps 1GBps
• Fibre channel 100MBps, 200MBps, 400MBps, 1GBps.
• Myrinet up to 9.6Gbps
• As a reference PCI bus 100MBps
• NIC may need to attach to the memory bridge

10

• A typical PC

11

• A workstation connected to a system area network

12

• When the number of end points is large, multiple
  switches will be needed.
• Topology
• Switching
• Routing

13

• Topology
• Static arrangement of channels and nodes in an
  interconnection network
• Trade-off between cost and performance
• Cost the number and complexity of chips, density
  and length of the interconnections, etc.
• Performance
• Bandwidth and latency also depend on other
  factors other than topology
• Topology performance metrics Bisection
  bandwidth, diameter, nodal degree, channel load

14

• A cut of a network is the set of channels that
  partitions the set of all nodes into two disjoint
  sets.
• A bisection of a network is a cut that partitions
  the network nodes in roughly half.
• The bisection bandwidth of a network is the
  minimum bandwidth over all bisections of the
  network.
• The diameter of a network is the largest minimal
  hop count over all pairs of nodes.
• Under a particular traffic pattern, the channel
  that carries the largest fraction of traffic
  determines the maximum channel load of the
  topology.

15

• Example topologies
• Regular or irregular
• Regular topologies are mostly derived from two
  main families butterflies (k-ary n-flies) or
  tori (k-ary n-cubes)

16

• Switching how a packet pass a switch
• Message/packet/flit

17

• Traditional scheme store-and-forward
• Time H (S P/B)

18

• Cut-through switch
• Forward to the next link after the header flit is
  received. Stop only when the next hop buffer is
  not available.
• Time H S P/B, when S ltlt P/B, the time does
  not depend on the number of hops!!!

19

• Wormhole routing
• Cut-through switches still allocate buffer to
  packets. May require a large amount of buffers
• Wormhole routing only allocates buffer for one
  flit for each packet.
• Latency is the same as cut-through switching.
• When the packet is block, the whole flit train
  is block, occupying links.
• Solution add more virtual channels.

20

• The deadlock problem in wormhole routing
• Need deadlock free routing scheme to select the
  right path

21

• Cut-through switch and wormhole switch are widely
  used in system are networks
• Routing in such systems is an issue!!
• Shortest path routing may result in deadlock.
• Deadlock free routing

22

• Cut-through switch and wormhole switch are widely
  used in system are networks
• Routing in such systems is an issue!!
• Shortest path routing may result in deadlock.
• Deadlock free routing
• Basic idea fix the priority of channels and
  using the channels with increasing priority.
• Example up/down routing

23

• Up/down routing
• Select a node as the root
• Build a spanning tree from the root
• Nodes are partitioned into layers based on the
  position in the spanning tree
• The channel from a lower layer node to a higher
  layer node is the up link, the channel from a
  higher layer node to a lower layer node is a down
  link, channels between nodes in the same layer
  are marked as up or down link based on the node
  number
• In the valid route an up channel cannot follow
  an down channel.
• These exists at least one valid path between each
  pair of nodes.

24

• Problems with deadlock free routing
• Load balancing is a problem, traffic are not
  evenly distributed
• Non-adaptive version of the deadlock free routing
  scheme is also a problem
• How to map the routes in order to get good
  performance (metrics maximum channel load?)
• More on the problem to be discussed later.

25

• Hardware/software codesign and software API
  issues
• What functionality should be implemented in the
  hardware.
• E.g. adaptive routing may imply out of order
  packets
• Chien04 paper gives good answers to some of
  these questions.