Distributed Memory and Cluster Computers

1
Distributed Memory and Cluster Computers

• Alistair Rendell
• (Adapted From Parallel Programming, Techniques
  and Applications using Networked Workstations and
  Parallel Computers, B. Wilkinson and M. Allen,
  Prentice Hall)

2
Message-Passing Multicomputer

• All communication by messages sent over the
  interconnection network
• More scalable than shared memory parallel
  computers
• (The boundary between distributed and shared
  memory systems is not clear cut)

3
Interconnection Networks

• Key characteristics
• latency, bandwidth, bisection bandwidth,
  diameter, cost
• Static
• exhaustive interconnections
• 2- and 3-dimensional meshes
• hypercubes
• Dynamic
• Crossbar
• Trees
• Multistage interconnection networks

4
Two-dimensional array (mesh)

• Also torus and 3-dimensional - used in some large
  high performance systems.

5
Three-dimensional hypercube
6
Four-dimensional hypercube

• Hypercubes popular in 1980s - not now

7
Crossbar switch
8
Tree

• Whats a FAT tree?

9
Multistage Interconnection NetworkExample Omega
network
10
Communication Methods

• Switching options (no direct link)
• Circuit switching (Intel IPSC-2)
• Packet switching (everything else)
• Store-and-forward
• message entirely received by intervening nodes
• Wormhole routing
• Message divided into flits that are pipelined
  through the communication network
• Care to avoid possible deadlock and/or livelock
  scenarios

11
Networked Computers as a Computing Platform

• A network of computers became a very attractive
  alternative to expensive supercomputers and
  parallel computer systems for high-performance
  computing in early 1990s.
• Several early projects. Notable
• Berkeley NOW (network of workstations) project.
  
• NASA Beowulf project. (Will look at this one
  later)

12
Key Drivers

• Amazing increase in performance of commercial off
  the shelf PCs at very low cost
• Availability of commodity high performance
  networking
• The GPL software revolution
• Advantages
• The latest processors can easily be incorporated
  into the system as they become available.
• Existing software can be used or modified.

13
Software Tools for Clusters

• Based upon Message Passing Parallel Programming
• Parallel Virtual Machine (PVM) - developed in
  late 1980s. Became very popular.
• Message-Passing Interface (MPI) - standard
  defined in 1990s.
• Both provide a set of user-level libraries for
  message passing. Use with regular programming
  languages (C, C, ...).

14
Beowulf Clusters

• A group of interconnected commodity computers
  achieving high performance with low cost.
• Typically using commodity interconnects - high
  speed Ethernet, and Linux OS.
• Beowulf comes from name given by NASA Goddard
  Space Flight Center cluster project.

15
Ethernet Interconnects
Frame check sequence (32 bits)
Data (lt1.5KB)
Type (16 bits)
Source Address (48bits)
Destination Address (48bits)
Preamble (64bits)
direction

• 48bits are the MAC address
• Data lt1.5KB takes one packet
• Collision protocol not ideal for HPC

16
Cluster Interconnects

• Beyond Ethernet
• Myrinet - 2.4 Gbits/sec - disadvantage single
  vendor
• cLan
• SCI (Scalable Coherent Interface)
• QNet (Quadrics)
• Infiniband - may be important as infininband
  interfaces may be integrated on next generation
  PCs
• Multiple parallel interconnects
• Channel bonding (effect on bandwidth and latency?)

17
Dedicated cluster with a master node