Parallel Computing

1
Parallel Computing

• Basic Concepts
• Computational Models
• Synchronous vs. Asynchronous
• The Flynn Taxonomy
• Shared versus Distributed Memory
• Interconnection networks
• Bus structures
• Static
• Dynamic
• Performance Issues
• Amdahl versus Gustafson-Barsis

2
Parallel Computing cont.

• Parallel Algorithms
• Languages for Parallel Computing
• PVM and MPI
• Memory Issues
• To be treated with memory in general

3
Basic Concepts

• Introduction
• Why? the von Neumann Bottleneck
• Problems the Grand Challenges
• The Gordon Bell Prize
• Common parallelism
• buses
• piplining
• Computational Models
• Synchronous vs Asychronous
• The Flynn Taxonomy

4
Flynn Taxonomy

• SISD
• superscalar architectures
• VLIW
• Vector processors
• SIMD
• fine grained parallelism
• generally many processors
• MISD
• not used
• MIMD
• coarse grained parallelism
• few processors

• Super-Flynn
• SPMD
• single program multiple data
• very common

5
Other Catagorizations

• Memory
• Shared versus Distributed
• tightly coupled versus loosely coupled
• multiprocessor versus multicomputer
• Interconnects
• Static versus Dynamic
• Hardwired versus switched

6

• Shared Memory
• Advantages
• Price/performance use off the shelf
  components
• Simplicity reliability
• Scalability
• Disadvantages
• Device contention
• Local cache coherency
• Topologies
• bus
• bus contention processes
• 1-D multiple buses
• crossbar switch
• multiport memory
• 2-D multiple buses
• hierarchical bus architecture
• NUMA non-uniform shared memory KSR1

7

• Static Interconnects
• Goals
• Fewest interconnects
• Fewest hops
• 1-D Linear
• 2-D Ring
• Grid / Mesh
• Toroid
• Fully Connected
• Tree, Quadtree
• Fat Tree
• Hypercube

• Dynamic Interconnects
• Crossbar
• Delta
• CLOS

8

• Shared Memory Machines
• Sequent Symmetry bus
• FX/8 bus
• Cray Y-MP multiport memory
• Distributed Memory Machines
• Intel Paragon 2-D grid
• Wavetracer 3-D grid
• MasPar MP-2 2-D torus and 3-stage CLOS
• nCUBE hypercube
• iPSC/2 hypercube
• CM - 2 2-D grid and 12-D hypercube
• CM-5 Fat Tree
• Kendall Square Research KSR1 rings
• Alliant Campus crossbars (multi)
• Paradigm bus(es) and interconnect

9
(No Transcript)