Parallel Processing

1
Parallel Processing

• Steve Terpe
• CS 147

2
Overview

• What is Parallel Processing
• Parallel Processing in Nature
• Parallel Processing vs. Multitasking
• Amdahls Law
• Challenges in Parallel Processing

3
What is Parallel Processing?

• How to make machines solve problems better and
  faster?
• Physical barriers limit the extent to which
  single processor performance can be
  improvedClock Speed Heat Dissipation
• The next most obvious solution is to distribute
  the computing load among several processors

4
What is Parallel Processing?

• Parallel Processing encompasses a wide variety of
  different things
• Intel Core Duo, Quad, Cell multiprocessors,
  networked and distributed computer systems,
  SETI_at_Home, Folding_at_Home, neural nets are all
  examples

5
Parallel Processing in Nature

• The worlds most powerful parallel processor
  comes standard
• For everything else theres American Express

6
Parallel Processing in Nature

• For humans parallel processing comes easy
• Human Vision
• Color, Motion,Depth, Shape

7
Parallel Processing in Nature

• Machines that are more like us, and hence more
  useful, will need to be able to process
  information more like usin parallel.
• Parallel Processing is key element in pattern
  recognition which distinguishes human machine
  intelligence
• This is not an easy hurdle

8
Parallel Processing vs. Multitasking

• Today, computers are great at multi-tasking, but
• Multi-tasking only creates the illusion of
  parallel processing
• The processor must switch between activitiesit
  is only the speed with which it does so that
  creates the illusion of simultaneous execution
• The illusion is most easily shattered when
  running virus scan while attempting to do
  anything else

9
Parallel Processing vs. Multitasking

• Think about it this way
• Multi-tasking
• Parallel Processing

10
Amdahls Law

• Consider a single processor
• Or two
• We tend to think that 2x as much work will be
  done in the same time
• Or that the same amount of work will be done in
  half the time

11
Amdahls Law

• Do n processors imply that a computational job
  should complete in 1/n time?
• Sadly, no

12
Amdahls Law

• 1967 Gene Amdahl recognizes the interrelationship
  of all components
• Overall speedup of a system depends on the
  speedup of a particular component how much that
  component is used
• S 1/(1- f f / k)
• S overall system speed
• f fraction of the work performed by the faster
  component
• k speedup of new component

13
Amdahls Law

• Additionally, no matter how well (or much) you
  parallelize an application there will always be a
  small portion of work that must be done serially.
• Other processors must simply sit and wait in this
  interval
• Every algorithm has a sequential part that limits
  potential speedup

14
Challenges in Parallel Processing

• Not always obvious where to split workload or
  even possible.
• If you dont use it, you lose itprograms not
  specifically written for parallel architecture
  run no more efficiently on parallel systems

15
Challenges in Parallel Processing

• Connecting your CPUs
• Dynamic vs Staticconnections can change from one
  communication to next
• Blocking vs Nonblockingcan simultaneous
  connections be present?
• Connections can be complete, linear, star, grid,
  tree, hypercube, etc.
• Bus-based routing
• Crossbar switchingimpractical for all but the
  most expensive super-computers
• 2X2 switchcan route inputs to different
  destinations

16
Challenges in Parallel Processing

• Dealing with memory
• Various options
• Global Shared Memory
• Distributed Shared Memory
• Global shared memory with separate cache for
  processors
• Potential Hazards
• Individual CPU caches or memories can become out
  of synch with each other. Cache Coherence
• Solutions
• UMA/NUMA machines
• Snoopy cache controllers
• Write-through protocols

17
Conclusion

• What is Parallel Processing
• Parallel Processing in Nature
• Parallel Processing vs. Multitasking
• Amdahls Law
• Challenges in Parallel Processing

18
Questions?