The Truth About Parallel Computing: Fantasy versus Reality

1
The Truth About Parallel ComputingFantasy
versus Reality

• William M. Jones, PhD
• Computer Science Department
• Coastal Carolina University

2
What is Parallel Computing ?

• Simply put
• Parallel computing is the simultaneous use of
  multiple compute resources to solve a
  computational problem.
• sounds simple

3
What typifies these computational problems ?

• Can be broken apart into discrete pieces of work
  that can be solved simultaneously
• Can be solved in less time with multiple compute
  resources than with a single compute resource
• however
• Problem decomposition can be exceedingly complex
• Ultimate performance depends on a rather large
  number of interacting factors

4
Some Typical ArchitecturesShared Memory

• Multiple CPU's with global address space
• Each CPU can work on a part of the problem as
  the same time
• Data sharing takes place in main-memory

5
Some Typical ArchitecturesDistributed Memory

• Multiple computers with local address space
• Each computer can work on a part of the problem
  as the same time
• Data sharing takes place by sending messages
  across the network

6
Example Programming ModelData Parallel

• Large data-set
• Partitioned across data
• Each task works on it's part of the data
• Suppose data needs to be exchanged at runtime
• Communication may be necessary

7
Program Granularity

• Communication / Computation Ratio
• Coarse-grain
• Fine-grain
• Communication can be a dramatic bottleneck
• Optimization

8
More Bad News Theoretical Limits to Parallel
Program Performance
9
Amdahl's Law, Continued
10
Conclusion

• Parallel computing can help
• Problem decomposition is often difficult
• Often, data exchange is necessary, thus
  communication is necessary
• Communication is often a bottleneck
• Even if no communication is necessary, ultimate
  performance is limited to fraction of program
  that is parallelizable
• Questions ?