Parallel Processing

1
Parallel Processing

• Parallel Processing occurs when more than one
  processor works together to solve a single
  problem.
• Your lab has a lot of PC's but that doesnt mean
  parallel processing is going on. Each separate
  PC is working on solving separate problems.
• Parallel processing speeds up solution times in
  two ways
• Multiple computers simply add "horsepower" to an
  existing algorithm
• New algorithms can be designed to actually lower
  the cost of the problem solution.
• Two kinds of parallelism
• Data parallelism when the data is divided up
  among processors
• Task parallelism When the parts of an algorithm
  are divided up.

2
Types of Parallel Processing

• Pipelining (not usually considered true
  parallelism)
• This occurs when separate processers, each with a
  specific job, work on a phase of a problems
  solution.
• Most common example is your PC. The "single"
  processor is actually made up of many smaller
  special purpose processors.

Data 5
Pipelining allows a result to appear at each
step.
Normal processing requires 3 steps to produce
each result.
Data 4
Data 5
Data 3
Data 4
Data 3
Data 2
Data 2
Result 1
Result 1
3
Types of Parallel Processing

• Parallel Processing
• Separate processors, usually identical work with
  one memory.
• Usually requires special purpose machines. These
  machines may contain thousands of simple
  processors (usually in a multiple of 2, like 4096
  or 8192, etc)

P01
P04
P05
P02
P03
P06
P07
MEMORY
4
Types of Parallel Processing

• Distributed Processing
• Separate processors of any type each with their
  own memory.
• Networks or the internet. This is the most
  general and flexible type of processing, but
  communication costs between processors may be
  high.

P01
P02
P03
P04
P05
P06
P07
M02
M03
M04
M05
M06
M07
M01
COMMUNICATION NETWORK
5
Stages of Parallel Computation

• Everything has a benefit and a cost - parallel
  processing has an overhead cost.
• The goal is to let the benefit of multiple
  processors working on a single problem outweigh
  the cost of keeping track of them.
• Stages of a parallel computation
• (cost) Division of problem/data and distribution
  to processors
• (cost) Start up of remote processes
• (benefit) Parallel Computation
• (cost) Transfer local results to a common
  processor
• (cost) Collate results and present

6
Examples

• Assume problem size n and number of processors
  p.
• Image processing Cost goes from O(n) to O(n/p).
• FindMax Cost goes from O(n) to O(log n).
• Matrix squaring Cost goes from O(n4) to O(n2).
• Sorting cost goes from O(n logn) to O(n).