Parallel Programming Models

1
Parallel Programming Models

• Introduction
• Parallel applications
• Parallel Models Tools
• Design of parallel algorithms
• Parallel programming paradigms
• Programming skeletons Templates
• Conclusions

2
Introduction

• Levels of parallelsim-Tasks- threads- data flow-
  multiple processor issues
• Implicit explicit-user does not specify the
  scheduling of calculations or the placement of
  data- User responsible of task decomposition,
  mapping tasks to processors, communication
  structure.
• Parallelizing compilers, parallel langs, message
  passing libs, DSM, object oriented prog,
  programming skeletons.
• Problems non-determinism- communication,
  synchronization- data partition distribution-
  load balancing- fault tolerance- heterogeneity-
  shared or distributed memory- deadlocks, and race
  conditions.
• Models for developing parallel applications
  shared memory model- distributed memory model
  (message passing)- and DSM model.

3
Parallel programming models

• Parallel langs SISAL, PCN-users are not willing
  to learn new langs. Use their langs C, C,
  Java, C, etc.
• Message passing libs for messaging environments-
  PVM, MPI.sys prog.
• VSM, parallel object programming, and
  programming skeletons.
• VSM Virtual shared memory- shared memory model-
  Linda- virtual shared associative memory-tuple
  space, in, out.
• DSMflat memory- read, write- replicated for
  performance.
• Parallel object computing- a la CORBA, DCOM.
  CC, and Mentat parallel obj oriented
  programming.
• Skeletons higher level templates. They hid from
  the user the specific details of the
  implementation. Paramaterized scripts- Skeleton
  Oriented programming (SOP). Can be implemented on
  top of message passing, OO, shared memory, or
  DSM. A skeleton corresponds to the instantiation
  of a specific // programming paradigm. The
  programmer is to provide those routines unique
  for the application.

4
Design of parallel algorithms

• Partitioning- communication-agglomeration-
  mapping.
• Partitioning Domain/data decomposition, and
  functional/computational decomposition.
• Communication Flow of info and coordination
  among tasks local/global, structured/unstructured
  , static/dynamic, and synchronous/asynchronous.
• Agglomeration if required, tasks are grouped
  into larger tasks to improve performance.Increase
  computation/communication granularity.
• Mapping assigning tasks to processors. Static
  (compile time) or dynamic (at run time).

5
Parallel programming paradigms

• Functional parallelism from the structure of the
  application. Type of parallelism inherent in the
  problem.
• Parallelism in the structure of data.
• Clinet/server RPC paradigm is not in parallel
  computing.
• Processor farms replication of independent jobs.
• Geometric decomposition parallelism of data
  structures.
• Algorithm parallelism which results in the use
  of data flow.
• Pipelining, divide and conquer, and master/slave.

6
Choice of paradigms

• Geometric decomposition the problem domain is
  broken up into smaller domains. Processes
  domains.
• Iterative decomposition loop and recursive
  unfolding. Queue of runnable tasks. Corresponds
  to task-farming paradigm.
• Recursive decomposition subproblems are similar
  to the original problem. Solve them in parallel.
  Divide and conquer.
• Speculative decomposition N solution techniques
  are tried simultanously, and N-1 are thrown away
  as soon as the first one returns a solution.
• Functional decomposition phases Pipelining.
• Classification based on temporal structure of the
  problems
• Synchronous regular computations on regular data
  domains.
• Loosely syn problems iterative calculations on
  geometrically irregular data domains.
• Asynch problems functional parallelism that is
  irregular in space and time. Event driven
  simulation.
• Embarrassingly parallel applications indep
  execution of disconnected components of the same
  program. Master/slave model.

7
Choice of paradigms

• Analysis of 84 real applications. 76 of 1 2,
  10 of 3, and 14 of 4 (master slave).
• Task Farming Master slave.
• Multiple salves. The master decompose to tasks
  and distribute to slaves, then gather partial
  results.
• Slave get a message with the task, process the
  task, send the result to master.
• Static load balancing known fixed no of slaves,
  and the master participate in computations.
• Dynamic load balancing no of tasks exceeds no of
  processors. Or unpredictable execution times.
• Master can become a bottleneck. Set of masters.
  Each controlling a different group of slaves.

8
SPMD

• Single program multiple data Same piece of code
  on different part of data. Geometric parallelism
  domain parallelism data parallelism.
• Processors communicate with neighbouring
  processors.
• May need to perform some global synchronization
  periodically.
• The communication pattern is highly structured.
• The data may initially be self-generated by each
  process.
• Highly sensitive to the loss of some process.
  Deadlock- none of the processes can advance
  beyond a global synchronization point.

9
Pipelining

• Functional decomposition. Tasks of the algorithm.
  Concurrent execution.
• Each process corresponds to a stage of the
  pipeline.
• The communication can be asynchronous.
• Provide multiple independent paths across the
  stages. To be robust.
• Data flow parallelism.

10
Divide and Conquer

• A problem is divided up into two or more
  subproblems. Each is solved independently and
  their results are combined to give the final
  result.
• Three generic computational operations split,
  compute, join.
• Task farming is a modified divide and conquer.
  Only the master does the split join.
• Tasks may be generated at runtime and may be
  added to a job queue.Or several job queues.

11
Decomposition Distribution

• Geometric parallelism static static.
• Function decomposition- pipelining static
  static.
• Task farming static dynamic.
• Divide and Conquer dynamic dynamic.

12
Speculative parallelism

• Problem complex data dependencies. Discrete
  event simulation. Asynchronous problem.
• Employ different algorithms for the same problem
  the first one to give the final solution is the
  one that is chosen.

13
Hybrid models

• Mix data and task parallelisms simultaneously or
  in different parts of the same program.
• Skeletons incomplete parameterized, not just by
  the no of processors, but by other factors such
  as granularity, topology, and data distribution.
• Package of parallel utilities (PUL) on top pf
  MPI. Part for parallel I/O
• Part for task farming (PUL-TF), regular domain
  decomposition (PUL-RD), irregular mesh based
  (PUL-SM).

14
Skeletons Templates

• ARNIA package Lib for master/slave applications.
  
• Another for domain decomposition paradigm.
• Another for global shared memory emulation.
• TINA skeleton generator. Reusability.
• TRCS reusable components, such as frams, grids,
  and pipes.
• Programmers are able to develop parts of programs
  simply by filling in the templates.
• Hide lower level details.
• Reusability.
• Portability.
• Efficiency- Flexibility- Productivity.

15
Other Reference Book