Message Passing Computing

1
Message Passing Computing

• Outline
• Basic Message Passing Programming
• Using Workstation Clusters
• Evaluating Parallel Programs
• Debugging and Evaluating Parallel Programs

2
Basic Message Passing Programming

• Three approaches to message passing computing
• Innovatively create a pure message-passing
  programming language (occam)
• Extending the syntax/reserved words of an
  existing sequential high-level language to handle
  message passing (CC and Fortran M)
• Using an existing sequential high-level language
  and providing a library of external procedures
  for message passing (commonly used)
• Parallelizing compilers or converting tools

3
Basic Message Passing Programming

• Using an existing sequential high-level language
  and providing a library of external procedures
  for message passing
• There are two primary methods in this approach
• Creating separate processes for execution on
  different computers
• Sending and receiving messages

4
Process Creation

• Review the concept of process
• Process id defined as a part of computational
  task.
• It is different concept than processor
• If one tests a program on different number of
  processors, more than one process may be mapped
  onto a single processor.
• In general, each process is mapped to each
  individual processor.
• It could be one processes mapped onto a single
  processor (testing by hiding network latencies)

5
process
processors
6
Multiple Processes in a program
Single processor
Not practical but can be used to test the case in
which network latency is hidden.
7
Process Creation

• Two methods of creating processes
• Static process creation
• All the processes are specified before execution
  and system will execute a fixed number of
  processes
• Programmer usually explicitly identifies the
  processes or programs prior to execution by
  command line actions
• Each process usually executes different part of
  task.
• Usually there is one controlling process, called
  master or manager processor. Other processes
  are declared as worker process.

8
Process Creation

• Two methods of creating processes
• Static process creation
• Single program multiple data (SPMD)
• Different processes (tasks) are merged into one
  program..
• Within the program, a programmer can introduce
  controlling statements (or branch statements) to
  assign different execution jobs to different
  processes.
• Each process will map to each corresponding
  processor
• This mapping can be implicit or explicit
• The program is compiled into executable code fro
  each processors
• Each process will load a copy of this code into
  its local memory for execution

9
Source file (single program)
compiling
Process 0
Process 1
mapping
Executable code
Executable code
memory
memory
Processor 0
Processor 1
10
Process Creation

• Two methods of creating processes
• Dynamic process creation
• Process can be crated and their execution
  initiated during the execution of their processes
• Processes-creation constructs or library/system
  calls are used to create processes.
• Processes can also be destroyed during execution
• Dynamic process creation is a more powerful
  technique than state process creation
• Most general model for dynamic process creation
  is the multiple program multiple data model (MPMD)

11
Process Creation

• Two methods of creating processes
• MPMD
• A complete separate and different program is
  written fro different processors
• Single processor executes a master program
  (master process) and other processes are started
  from within the master process.
• H3S (Jun Ni and Lili Huang, 2001) system actually
  utilizes such approach.
• Use spawn(number_of_process)
• PVM uses this approach

12
Process Creation

• Message-passing routines
• Basic Send and Receive routines