Introduction to Message Passing Interface MPI Part I

1
Introduction toMessage Passing Interface
(MPI)Part I

• SoCal Annual AAP workshop
• October 30, 2006
• San Diego Supercomputer Center

2
Overview

• MPI Background and Key Concepts
• Compilers and Include files for MPI
• Initialization
• Basic Communications in MPI
• Data types
• Summary of 6 basic MPI calls
• Example using basic MPI calls

3
Message Passing Interface Background

• MPI - Message Passing Interface
• Library standard defined by committee of vendors,
  implementers, and parallel programmers
• Used to create parallel SPMD programs based on
  message passing
• Available on almost all parallel machines in C
  and Fortran
• About 125 routines including advanced routines
• 6 basic routines

4
MPI Implementations

• Most parallel machine vendors have optimized
  versions
• Some other popular implementations include
• http//www-unix.mcs.anl.gov/mpi/mpich/
• http//www.lam-mpi.org/
• http//www.open-mpi.org/
• http//icl.cs.utk.edu/ftmpi/
• http//public.lanl.gov/lampi/

5
MPI Key Concepts

• Parallel SPMD programs based on message passing.
• Universal multiprocessor model that fits well on
  separate processors connected by fast/slow
  network.
• Normally the same program is running on several
  different nodes.
• Nodes communicate using message passing.
• MP allows a way for the programmer to explicitly
  associate specific data with processes and allows
  the compiler and cache management hardware to
  function fully.

6
MPI Compilers on SDSC machines

• DataStar Based on IBM xlf xlc compilers
• FORTRAN mpxlf_r, mpxlf90_r, mpxlf95_r
• C mpcc_r
• C mpCC_r
• Blue Gene Based on IBM xlf xlc compilers
• FORTRAN mpxlf, mpxlf90, mpxlf95
• C mpcc
• C mpCC
• TG IA-64 cluster Based on Intel ifort icc
  compilers
• FORTRAN mpif77, mpif90
• C mpicc
• C mpiCC

7
MPI Include files

• The MPI include file
• C mpi.h
• Fortran mpif.h (a f90 module is a good place
  for this)
• Defines many constants used within MPI programs
• In C defines the interfaces for the functions
• Compilers know where to find the include files

8
Initialization

• !ccccccccccccccccccccccccccccccccccccccccccccccccc
  ccccccccccccc!
• ! This code illustrates the use of some basic MPI
  routines !
• ! MPI_INIT Initialization of MPI
  !
• ! MPI_COMM_RANK Find the ID of given task
  !
• ! MPI_COMM_SIZE Find the total number of
  tasks !
• ! MPI_FINALIZE Close all MPI tasks
  !
• !ccccccccccccccccccccccccccccccccccccccccccccccccc
  ccccccccccccc!
• PROGRAM init
• IMPLICIT NONE
• INCLUDE 'mpif.h'
• INTEGER my_id, ntasks, ierr
• CALL MPI_INIT( ierr )
• CALL MPI_COMM_RANK( MPI_COMM_WORLD, my_id,
  ierr )
• CALL MPI_COMM_SIZE( MPI_COMM_WORLD, ntasks,
  ierr )
• WRITE(,) "I am task number", my_id,

9
Initialization

• COMPILING CODE
• ds100 mpxlf -o init initial.f
• init End of Compilation 1
• 1501-510 Compilation successful for file
  initial.f.
• SUBMIT JOB
• ds100 llsubmit LL.cmd
• Found valid account 'USE300' for queue 'express'
• Using ACL 'sdsc_datastarmahidharuse300sstrnp'
  
• on DataStar NPACI nodes all queues
• Job passed jobfilter
• llsubmit Processed command file through Submit
  Filter "/users00/loadl/loadl/jobfilter-interactiv
  e.pl".
• llsubmit The job "ds100.235768" has been
  submitted.
• CODE OUTPUT (in ouput file)
• 0 I am task number 0 . The total number of tasks
  is 4
• 1 I am task number 1 . The total number of tasks
  is 4
• 2 I am task number 2 . The total number of tasks
  is 4

10
Communicators

• Communicators
• A parameter for most MPI calls
• A collection of processors working on some part
  of a parallel job
• MPI_COMM_WORLD is defined in the MPI include file
  as all of the processors in your job
• Can create subsets of MPI_COMM_WORLD
• Processors within a communicator are assigned
  numbers 0 to n-1

11
Minimal MPI program

• Every MPI program needs these
• C version
• Commonly Used
• C version

include ltmpi.hgt / the mpi include file / /
Initialize MPI / ierrMPI_Init(argc, argv) /
How many total PEs are there / ierrMPI_Finalize(
)
ierrMPI_Comm_size(MPI_COMM_WORLD, nPEs) /
What node am I (what is my rank?)
/ ierrMPI_Comm_rank(MPI_COMM_WORLD, iam) ...
In C MPI routines are functions and return an
error value
12
Minimal MPI program

• Every MPI program needs these
• Fortran version

include 'mpif.h' ! MPI include file c
Initialize MPI call MPI_Init(ierr) c Find
total number of PEs call MPI_Comm_size(MPI_COMM_
WORLD, nPEs, ierr) c Find the rank of this
node call MPI_Comm_rank(MPI_COMM_WORLD, iam,
ierr) ... call MPI_Finalize(ierr)
In Fortran, MPI routines are subroutines, and
last parameter is an error value
13
Basic Communications in MPI

• Data values are transferred from one processor to
  another
• One process sends the data
• Another receives the data
• Standard, Blocking
• Call does not return until the message buffer is
  free to be reused
• Standard, Nonblocking
• Call indicates a start of send or received, and
  another call is made to determine if finished

14
Standard, Blocking Send

• MPI_Send Sends data to another processor
• Use MPI_Recv to "get" the data
• C
• MPI_Send(buffer,count,datatype,
  destination,tag,communicator)
• Fortran
• Call MPI_Send(buffer, count, datatype,destination,
  tag, communicator, ierr)
• Call blocks until message on the way

15
MPI_Send

• Call MPI_Send(buffer, count, datatype,
  destination, tag, communicator, ierr)
• Buffer The data
• Count Length of source array (in elements, 1
  for scalars)
• Datatype Type of data, for example
  MPI_DOUBLE_PRECISION, MPI_INT, etc
• Destination Processor number of destination
  processor in communicator
• Tag Message type (arbitrary integer)
• Communicator Your set of processors
• Ierr Error return (Fortran only)

16
Standard, Blocking Receive

• Call blocks until message is in buffer
• C
• MPI_Recv(buffer,count, datatype, source, tag,
  communicator, status)
• Fortran
• Call MPI_ RECV(buffer, count, datatype,
  source,tag,communicator, status, ierr)
• Status - contains information about incoming
  message
• C
• MPI_Status status
• Fortran
• Integer status(MPI_STATUS_SIZE)

17
Status

• In C
• status is a structure of type MPI_Status which
  contains three fields MPI_SOURCE, MPI_TAG, and
  MPI_ERROR
• status.MPI_SOURCE, status.MPI_TAG, and
  status.MPI_ERROR contain the source, tag, and
  error code respectively of the received message
• In Fortran
• status is an array of INTEGERS of length
  MPI_STATUS_SIZE, and the 3 constants MPI_SOURCE,
  MPI_TAG, MPI_ERROR are the indices of the entries
  that store the source, tag, error
• status(MPI_SOURCE), status(MPI_TAG),
  status(MPI_ERROR) contain respectively the
  source, the tag, and the error code of the
  received message.

18
MPI_Recv

• Call MPI_Recv(buffer, count, datatype, source,
  tag, communicator, status, ierr)
• Buffer The data
• Count Max. number of elements that can be
  received
• Datatype Type of data, for example
  MPI_DOUBLE_PRECISION, MPI_INT, etc
• Source Processor number of source processor in
  communicator
• Tag Message type (arbitrary integer)
• Communicator Your set of processors
• Status Information about message
• Ierr Error return (Fortran only)

19
Data types

• Data types
• When sending a message, it is given a data type
• Predefined types correspond to "normal" types
• MPI_REAL , MPI_FLOAT -Fortran and C real
• MPI_DOUBLE_PRECISION , MPI_DOUBLE - Fortan and C
  double
• MPI_INTEGER and MPI_INT - Fortran and C integer
• Can create user-defined types

20
Basic MPI Send and Receive

• A parallel program to send receive data
• Initialize MPI
• Have processor 0 send an integer to processor 1
• Have processor 1 receive an integer from
  processor 0
• Both processors print the data
• Quit MPI

21
Simple Send Receive Program

• include ltstdio.hgt
• include "mpi.h"
• /
  
• This is a simple send/receive program in MPI
• 
  /
• int main(argc,argv)
• int argc
• char argv
• int myid
• int tag,source,destination,count
• int buffer
• MPI_Status status
• MPI_Init(argc,argv)
• MPI_Comm_rank(MPI_COMM_WORLD,myid)

22
Simple Send Receive Program (cont.)

• tag1234
• source0
• destination1
• count1
• if(myid source)
• buffer5678
• MPI_Send(buffer,count,MPI_INT,destination,ta
  g,MPI_COMM_WORLD)
• printf("processor d sent
  d\n",myid,buffer)
• if(myid destination)
• MPI_Recv(buffer,count,MPI_INT,source,tag,MPI_COM
  M_WORLD,status)
• printf("processor d got
  d\n",myid,buffer)
• MPI_Finalize()

23
Simple Send Receive Program

• program send_recv
• include "mpif.h
• ! This is MPI send - recv program
• integer myid, ierr,numprocs
• integer tag,source,destination,count
• integer buffer
• integer status(MPI_STATUS_SIZE)
• call MPI_INIT( ierr )
• call MPI_COMM_RANK( MPI_COMM_WORLD, myid,
  ierr )
• call MPI_COMM_SIZE( MPI_COMM_WORLD,
  numprocs, ierr )
• tag1234
• source0
• destination1
• count1

24
Simple Send Receive Program (cont.)

• if (myid .eq. source) then
• buffer5678
• Call MPI_Send(buffer, count,
  MPI_INTEGER,destination,
• tag, MPI_COMM_WORLD, ierr)
• write(,)"processor ",myid," sent
  ",buffer
• endif
• if (myid .eq. destination) then
• Call MPI_Recv(buffer, count,
  MPI_INTEGER,source,
• tag, MPI_COMM_WORLD, status,ierr)
• write(,)"processor ",myid," got
  ",buffer
• endif
• call MPI_FINALIZE(ierr)
• stop
• end

25
Simple Send Receive Program

• Run the simple send receive code with 2
  processors.
• Output
• ds100 more LL_out.235771
• 0 processor 0 sent 5678
• 1 processor 1 got 5678

26
The 6 Basic MPI Calls

• MPI is used to create parallel programs based on
  message passing
• Usually the same program is run on multiple
  processors
• The 6 basic calls in MPI are
• MPI_INIT( ierr )
• MPI_COMM_RANK( MPI_COMM_WORLD, myid, ierr )
• MPI_COMM_SIZE( MPI_COMM_WORLD, numprocs, ierr )
• MPI_Send(buffer, count,MPI_INTEGER,destination,
  tag, MPI_COMM_WORLD, ierr)
• MPI_Recv(buffer, count, MPI_INTEGER,source,tag,
  MPI_COMM_WORLD, status,ierr)
• MPI_FINALIZE(ierr)

27
Example MPI program using basic routines

• MPI is used to create parallel programs based on
  message passing
• Usually the same program is run on multiple
  processors
• The 6 basic calls in MPI are
• MPI_INIT( ierr )
• MPI_COMM_RANK( MPI_COMM_WORLD, myid, ierr )
• MPI_COMM_SIZE( MPI_COMM_WORLD, numprocs, ierr )
• MPI_Send(buffer, count,MPI_INTEGER,destination,
  tag, MPI_COMM_WORLD, ierr)
• MPI_Recv(buffer, count, MPI_INTEGER,source,tag,
  MPI_COMM_WORLD, status,ierr)
• MPI_FINALIZE(ierr)

28
Simple Application using MPI 1-D Heat Equation

• ?T/?t a(?2T/?x2) T(0) 0 T(1) 0 (0 x1)
• T(x,0) is know as an initial condition.
• Discretizing for numerical solution we get
• T(n1)i T(n)i (a?t/?x2)(T(n)i-1-2T(n)iT(n
  )i1)
• (n is the index in time and i is the index in
  space)
• In this example we solve the problem using 11
  points and we distribute this problem over 3
  processors shown graphically below

29
Simple Application using MPI 1-D Heat Equation
Processor 0 Value at 0 known Get 4 from
processor 1 Solve the equation at (1,2,3) Send
3 to processor 1 Local Data Index ilocal 0 ,
1, 2, 3, 4 Global Data Index iglobal 0, 1, 2,
3, 4 Processor 1 Get 3 from processor 0 Get 7
from processor 2 Solve the equation at (4,5,6)
Send 4 to processor 0 and send 6 to processor
2 Local Data Index ilocal 0, 1, 2, 3,
4 Global Data Index iglobal 3, 4, 5, 6,
7 Processor 2 Get 6 from processor 1 Solve the
equation at (7,8,9) Value at 10 known Send 7 to
processor 1 Local Data Index ilocal 0, 1, 2,
3, 4 Global Data Index iglobal 6, 7, 8, 9, 10
30
FORTRAN CODE 1-D Heat Equation
Initial Conditions
pi
4d0datan(1d0) do ilocal 0, 4
iglobal 3my_idilocal T(0,ilocal)
dsin(pidelxdfloat(iglobal)) enddo
write(,)"Processor", my_id, "has finished
setting initial conditions"
Iterations
do itime 1 , 3 if (my_id.eq.0)
then write(,)"Running Iteration Number
", itime endif do ilocal 1, 3
T(itime,ilocal)T(itime-1,ilocal)
xalpdelt/delx/delx (T(itime-1,ilocal-1)
-2T(itime-1,ilocal)T(itime-1,ilocal1))
enddo if (my_id.eq.0) then
write(,)"Sending and receiving overlap points"
dest 1 msg_size 1
PROGRAM HEATEQN include "mpif.h"
implicit none integer iglobal, ilocal,
itime integer ierr, nnodes, my_id
integer IIM, IIK1, IIK2, IIK3, IIK4
integer dest, from, status(MPI_STATUS_SIZE),tag
integer msg_size real8
xalp,delx,delt,pi real8 T(0100,05),
TG(010) CHARACTER(20) FILEN delx
0.1d0 delt 1d-4 xalp 2.0d0
call MPI_INIT(ierr) call
MPI_COMM_SIZE(MPI_COMM_WORLD, nnodes, ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, my_id, ierr)
print , "Process ", my_id, "of", nnodes
,"has started"
31
Fortran Code 1-D Heat Equation (Contd.)
call MPI_SEND(T(itime,3),msg_size,MPI_DOUBL
E_PRECISION,dest,
tag,MPI_COMM_WORLD,ierr) endif if
(my_id.eq.1) then from 0 dest
2 msg_size 1 call
MPI_RECV(T(itime,0),msg_size,MPI_DOUBLE_PRECISION,
from, tag,MPI_COMM_WORLD,st
atus,ierr) call MPI_SEND(T(itime,3),msg_s
ize,MPI_DOUBLE_PRECISION,dest,
tag,MPI_COMM_WORLD,ierr) endif
if (my_id.eq.2) then from 1
dest 1 msg_size 1 call
MPI_RECV(T(itime,0),msg_size,MPI_DOUBLE_PRECISION,
from, tag,MPI_COMM_WORLD,st
atus,ierr) call MPI_SEND(T(itime,1),msg_s
ize,MPI_DOUBLE_PRECISION,dest,
tag,MPI_COMM_WORLD,ierr) endif
if (my_id.eq.1) then from 2
dest 0 msg_size 1
call MPI_RECV(T(itime,4),msg_size,MPI_DOUBLE_PREC
ISION,from,
tag,MPI_COMM_WORLD,status,ierr) call
MPI_SEND(T(itime,1),msg_size,MPI_DOUBLE_PRECISION,
dest, tag,MPI_COMM_WORLD,ie
rr) endif if (my_id.eq.0) then
from 1 msg_size 1 call
MPI_RECV(T(itime,4),msg_size,MPI_DOUBLE_PRECISION,
from, tag,MPI_COMM_WORLD,st
atus,ierr) endif enddo if
(my_id.eq.0) then write(,)"SOLUTION
SENT TO FILE AFTER 3 TIMESTEPS" endif
FILEN 'data'//char(my_id48)//'.dat'
open (5, fileFILEN) write(5,)"Processor
",my_id do ilocal 0 , 4 iglobal
3my_id ilocal write(5,)"ilocal",ilocal
,"iglobal",iglobal,"T",T(3,ilocal)
enddo close(5) call
MPI_FINALIZE(ierr) END
32
Simple Application using MPI 1-D Heat Equation

• Compilation
• Fortran mpxlf O3 heat-1d.f
• Running Interactively on DataStar
• Log on to dspoe.sdsc.edu
• poe a.out nodes 1 tasks_per_node 3

33
Simple Application using MPI 1-D Heat Equation
OUTPUT FROM SAMPLE PROGRAM Process 0 of 3 has
started Processor 0 has finished
setting initial conditions Process 1 of 3
has started Processor 1 has finished
setting initial conditions Process 2
of 3 has started Processor 2 has finished
setting initial conditions
Running Iteration Number 1 Sending and
receiving overlap points Running Iteration
Number 2 Sending and receiving overlap points
Running Iteration Number 3 Sending and
receiving overlap points SOLUTION SENT TO FILE
AFTER 3 TIMESTEPS
34
Simple Application using MPI 1-D Heat Equation
ds100 more data0.dat Processor 0 ilocal 0
iglobal 0 T 0.000000000000000000E00 ilocal
1 iglobal 1 T 0.307205621017284991 ilocal 2
iglobal 2 T 0.584339815421976549 ilocal 3
iglobal 3 T 0.804274757358271253 ilocal 4
iglobal 4 T 0.945481682332597884
ds100 more data2.dat Processor 2 ilocal 0
iglobal 6 T 0.945481682332597995 ilocal 1
iglobal 7 T 0.804274757358271253 ilocal 2
iglobal 8 T 0.584339815421976660 ilocal 3
iglobal 9 T 0.307205621017285102 ilocal 4
iglobal 10 T 0.000000000000000000E00
ds100 more data1.dat Processor 1 ilocal 0
iglobal 3 T 0.804274757358271253 ilocal 1
iglobal 4 T 0.945481682332597884 ilocal 2
iglobal 5 T 0.994138272681972301 ilocal 3
iglobal 6 T 0.945481682332597995 ilocal 4
iglobal 7 T 0.804274757358271253
35
References

• LLNL MPI tutorial
• http//www.llnl.gov/computing/tutorials/mpi/
• NERSC MPI tutorial
• http//www.nersc.gov/nusers/help/tutorials/mpi/int
  ro/
• LAM MPI tutorials
• http//www.lam-mpi.org/tutorials/
• Tutorial on MPI The Message-Passing Interface
• by William Gropp
• http//www-unix.mcs.anl.gov/mpi/tutorial/gropp/tal
  k.html