MPI Message Passing Interface Basics

1
MPI(Message Passing Interface)Basics
2
MPI(Message Passing Interface)

• Message passing library standard developed by
  group of academics and industrial partners to
  foster more widespread use and portability.
• Defines routines, not implementation.
• Several free implementations exist.

3

• MPI designed
• To address some problems with earlier
  message-passing system such as PVM.
• To provide powerful message-passing mechanism and
  routines - over 126 routines
• (although it is said that one can write
  reasonable MPI programs with just 6 MPI
  routines).

4
Unsafe Message PassingExample
Intended behavior
Process 0
Process 1
Destination
send(,1,)
send(,1,)
lib()
Source
recv(,0,)
lib()
recv(,0,)
5
Unsafe Message PassingExample
Possible behavior
Process 0
Process 1
Destination
send(,1,)
send(,1,)
lib()
Source
recv(,0,)
lib()
recv(,0,)
6

• Message tags not sufficient to deal with such
  situations -- library routines might use same
  tags.
• MPI introduces concept of a communicator -
  which defines the scope of communication.

7
MPI Communicators

• Defines a communication domain - a set of
  processes that are allowed to communicate between
  themselves.
• Communication domains of libraries can be
  separated from that of a user program.
• Used in all point-to-point and collective MPI
  message-passing communications.

8
Default Communicator MPI_COMM_WORLD

• Exists as first communicator for all processes
  existing in the application.
• A set of MPI routines exists for forming
  communicators.
• Processes have a rank in a communicator.

9
MPIProcess Creation and Execution

• Purposely not defined - Will depend upon
  implementation.
• Only static process creation supported in MPI
  version 1. All processes must be defined prior to
  execution and started together.
• Originally SPMD model of computation.

10
SPMD Computational Model

• main (int argc, char argv)
• MPI_Init(argc, argv)
• .
• .
• MPI_Comm_rank(MPI_COMM_WORLD, myrank) /get
  rank/
• if (myrank 0)
• master() / routine for master to execute /
• else
• slave() / routine for slaves to execute /
• .
• .
• MPI_Finalize()

11
MPI Point-to-Point Communication

• Uses send and receive routines with message tags
  (and communicator)
• Wild card message tags available

12
MPI Blocking Routines

• Return when locally complete
• when location used to hold message can be used
  again or altered without affecting message being
  sent.
• Blocking send will send message and return
• does not mean that message has been received,
  just that process free to move on without
  adversely affecting message.

13
Parameters of blocking send
14
Parameters of blocking receive
15
Example

• To send an integer x from process 0 to process 1,
• int x
• MPI_Comm_rank(MPI_COMM_WORLD,myrank) / find
  rank /
• if (myrank 0)
• MPI_Send(x, 1, MPI_INT, 1, msgtag,
  MPI_COMM_WORLD)
• else if (myrank 1)
• MPI_Recv(x,1,MPI_INT,0,msgtag,MPI_COMM_WORLD,sta
  tus)

16
MPI Nonblocking Routines

• Nonblocking send - MPI_Isend()
• will return immediately even before source
  location is safe to be altered.
• Nonblocking receive - MPI_Irecv()
• will return even if no message to accept.

17
Nonblocking Routine Formats

• MPI_Isend(buf,count,datatype,dest,tag,comm,request
  )
• MPI_Irecv(buf,count,datatype,source,tag,comm,
  request)
• Completion detected by MPI_Wait() and MPI_Test().
• MPI_Wait() waits until operation completed,
  returns then.
• MPI_Test() returns with flag set indicating
  whether operation completed at that time.
• Need to know whether particular operation
  completed.
• Determined by accessing request parameter.

18
Example

• To send an integer x from process 0 to process 1
  and allow process 0 to continue,
• MPI_Comm_rank(MPI_COMM_WORLD, myrank)/ find
  rank /
• if (myrank 0)
• MPI_Isend(x,1,MPI_INT,1,msgtag,MPI_COMM_WORLD,
  req1)
• compute()
• MPI_Wait(req1, status)
• else if (myrank 1)
• MPI_Recv(x,1,MPI_INT,0,msgtag,MPI_COMM_WORLD,sta
  tus)

19
Send Communication Modes

• Standard Mode Send - Not assumed that
  corresponding receive routine has started. Amount
  of buffering not defined by MPI. If buffering
  provided, send could complete before receive
  reached.
• Buffered Mode - Send may start and return before
  a matching receive. Necessary to specify buffer
  space via routine MPI_Buffer_attach().
• Synchronous Mode - Send and receive can start
  before each other but can only complete together.
• Ready Mode - Send can only start if matching
  receive already reached, otherwise error. Use
  with care.

20

• Each of the four modes can be applied to both
  blocking and nonblocking send routines.
• Only the standard mode is available for the
  blocking and nonblocking receive routines.
• Any type of send routine can be used with any
  type of receive routine.

21
Collective Communication

• Involves set of processes, defined by an
  intra-communicator. Message tags not present.
  Principal collective operations
• MPI_Bcast() - Broadcast from root to all other
  processes
• MPI_Gather() - Gather values for group of
  processes
• MPI_Scatter() - Scatters buffer in parts to group
  of processes
• MPI_Alltoall() - Sends data from all processes to
  all processes
• MPI_Reduce() - Combine values on all processes to
  single value
• MPI_Reduce_scatter() - Combine values and scatter
  results
• MPI_Scan() - Compute prefix reductions of data on
  processes

22
Example

• To gather items from group of processes into
  process 0, using dynamically allocated memory in
  root process
• int data10 /data to be gathered/
• MPI_Comm_rank(MPI_COMM_WORLD, myrank) / find
  rank /
• if (myrank 0)
• MPI_Comm_size(MPI_COMM_WORLD, grp_size) /find
  group size/
• buf (int )malloc(grp_size10sizeof(int))
  /allocate memory/
• MPI_Gather(data,10,MPI_INT,buf,grp_size10,MPI_INT
  ,0,MPI_COMM_WORLD)
• MPI_Gather() gathers from all processes,
  including root.

23
Barrier

• As in all message-passing systems, MPI provides a
  means of synchronizing processes by stopping each
  one until they all have reached a specific
  barrier routine.

24
Barrier Concept
25
Using Library Routines
26
(No Transcript)
27
Measuring Execution Time

• To measure execution time between point L1 and
  point L2 in the code, might have
• .
• L1 time(t1) / start timer /
• .
• .
• L2 time(t2) / stop timer /
• .
• elapsed_t difftime(t2, t1) /t2 - t1/
• printf(Elapsed time 5.2f,elapsed_t)
• MPI provides MPI_Wtime() for returning time (in
  seconds).

28
Executing MPI programs

• MPI version 1 standard does not address
  implementation and did not specify how programs
  are to be started and each implementation has its
  own way.

29

• Several MPI implementations, such as MPICH and
  LAM MPI, use command
• mpirun -np prog
• where is the number of processes and prog is
  the program.
• Additional argument specify computers (see
  later).

30
MPI-2

• The MPI standard, version 2 does recommend a
  command for starting MPI programs, namely
• mpiexec -n prog
• where is the number of processes and prog is
  the program.

31
Sample MPI Programs
32
Hello World

• include "mpi.h"
• include ltstdio.hgt
• int main(int argc,char argv)
• MPI_Init(argc, argv)
• printf("Hello World\n")
• MPI_Finalize()
• return 0

33
Hello WorldPrinting out rank of process

• include "mpi.h"
• include ltstdio.hgt
• int main(int argc,char argv)
• int myrank, numprocs
• MPI_Init(argc, argv)
• MPI_Comm_rank(MPI_COMM_WORLD,myrank)
• MPI_Comm_size(MPI_COMM_WORLD,numprocs)
• printf("Hello World from process d of d\n",
  myrank, numprocs)
• MPI_Finalize()
• return 0

34
Question

• Suppose this program is compiled as helloworld
  and is executed on a single computer with the
  command
• mpirun -np 4 helloworld
• What would the output be?

35
Answer

• Several possible outputs depending upon order
  processes are executed.
• Example
• Hello World from process 2 of 4
• Hello World from process 0 of 4
• Hello World form process 1 of 4
• Hello World form process 3 of 4

36

• Adding communication to get process 0 to print
  all messages
• include "mpi.h"
• include ltstdio.hgt
• int main(int argc,char argv)
• int myrank, numprocs
• char greeting80 / message sent from
  slaves to master /
• MPI_Status status
• MPI_Init(argc, argv)
• MPI_Comm_rank(MPI_COMM_WORLD,myrank)
• MPI_Comm_size(MPI_COMM_WORLD,numprocs)
• sprintf(greeting,"Hello World from process d of
  d\n",rank,size)
• if (myrank 0 ) / I am going print
  out everything /
• printf("s\n",greeting) / print
  greeting from proc 0 /
• for (i 1 i lt numprocs i) /
  greetings in order /
• MPI_Recv(geeting,sizeof(greeting),MPI_CHAR,i,1,M
  PI_COMM_WORLD,
• status)
• printf(s\n", greeting)
• else

37
MPI_Get_processor_name()

• Return name of processor executing code (and
  length of string). Arguments
• MPI_Get_processor_name(char name,int resultlen)
• Example
• int namelen
• char procnameMPI_MAX_PROCESSOR_NAME
• MPI_Get_processor_name(procname,namelen)

returned in here
38

• Easy then to add name in greeting with
• sprintf(greeting,"Hello World from process d of
  d on s\n", rank, size, procname)

39
Pinging processes and timingMaster-slave
structure

• include ltmpi.hgt
• void master(void)
• void slave(void)
• int main(int argc, char argv)
• int myrank
• printf("This is my ping program\n")
• MPI_Init(argc, argv)
• MPI_Comm_rank(MPI_COMM_WORLD, myrank)
• if (myrank 0)
• master()
• else
• slave()
• MPI_Finalize()
• return 0

40
Master routine

• void master(void)
• int x 9
• double starttime, endtime
• MPI_Status status
• printf("I am the master - Send me a message when
  you receive this number d\n", x)
• starttime MPI_Wtime()
• MPI_Send(x,1,MPI_INT,1,1,MPI_COMM_WORLD)
• MPI_Recv(x,1,MPI_INT,1,1,MPI_COMM_WORLD,status)
  
• endtime MPI_Wtime()
• printf("I am the master. I got this back d \n",
  x)
• printf("That took f seconds\n",endtime -
  starttime)

41
Slave routine

• void slave(void)
• int x
• MPI_Status status
• printf("I am the slave - working\n")
• MPI_Recv(x,1,MPI_INT,0,1,MPI_COMM_WORLD,status)
  
• printf("I am the slave. I got this d \n", x)
• MPI_Send(x, 1, MPI_INT, 0, 1, MPI_COMM_WORLD)

42
Example using collective routinesMPI_Bcast()MPI
_Reduce()

• Adding numbers in a file.

43

• include mpi.h
• include ltstdio.hgt
• include ltmath.hgt
• define MAXSIZE 1000
• void main(int argc, char argv)
• int myid, numprocs
• int dataMAXSIZE, i, x, low, high, myresult,
  result
• char fn255
• char fp
• MPI_Init(argc,argv)
• MPI_Comm_size(MPI_COMM_WORLD,numprocs)
• MPI_Comm_rank(MPI_COMM_WORLD,myid)
• if (myid 0) / Open input file and
  initialize data /
• strcpy(fn,getenv(HOME))
• strcat(fn,/MPI/rand_data.txt)
• if ((fp fopen(fn,r)) NULL)
• printf(Cant open the input file s\n\n,
  fn)
• exit(1)

44
Compiling/Executing MPI ProgramsPreliminaries

• Set up paths
• Create required directory structure
• Create a file listing machines to be used
  (hostfile)

45

• Before starting MPI for the first time, need to
  create a hostfile
• Sample hostfile
• terra
• venus //Currently not used, commented out
• leo1
• leo2
• leo3
• leo4
• leo5
• leo6
• leo7
• leo8

46
Compiling/executing (SPMD) MPI program

• For LAM MPI version 6.5.2. At a command line
• To start MPI
• First time lamboot -v hostfile
• Subsequently lamboot
• To compile MPI programs
• mpicc -o file file.c
• or mpiCC -o file file.cpp
• To execute MPI program
• mpirun -v -np no_processors file
• To remove processes for reboot
• lamclean -v
• Terminate LAM
• lamhalt
• If fails wipe -v lamhost

47
Compiling/Executing Multiple MPI Programs

• Create a file, say called appfile, specifying
  programs
• Example
• 1 master and 2 slaves, appfile contains
• n0 master
• n0-1 slave
• To execute
• mpirun -v appfile
• Sample output
• 3292 master running on n0 (o)
• 3296 slave running on n0 (o)
• 412 slave running on n1

48
Parallel Programming Home Page

• http//www.cs.uncc.edu/par_prog
• Gives step-by-step instructions for compiling and
  executing programs, and other information.