MPI: Message Passing Interface

1
MPI Message Passing Interface

• Prabhaker Mateti
• Wright State University

2
Overview

• MPI Hello World!
• Introduction to programming with MPI
• MPI library calls

3
MPI Overview

• Similar to PVM
• Network of Heterogeneous Machines
• Multiple implementations
• Open source
• MPICH
• LAM
• Vendor specific

4
MPI Features

• Rigorously specified standard
• Portable source code
• Enables third party libraries
• Derived data types to minimize overhead
• Process topologies for efficiency on MPP
• Can fully overlap communication
• Extensive group communication

5
MPI 2

• Dynamic Process Management
• One-Sided Communication
• Extended Collective Operations
• External Interfaces
• Parallel I/O
• Language Bindings (C and Fortran-90)
• http//www.mpi-forum.org/

6
MPI Overview

• 125 functions
• typical applications need only about 6

7
MPI managerworkers

• include ltmpi.hgt
• main(int argc, char argv)
• int myrank
• MPI_Init(argc, argv)
• MPI_Comm_rank(MPI_COMM_WORLD, myrank)
• if (myrank 0) manager()
• else worker()
• MPI_Finalize()

• MPI_Init initializes the MPI system
• MPI_Finalize called last by all processes
• MPI_Comm_rank identifies a process by its rank
• MPI_COMM_WORLD is the group that this process
  belongs to

8
MPI manager()

• manager()
• MPI_Status status
• MPI_Comm_size(
• MPI_COMM_WORLD, ntasks)
• for (i 1i lt ntasksi)
• work nextWork() MPI_Send(work, 1,
  MPI_INT,i,WORKTAG, MPI_COMM_WORLD)
• MPI_Reduce(sub, pi, 1, MPI_DOUBLE, MPI_SUM,
  0, MPI_COMM_WORLD)

• MPI_Comm_size
• MPI_Send

9
MPI worker()

• worker()
• MPI_Status status
• for ()
• MPI_Recv(work, 1, MPI_INT, 0, MPI_ANY_TAG,
• MPI_COMM_WORLD, status)
• result doWork()
• MPI_Send(result, 1, MPI_DOUBLE, 0, 0,
  MPI_COMM_WORLD)

• MPI_Recv

10
MPI computes ?

• include "mpi.h"
• int main(int argc, char argv)
• MPI_Init(argc,argv)
• MPI_Comm_size(MPI_COMM_WORLD,np)
• MPI_Comm_rank(MPI_COMM_WORLD,myid)
• n ... / intervals /
• MPI_Bcast(n, 1, MPI_INT, 0, MPI_COMM_WORLD)
• sub series_sum(n, np)
• MPI_Reduce(sub, pi, 1, MPI_DOUBLE, MPI_SUM,
  0, MPI_COMM_WORLD)
• if (myid 0)
• printf("pi is .16f\n", pi)
• MPI_Finalize()
• return 0

11
Process groups

• Group membership is static.
• There are no race conditions caused by processes
  independently entering and leaving a group.
• New group formation is collective and group
  membership information is distributed, not
  centralized.

12
MPI_Send blocking send

• MPI_Send(
• sendbuffer, / message buffer /
• n, / n items of /
• MPI_type, / data type in message /
• destination, / process rank / WORKTAG, /
  user chosen tag / MPI_COMM / group /
• )

13
MPI_Recv blocking receive

• MPI_Recv(
• recvbuffer, / message buffer /
• n, / n data items /
• MPI_type, / of type /
• MPI_ANY_SOURCE, / from any sender /
• MPI_ANY_TAG, / any type of message /
• MPI_COMM, / group /
• status
• )

14
Send-receive succeeds

• Senders destination is a valid process rank
• Receiver specified a valid source process
• Communicator is the same for both
• Tags match
• Message data types match
• Receivers buffer is large enough

15
Message Order

• P sends messages m1 first then m2 to Q
• Q will receive m1 before m2
• P sends m1 to Q, then m2 to R
• In terms of a global wall clock, conclude nothing
  re R receiving m2 before/after Q receiving m1.

16
Blocking and Non-blocking

• Send, receive can be blocking or not
• A blocking send can be coupled with a
  non-blocking receive, and vice-versa
• Non-blocking send can use
• Standard mode MPI_Isend
• Synchronous mode MPI_Issend
• Buffered mode MPI_Ibsend
• Ready mode MPI_Irsend

17
MPI_Isend non-blocking

• MPI_Isend(
• buffer, / message buffer /
• n, / n items of /
• MPI_type, / data type in message /
• destination, / process rank / WORKTAG, /
  user chosen tag / MPI_COMM, / group /
• handle
• )

18
MPI_Irecv

• MPI_Irecv(
• result, / message buffer /
• n, / n data items /
• MPI_type, / of type /
• MPI_ANY_SOURCE, / from any sender /
• MPI_ANY_TAG, / any type of message /
• MPI_COMM_WORLD, / group /
• handle
• )

19
MPI_Wait

• MPI_Wait(
• handle,
• status
• )

20
MPI_Wait, MPI_Test

• MPI_Wait(
• handle,
• status
• )

• MPI_Test(
• handle,
• status
• )

21
Collective Communication
22
MPI_Bcast

• MPI_Bcast(
• buffer,
• count, MPI_Datatype, root,
• MPI_Comm
• )

• All processes use the same count, data type,
  root, and communicator. Before the operation, the
  roots buffer contains a message. After the
  operation, all buffers contain the message from
  the root

23
MPI_Scatter

• MPI_Scatter(
• sendbuffer,
• sendcount, MPI_Datatype,
• recvbuffer,
• recvcount, MPI_Datatype,
• root,
• MPI_Comm)

• All processes use the same send and receive
  counts, data types, root and communicator. Before
  the operation, the roots send buffer contains a
  message of length sendcount N', where N is the
  number of processes. After the operation, the
  message is divided equally and dispersed to all
  processes (including the root) following rank
  order.

24
MPI_Gather

• MPI_Gather(
• sendbuffer,
• sendcount, MPI_Datatype, recvbuffer,
• recvcount, MPI_Datatype, root,
• MPI_Comm)

• This is the reverse of MPI_Scatter(). After the
  operation the root process has in its receive
  buffer the concatenation of the send buffers of
  all processes (including its own), with a total
  message length of recvcount N, where N is the
  number of processes. The message is gathered
  following rank order.

25
MPI_Reduce

• MPI_Reduce(
• sndbuf,
• rcvbuf,
• count, MPI_Datatype datatype, MPI_Op,
• root,
• MPI_Comm)

• After the operation, the root process has in
  its receive buffer the result of the pair-wise
  reduction of the send buffers of all processes,
  including its own.

26
Predefined Reduction Ops

• MPI_MAX
• MPI_MIN
• MPI_SUM
• MPI_PROD
• MPI_LAND
• MPI_BAND
• MPI_LOR
• MPI_BOR
• MPI_LXOR
• MPI_BXOR

• MPI_MAXLOC
• MPI_MINLOC
• L logical
• B bit-wise

27
User Defined Reduction Ops

• void myOperator (
• void invector,
• void inoutvector,
• int length,
• MPI_Datatype datatype)

28
Ten Reasons to Prefer MPI over PVM

• MPI has more than one free, and quality
  implementations.
• MPI can efficiently program MPP and clusters.
• MPI is rigorously specified.
• MPI efficiently manages message buffers.
• MPI has full asynchronous communication.
• MPI groups are solid, efficient, and
  deterministic.
• MPI defines a 3rd party profiling mechanism.
• MPI synchronization protects 3rd party software.
• MPI is portable.
• MPI is a standard.

29
Summary

• Introduction to MPI
• Reinforced Manager-Workers paradigm
• Send, receive blocked, non-blocked
• Process groups

30
MPI resources

• Open source implementations
• MPICH
• LAM
• Books
• Using MPIWilliam Gropp, Ewing Lusk, Anthony
  Skjellum
• Using MPI-2William Gropp, Ewing Lusk, Rajeev
  Thakur
• On-line tutorials
• www.tc.cornell.edu/Edu/Tutor/MPI/