MPI Code Debugging and Verification Tool MARMOT

1
MPI Code Debugging and Verification Tool MARMOT
CrossGrid Tutorial Exercise 2.2

Bettina Krammer, Matthias Müller High Performance
Computing Center Stuttgart
2
Introduction

• MARMOT is a tool to verify if an application
  conforms to the MPI standard.
• The MPI standard leaves some decisions to
  implementors, thus an application that runs on
  one platform may have difficulties on another.
• MARMOT automatically performs a runtime-analysis
  of an application and can detect non-portable
  constructs, deadlocks etc.

3
How to use an application with MARMOT
4
Basics of MARMOT

• Library written in C that will be linked to the
  application.
• No source code modification of the application is
  required.
• Additional process working as debug server, i.e.
  the application will have to be run with n1
  instead of n processes.
• Implementation of C and Fortran language binding
  of MPI

5
Basics of MARMOT (contd)

• Environment variables for tool behaviour and
  output (report of errors, warnings and/or
  remarks, trace-back, etc.).
• After the execution of the program the user can
  read a logfile to check for potential problems.
• Marmot contains some simple correct/incorrect
  test programs to play with.

6
Functionality of MARMOT

• Deadlock detection
• Verification of MPI standard conformance
  (verification of tag ranges, request usage etc.)
• Violation of the MPI standard is reported as
  error.
• Unusual behaviour or possible problems are
  reported as warnings.
• Notes are displayed when harmless but remarkable
  behaviour occurs.

7
Functionality of MARMOT (contd)

• The MPI calls are traced on each node throughout
  the whole application.
• When detecting a deadlock the last few calls (as
  configured by the user) can be traced back on
  each node.
• The output is written to standard output (stdout
  and stderr) by default and can be redirected to a
  file by the user. The format is human readable

8
Example

• / This program produces a deadlock.
• At least 2 nodes are required to run the
  program.
• Rank 0 receives a message from Rank 1.
• Rank 1 receives a message from Rank 0.
• AFTERWARDS
• Rank 0 sends a message to Rank 1.
• Rank 1 sends a message to Rank 0.
• /

9
Example (contd)

• include ltstdio.hgt
• include "mpi.h"
• int main( int argc, char argv )
• int rank 0
• int size 0
• int dummy 0
• MPI_Status status
• MPI_Init( argc, argv )
• MPI_Comm_rank( MPI_COMM_WORLD, rank )
• MPI_Comm_size( MPI_COMM_WORLD, size )

10
Example (contd)

• if ( rank 0 )
• MPI_Recv(dummy, 1, MPI_INT, 1, 17,
• MPI_COMM_WORLD, status)
• MPI_Send(dummy, 1, MPI_INT, 1, 18,
• MPI_COMM_WORLD)
• if ( rank 1 )
• MPI_Recv(dummy, 1, MPI_INT, 0, 18,
• MPI_COMM_WORLD, status)
• MPI_Send(dummy, 1, MPI_INT, 0, 17,
  MPI_COMM_WORLD)
• MPI_Finalize()

11
Example (output log)

• mpirun -np 3 deadlock
• 1 rank 0 performs MPI_Init
• 2 rank 1 performs MPI_Init
• 3 rank 0 performs MPI_Comm_rank
• 4 rank 1 performs MPI_Comm_rank
• 5 rank 0 performs MPI_Comm_size
• 6 rank 1 performs MPI_Comm_size
• 7 rank 0 performs MPI_Recv
• 8 rank 1 performs MPI_Recv
• 8 Rank 0 is pending!
• 8 Rank 1 is pending!
• WARNING deadlock detected, all clients are
  pending

12
Example (output log contd)

• Last calls (max. 10) on node 0
• timestamp 1 MPI_Init( argc, argv )
• timestamp 3 MPI_Comm_rank( comm, rank )
• timestamp 5 MPI_Comm_size( comm, size )
• timestamp 7 MPI_Recv( buf, count -1,
  datatype non-predefined datatype, source -1,
  tag -1, comm, status)
• Last calls (max. 10) on node 1
• timestamp 2 MPI_Init( argc, argv )
• timestamp 4 MPI_Comm_rank( comm, rank )
• timestamp 6 MPI_Comm_size( comm, size )
• timestamp 8 MPI_Recv( buf, count -1,
  datatype non-predefined datatype, source -1,
  tag -1, comm, status )

13
For More Information

• Additional information
• http//www.hlrs.de/organization/tsc/projects/marmo
  t
• MARMOT is available from
• http//www.eu-crossgrid.org/