The MPI Testing Tool MTT

1
The MPI Testing Tool (MTT)?

• Ethan Mallove - Sun Microsystems
• Jeff Squyres - Cisco Systems
• Josh Hursey - Indiana University
• Andrew Lumsdaine - Indiana University
• EuroPVM/MPI 2007

2
Overview

• Why MTT?
• MTT Usage
• Configuration
• Test specification
• Test execution
• Reporting Tool
• Case study

3
The problem with testing MPI

• Portable MPI implementation needs regular
  (correctness and performance) testing
• On multiple platforms / architectures / operating
  systems
• Across multiple organizations / environments
• With many different test suites
• Exercising hundreds of run-time configuration
  options
• Require repeatability

4
Available testing frameworks

• TET no software retrieval
• Perfbase storage model did not meet our
  requirements
• DejaGNU test creator
• Others Dart, Boost, Mozilla Tinderbox, Testopia,
  Buildbot, etc.

5
Testing Flow

• MPI Get obtain the MPI implementation(s) to test
• MPI Install install the MPI implementation(s)?
• Test Get obtain the test suite(s) to run
• Test Build build the test suite(s) against each
  MPI implementation
• Test Run run all the test suite(s) against each
  MPI implementation

6
Testing Flow (cont.)?
7
MTT Configuration

• INI-style text file
• MPI Details section - how to run a job under this
  MPI
• MPI get/install sections - how to obtain /
  install MPI
• Test get/build/run sections - how to obtain /
  install / what tests to run
• Reporter section - what to do with the results

8
MTT Funclets

• Perl-like functions used in the INI file to do
  the following
• Multiplicative effectexec mpirun -np _at_np_at_
  executable()np enumerate(2, 4, 8, 16)?
• Logicpass eq(wifexited(), 0)timeout
  max(30, multiply(10, test_np()))?

9
MTT Design

• The client is written in Perl
• Modular plug-in framework
• Plug-ins for
• MPI specific functionality
• Test suite specific functionality
• Reporter specific functionality

10
MTT Test Specification

• Plug-ins carry out
• Complex builds
• Test grouping
• Control over pass/fail/time-out conditions

11
MTT Test Execution

• Managed via MPI Details INI sections
• Flexible enough to test any MPI
• Examples

MPI Details MPICH exec mpirun -np 2
executable()? MPI Details Open MPI exec
mpirun --mca btl _at_btls_at_ -np np()
executable()? btls enumerate("self,sm,tcp",
"self,sm,openib")?
12
MTT Reports
Web-driven reporting tool covers many use cases
for searching on test results

• High-level summaries
• Root causing
• Performance results
• E-mailed reports and permalinks

13
MTT Reporting Tool

• Web-driven tool
• "Drill down" and color-coded approach to managing
  sea of test results
• Summary and Detail views
• Results are never discarded
• Flexible search engine

14
Case Study The Open MPI Project

• Multitude of runtime parameters
• Many configurations
• Many platforms
• Many test suites
• etc.

The combinatoric explodes ...
15
Case Study The Open MPI Project (cont.)?

• Leverage a pool of software/hardware testing
  resources
• Unite the community of developers and testers
• Central store of test results makes collaboration
  easier

16
MTT Future Work

• Heterogeneous testing environments
• "Disconnected" scenarios (e.g., build on one
  machine, run in another)?
• Build parallelism into the testing framework
• Generalize for other software

17
Try MTT Yourselves!

• http//www.open-mpi.org/projects/mtt

18
MTT Reports
19
Summary

• Summarize the most important points of your
  presentation here

20
Discussion

• Present some ideas concerning the main points of
  your presentation
• Provoke questions and comments from your audience