Process Manager Update

1
Process Manager Update May 6

• The Process Manager component (PM)
• The Process Manager implementation (MPD2)
• Issues generated for other components by process
  management

2
The Process Manager Component

• Added limits to interface definition
• Example on next page
• Not implemented yet in terms of parsing and
  passing on to MPD
• Dynamic jobs (MPI_Comm_spawn)
• Current interface allows the process manager to
  be given a list of nodes and a number of
  processes to start, independently.
• Process manager implementation can then use
  unused nodes to start spawned processs (or not)
• MPI_UNIVERSE_SIZE allows MPI job to get hint
  about how many processes can usefully be spawned

3
Limits Specification Example

• ltcreate-process-group
•            totalprocs'2'gt
•       ltprocess-spec    range'0'   
  cwd'/home/rbutler/mpd2'  exec'infloop'gt
•         ltarg idx'1' value"hello"gt lt/arggt
•         ltarg idx'2' value"from 0"gt lt/arggt
•         ltlimit type'cpu' value"2"/gt
•       lt/process-specgt
•       ltprocess-spec  range'1'  
  cwd'/home/rbutler/mpd2'     exec'infloop'gt
•         ltarg idx'1' value"hello"gt lt/arggt
•         ltarg idx'2' value"from 1"gt lt/arggt
•         ltenv name'foo' value"bar"gt lt/envgt
•         ltlimit type'cpu' value"3"/gt
•       lt/process-specgt
•       lthost-specgt
•            magpie
•       lt/host-specgt
• lt/create-process-groupgt

4
The Process Manager Implementation

• Improvements to MPD resulting from production use
  on Chiba
• Mostly in recovering from errors and crashes by
  applications
• Support for limits (those supported in setrlimit)
• Improvements in configuring and building along
  with MPICH2
• Support for MPI_Comm_spawn through PMI interface
  to MPICH2 application
• Interactive debugging via mpigdb

5
Coercing gdb Into Functioning as a Primitive
Parallel Debugger

• Key is control of stdin, stdout, stderr by MPD,
  through mpigdb
• Replaces mpiexec or mpirun on interactive command
  line
• Usable through SSS process manager component
• Stdout, stderr collected in tree, labeled by
  rank, and merged for scalability
• (0-9) (gdb) p x
• (0-2) 1 3.4
• (3) 1 3.8
• (4-9) 1 4.1
• Stdin can be broadcast to all or to a subset of
  processes
• z 3 (to send input to process 3 only)
• Same for interrupts
• Can run under debugger control, interrupt and
  query hung processes, parallel attach to running
  parallel job

6
Coercing gdb Into a Parallel Debugger

• donner mpigdb -n 5 cpi
• 0-4  (gdb) b 30
• 0-4  Breakpoint 1 at 0x804930c file
  /home/lusk/mpich2/examples/cpi.c, line 30.
• 0-4  (gdb) r
• 0-4  Starting program /sandbox/lusk/mpich2-mpd-m
  ar25/examples/cpi
• 0-4  Breakpoint 1, main (argc1,
  argv0xbffff224)
• 0-4      at /home/lusk/mpich2/examples/cpi.c30
• 0-4  30            n 10000                  /
  default of rectangles /
• 0-4  (gdb) n
• 0-4  31            if (myid 0)
• 0-4  (gdb) p n
• 0-4  1 10000
• 0-4  (gdb) p myid
• 0  2 0
• 2  2 2
• 1  2 1
• 3  2 3
• 4  2 4
• 0-4  (gdb)

7
mpigdb (cont)

• 0-4  (gdb) n
• 0  32          startwtime MPI_Wtime()
• 1-4  34            MPI_Bcast(n, 1, MPI_INT, 0,
  MPI_COMM_WORLD)
• 0-4  (gdb) z 0
• (gdb) n
• 0  34      MPI_Bcast(n, 1, MPI_INT, 0,
  MPI_COMM_WORLD)
• 0  (gdb) z 0-4
• 0-4  (gdb) n
• 0-4  36            h   1.0 / (double) n
• 0-4  (gdb) n
• ..............
• 0-4  42                sum f(x)
• 0-4  (gdb) p sum
• 0  4 23.999941940268087
• 2  4 23.999928500379845
• 1  4 23.999935460320089
• 3  4 23.999921060448177
• 4  4 23.99991314052599
• 0-4  (gdb) q

8
Issues Generated For Other Components

• Job steps
• Option 1 QM handles (preferred)
• Process manager starts process groups directly
• Need public definition of user interface to QM
• Option 2 PM implementation handles PBS-like
  scripts from QM
• A bit weird mpirun in a PBS script is trapped
  by extra layer (MPISH) because the real mpirun
  is a call to MPD itself
• In use on Chiba
• QM interface for requesting allocation of some
  number of nodes but starting up on different
  number of nodes, particularly for option 1.
• QM interface for requesting dynamic rebuilds
• Limits in QM interface?

9
Tale of Two Queue Manager Implementations
Same XML syntax different content
QM Interface (XML)
QM1
qsub1
totalprocs1, execmyscript
mycript contains mpirun np 64 cpi
(mpirun intercepted)
PM
Different XML syntax
Underlying process manager (MPD)
totalprocs64, execcpi
QM2
qsub2
(mpirun is interactive interface to underlying
process manager)
QM Interface (XML)