CrossBroker: Scheduling interactive MPI applications to the Grid

1
CrossBroker Scheduling interactive MPI
applications to the Grid
Enol Fernández1, Alvaro Fernández2, Antonio
Hervás1, Ania Morajko1, Elisa Heymann1, Miquel
A. Senar1 1Universitat Autònoma de Barcelona
(UAB) 2Instituto de Física Corpuscular (IFIC)

2
CrossGrid Resource Management Goals

• Provide scheduling support for interactive and
  parallel applications submitted to the grid
• responsible for allocating application
  tasks to computing resources in a way that tries
  to guarantee that applications are executed
  achieving a reasonable trade-off between resource
  efficiency and application speedup.
• A more precise definition of the problem has been
  stated according to actual needs of real
  scientific applications developed in the project.
• Users must be able to submit
• Parallel applications (MPI) that run over a
  single site or over multiple sites.
• Interactive applications that should run (near)
  immediately.
• Composite applications (workflows).
• Take advantage of pre-existing technologies and
  be compatible with DataGrids middleware.
• Lack of centralized (absolute) control. Need for
  error recovery mechanisms (homogeneous
  environment, but nothing should be assumed as
  granted)

3
System Architecture
4
CrossBrokers Functionality (1)

• Automatic job management for parallel
  applications
• Search and selection of available resources, job
  conditioning, job launching, job monitoring, job
  retry (in case of failures) and results
  retrieval.
• MPICH-P4 (intra-cluster)
• MPICH-G2 (inter-cluster)
• Computational Workflows
• Best effort approach to deal with
  failures/problems

5
CrossBrokers Functionality (2)

• Jobs described in a text file using JDL (Job
  Description Language)

Type Job" JobType interactive,
mpich-g2" Executable tester-mpichg2" Argument
s "" NodeNumber 7 InputSandbox "
tester-mpichg2 " VirtualOrganisation
"cg" Rank other.GlueHostBenchmarkSI00
Requirements other.GlueCEStateStatus"Product
ion"
6
CrossBrokers Functionality (3)

• Job conditioning for interactive input/output
  streaming (sequential, MPICH-P4 and MPICH-G2)

Interactive Job console
started for https//aorbgrid.uab.es9000/IAYUQS7E
6J4aySd3bjImVQ Please press C to exit from the
session
Subjob0 my_id 0
numprocs 2 Subjob0 Number of trips around the
ring ? 2 Subjob0 Verbosity (yes/no) ?
yes Subjob0 Processor name cg05.ific.uv.es Subjo
b0 Starting trip 1 of 2 before sending num1 to
dest1 Subjob0 Inside trip 1 of 2 before
receiving from source1 Subjob1 my_id 1 numprocs
2 Subjob1 Processor name cg04.ific.uv.es Subjob1
Top of trip 1 of 2 before receiving from
source0 Subjob1 Inside trip 1 of 2 after
receiving passed_num1 from source0
7
CrossBrokers Functionality (4)

• Time-sharing support for interactive jobs. Fast
  response time under heavy occupancy of resources
  by batch jobs

CrossBroker
Grid Resource
JobManager
Scheduler
PBS
Condor-G
8
CrossBrokers Functionality (5)

• Integrated with Grid Monitoring and Data Analysis
  Tool (GMDAT) monitoring data obtained by
  external tools can be used during the scheduling
  process according to user requirements or
  preferences

Type Job" JobType interactive,
mpich-g2" Executable tester-mpichg2" Argument
s "" NodeNumber 7 InputSandbox "
tester-mpichg2 " VirtualOrganisation
"cg" Rank ppt_getClusterParameter("sum","idle_b
ogomips",other.groupGlueCEUniqueId,0)
ppt_getNetworkParameter("avg","bandwidth",other.g
roupGlueCEUniqueId,0)
GMDAT metrics
9
CrossBrokers Functionality (and 6)

• Text-based command-line interface
• Integrated with other CrossGrids user-friendly
  interfaces through the Roaming Access Server
  (Migrating Desktop and Web Portal)
• Simple API allows an easy interface of
  CrossBroker with other external monitoring tools
• Several internal mechanisms to avoid or recover
  from problems in Grid resources in a transparent
  way

10
CrossBroker

• Further information at
• www.eu-crossgrid.org (WP3 section)
• Contact
• miquelangel.senar_at_uab.es
• elisa.heymann_at_uab.es
• Questions?