Grid Metaschedulers

1
Grid Metaschedulers

• An Overview and Up-to-date Solutions
• Christian Grimme
• christian.grimme_at_udo.edu

2
Outline

• Grid Metaschedulers and Grid Brokers
• A unified view on Grid Scheduling Architecture
• Common functions of Grid Scheduling
• Up-to-date Grid Scheduling Solutions
• CSF (Community Scheduler Framework)
• GridWay
• Grid Service Broker (in GridBus)
• GRMS (GridLab Resource Management System)
• Others (Moab/Maui, WMS, Nimrod/G, Condor-G)

3
Grid Metaschedulers and Grid Brokers

• General Problem
• Allocation and scheduling of applications in a
  heterogeneous and dynamically changing set of
  resources
• Three Scenarios
• Enterprise Grid
• resources owned by single provider
• central broker which interacts with local
  resource managers
• HPC Grid
• cooperation of HPC sites to maximize
  computational power
• distributed administrative control, hierarchical
  brokering
• Global Grid
• all kind of resources
• a broker accepts all kinds of jobs to forward
  them to other brokers or local resource schedulers

4
Common Functions of Grid Scheduling

• Information
• Discover and match resources for applications
• Monitor internal strategies, feedback to user
• Job Management
• Submission of jobs and workflows
• Scheduling functionality, forecasting,
  co-allocation
• Reservation management
• Security
• Rights delegation
• Access control
• Agreement and Accounting
• Service Level Agreements
• Data access management

5
CSF (Community Scheduler Framework)

• Open source project as part of the Globus project
  in cooperation with Platform Computing
• Supports implementation of Grid metaschedulers
  based on GT services (MDS, GRAM, RFT, etc.)
• Properties
• Enables communication between heterogeneous
  schedulers on local level
• Allows reservations on resources
• Offers simple scheduling mechanisms
• Extensible via scheduling plug-ins

6
CSF Architecture
Loads queue parameters and associated schedulers
MetaScheduler
Queuing Service
Scheduling Plugin
Info Service
Reservation Service
Job Service
User Request
Create and administer compute jobs
Specified in RSL
Reserve a resource if supported by LRM
GRAM PBS
GRAM SGE
RM Adapter
Binding to LRM
PBS
SGE
LSF
7
GridWay

• Open source component (Globus Proto Project)
• Lightweight metascheduler on top of GT 2.4 4.x
• Properties
• Support of GGF DRMA standard API for job
  submission and management
• Simple scheduling mechanisms but extensible
• Allows job dependencies (workflow)
• Supports job migration/adaptive execution (Grid-
  and application-initiated)

8
GridWay Architecture
GRAM
RFT
MDS
DRMAA Library
CLI
Job control operations
GridWay Core
Request Manager
Job pool
Host pool
Scheduler
Dispatch Manager
Matchmaking, execution and migration
Information Manager
Execution Manager
Transfer Manager
GRAM
RFT
MDS
Execution of jobs on LRM
Resource
9
GSB Grid Service Broker (GridBus)

• Metascheduler to access computational and data
  Grids
• On top of various middlewares (GT 2.4 4.x,
  Alchemi, Unicore)
• Properties
• Collaborates with Grid Economy, Trading and
  Accounting Services
• Interacts with Workflow Management Service
• Leverages economic scheduling model

10
GSB Architecture
Analysis of job parameters, matchmaking
Grid Economy Services
Data Service Catalogue
Parameter Resolver
Resource Discovery
Resource Catalogue
Workflow Manage- ment
Bookkeeper
Grid Scheduler

• Application Data
• Parameters
• Tasks

Middleware binding for local job execution
Grid Info Service
Actuator Monitor
Simulation
Mapping of jobs to resources
Remote Data Host
Agent
Local Data
Wraps and controls job actions (even transfer of
distant data)
11
GRMS - GridLab Resource Management System

• Open Source Metascheduler as part of the GridLab
  project
• Based on GT 2.4 (assured to work on GT 4.x)
• Properties
• Job dependencies are planned to be supported
• Defines a Job Definition Language (GJD)
• Simple scheduling mechanisms, extensible
• Allows job migration
• Provides a GSI enabled Web Service interface

12
GRMS Architecture
Scheduling and matchmaking
Jobs in GJD (with dependencies)
Other GridLab Services
GSI enabled Web Service Interface
Workflow Module
Matchmaking Scheduler Plug-ins
Replica Management
Broker Module
Job Queue
File Movement
Queuing of jobs
...
Job Registry
Job Manager
Resource Discovery
Middleware Service Layer
GridFTP
GRAM
Monitoring
13
Others

• Moab/Maui
• commercial Metascheduler with advanced scheduling
  an monitoring capabilities
• on top of most major scheduling systems
• Nimrod/G
• Grid aware solution of Nimrod for compute
  intensive tasks
• works with GT, Legion, Condor
• WMS
• no metascheduler, submits only to gLite CEs
• has some interesting architecture features
  (push/pull models)
• Condor-G
• simple queue submission system
• treats a Grid as local resource

14
More information ...

• Literature
• Tonellotto, N., Yahyapour, R., Wieder, Ph., A
  Proposal for a Generic Grid Scheduling
  Architecture, CoreGRID Technical Report 0015,
  Institute on Resource Management, 2006
• Buyya, R., Venugopal, S., A Gentle Introduction
  to Grid Computing and Technologies, CSI
  Communications, 2005
• Web Links
• CSF www.globus.org/toolkit/docs/4.0/contributions
  /csf/
• GridWay www.gridway.org
• GSB www.gridbus.org
• GRMS www.gridlab.org
• Moab/Maui www.clusterresources.com/pages/products
  /moab-grid-suite.php
• WMS http//egee-jra1-wm.mi.infn.it/egee-jra1-wm/w
  ms.shtml
• Nimrod/G www.csse.monash.edu.au/davida/nimrod/in
  dex.htm
• Condor-G www.cs.wisc.edu/condor/