Meta Scheduling Framework for Grid Computing Environments

1
Meta Scheduling Framework for Grid Computing
Environments

• Jaeyoung Choi
• Soongsil University
• choi_at_comp.ssu.ac.kr

2
Introduction

• Motivation
• Restricted services of the Grid middleware system
  (GT2)
• Not easy to execute a complicated job
• Efficient use of Grid resources
• Not easy to use
• Research Goal
• Provide an integrated Grid computing environment,
  which supports Grid Workflow Service

3
Objective

• Provide a new framework to support services,
  which are required for Grid applications
• Effective use of resources
• Prevent wasting resources by allocating resources
  to a job optimally
• Possible to execute complicated jobs
• A grid job may have several processes and
  execution steps, which should be executed in
  order
• Support methods to describe and execute the
  complicated job directly
• Provide useful grid environments
• Provide user-centric job control environments

4
Requirements

• Job description language
• Need to model and describe complex jobs, that
  include various requirements
• Manage complicated Grid job flow
• Need a managing system that execute and control
  Grid jobs in dynamic Grid environments
• Easy to use working environments
• Need a working environment to monitor and
  manipulate Grid jobs
• Compatible with Globus Toolkit (GT2)
• Need to be compatible with GT2, which is a de
  facto standard of grid infrastructure

5
Meta Scheduling Framework

• Major Components
• Grid Workflow Management System (WFMS)
• Provide a workflow management system, which
  executes and controls various Grid jobs
• Job Control Markup Language Editing System
• A language for describing a workflow on grid
  environments
• GUI-based integrated editing system for JCML
• MSF Infrastructure
• Provide an interface to Grid middleware (GT2)
• Support interoperability between WFMS and Grid
  middleware

6
MSF Architecture
7
Services

• MSF services
• Definition
• Define a job on a Grid
• Describe the Job Definition List (JDL) using JCML
  Editor
• Preparation
• Search of optimal resource after analyzing the
  JDL
• Prepare to allocate resources
• Execution
• Allocate the selected resource to the job
• Launch the task according to the sequence order
  of workflow process

8
JCML Editor

• Job Control Markup Language (JCML)
• A description language for the Job Definition
  List (JDL) in workflow management system
• Possible to describe details of processing steps
  of a job and specify the required resources in
  each step
• Editing System
• User-friendly workspace
• Graph-based representation to understand the grid
  job easily

9
JCML

• Major Elements
• Info
• Basic information of JCML Doc.
• Resource
• Information of resources to process requested
  jobs
• Component
• Information of Nodes Groups
• A Grid job is composed of Nodes Groups
• Dependency
• Information of dependency among Nodes (or Groups)

10
JCML Schema
11
JCML Editing System

• JCML Editor

12
JCML Editing System (cont)
13
Grid Workflow Service

• Overview
• Define, prepare, and execute a Grid job

14
WFMS Architecture
15
MSF Infrastructure

• Consist of 4 major components
• Access manager (AM)
• Resource manager (RM)
• Execution manager (EM)
• Information manager (IM)

16
Access Manager

• Provide an interface to a user
• Execute a grid job as proxy using user agents
• User authentication, Session management,
  Job submit/user command processing, Monitor
  View

17
Resource Manager

• Search for resource by analyzing JDL
• Optimal resource allocation with appropriate
  algorithms for each job

18
Execution Manager

• Execute and manage grid jobs
  (including workflow runtime-engine)
• Task launching, monitoring

19
Information Manager

• Control all information on MSF
• System event detection and recovery of system
  failure
• Job status check pointing for fault tolerant
  monitoring the information

20
Example (1)

• Virtual Screening for New Drug Design
• Autodock for Linux

Autodock process
JCML modeling
21
Example (2)

• Editing JCML

22
WorkFlow Engine Console (1)
Grid Resource Information
WorkFlow Monitoring Control
23
WorkFlow Engine Console (2)
WorkFlow Queue
SubTask Queue
24
WorkFlow Engine Console (3)
WorkFlow Graph
JCML
WorkList ML
SubTask Queue
25
Progress Status

• Meta Scheduling Framework
• JCML Editing System
• Designed JCML job control language JCML schema
• Implemented GUI Based JCML Editor
• Workflow Management System
• Designed Worklist schema implemented RSL
  interpreter
• Impelmented JCML ? Worklist Converter
• Implemented Workflow engine
• Included powerful GUI based console
• Infrastructure
• All components (AM, RM, EM, IM) are designed

26
Q A