CSF4 Meta-Scheduler Tutorial

1
CSF4 Meta-Scheduler Tutorial

• 1st PRAGMA Institute
• Zhaohui Ding
• zhding_at_ucsd.edu or zhaohui.ding_at_email.jlu.edu.cn
• College of Computer Science Technology
• Jilin University
• National Biomedical Computing Resource,
  University of California, San Diego

2
Agenda

• Meta-scheduler CSF4 Introduction
• CSF4 Architecture
• CSF4 Functionalities
• Future Work
• Demo and Practice

3
What is Meta-Scheduler

• Resource Allocation Management
• Heterogeneous
• Distributed
• Dynamic
• Local Scheduler VS Meta-scheduler

4
Local Scheduler VS Meta-Scheduler
Local Scheduler Meta Scheduler
Administrative scope Cluster, Single Domain Grid, Multiple Domains, Virtual Organizations
Hardware Software (OS) Homogeneous Heterogeneous OS-independent
Data management LAN file system (NFS, FTP, scp) Global file system (Gridftp, Gfarm)
Security OS user/passwd, NIS, ssh public key Grid Security Infrastructure(GSI)
Resource Management Protocol Specified, Private Protocols for different local scheduler Standard, Open, General-Purpose Protocols (GRAM)
Scheduling mode Centralized Centralized / Distributed
5
Meta-Scheduler VS Local Scheduler

• Local Scheduler
• LSF (Load Sharing Facility)
• PBS (Portable Batch System)
• SGE (Sun Grid Engine)
• Condor
• IBM Loadleveler
• Meta-Scheduler
• CSF
• Maui (Silver)
• Gridway
• Nimrod-G
• Condor-G

6
CSF4

• What is CSF Meta-Scheduler
• Full Name Community Scheduler Framework
• CSF4 contains a group of grid services host in
  GT4
• CSF4 is a full WSRF compliant meta-scheduler.
• Open Source project and can be accessed at
  http//sourceforge.net/projects/gcsf
• Developed by Lab. of Distributed Computing and
  System Architecture, Jilin University, China
• CSF4 has been added to Globus Toolkit 4 as an
  Execution Component

7
CSF4 in Globus Toolkit 4
8
A typical deployment
9
What CSF4 Can Do?

• Basic Functionalities
• Submit jobs to Grid without Specifying Cluster
• Monitor and Control Jobs
• Provide Queuing Service
• Schedule jobs and resource by custom-built
  polices
• CSF4 Portlet (A Web browser based User Interface)

10
What CSF4 Can Do? (cont.)

• Advanced Functionalities
• Multiple Domains Resource Information Sharing
• Multiple scale resource scheduling policies
• Automatic user credentials delegation
• Automatic data-staging
• Extensible scheduling framework
• Supporting grid parallel jobs (MPIMPICH-G2)

11
CSF4 Architecture
12
CSF4 Architecture User view
13
Local Scheduler And Infrastructure Supported by
CSF4

• Local Scheduler Supported
• LSF
• PBS
• SGE
• Condor
• Infrastructure Supported
• Globus Toolkit 4
• Globus Toolkit 2

14
CSF4 Functionalities Scheduling Plug-in
Framework

• Designed For Queuing Service
• Provide A set of policies
• Customizable
• Extensible

15
Existent Scheduling Policies

• FCFS (First Come First Serve) round-robin
• Default policy
• Throttle
• Restrict the number of jobs in a scheduling cycle
• Array Job Plug-in
• Design for life science applications (such as
  AutoDock, BLAST)
• MPICH-G2 Plug-in (under-developing)
• The plug-in guarantee the synchronized resource
  allocation can be successful
• Data intensive applications plug-in
  (under-developing)

16
Schedule plug-in scheduling policies

• Each policy is implemented inside a scheduling
  plugin module
• A queue can load multiple plugin modules

17
Resource Information Sharing

• A MDS information provider for CSF4
• Multiple CSF4 can share the resource information

18
CSF4 Functionalities (cont.)

• Deploy Multiple CSF4 in a Grid Community

19
Array Job

• AutoDock and Blast-like applications
• A large number of sub-jobs.
• Execute same binary
• Different input/output files

20
Array Job (cont.)

• Advantages
• Submit job only once
• Save submission time and memory storage

21
Data Staging

• Manual Data Staging
• Which clusters I can use?
• Which clusters my jobs will running on?
• Where is the output data?
• When will the job finish, so that I can
  stage-out the output data?

22
Manual Data Staging

• Without Meta-Scheduler

23
Automatic Data Staging

• With CSF4 Automatic Data Staging

24
Integrate CSF4 with Gfarm

• With CSF4 Automatic Data-Staging and Gfarm

25
Application Based Scheduling
26
CSF4 User Interface

• CSF4 Portal

27
CSF4 User Interface

• CSF4 Command Line

28
Under-Developing work
29
Demo Practice

• https//www.nbcr.net/pub/wiki/index.php?titleCSF4
  _Tutorial_PRAGMA13

30
Thank you???????????????????