Globus Toolkit Version 4'0

1
Globus Toolkit Version 4.0

• Kyung-Lang Park
• (2005. 8. 27)

2
Reference

• Globus public homepage, http//www.globus.org
• A Globus Primer (Ver 0.6)
• GT4 Admin Guide
• GT4 Programmers tutorial

3
Contents

• The Role of the Globus Toolkit
• Basic Principles
• GT4 Architecture
• Execution Management
• Other Components
• Conclusion

4
Role of the Globus Toolkit

• Provide standard system components that can
  support a wide variety of highly customized
  applications and user interfaces without
  requiring a completely unique infrastructure to
  be developed for each application
• GTK doesnt provide a complete solution for Grid
  projects
• Collection of tools
• Solutions to sub-problems
• It provides
• Two very important SDK
• Web Services Core implementation
• Grid security Infrastructure Implementation
• A Variety of Basic Grid Services
• Developer APIS
• Tools and Examples

5
Motivating Examples

• Enterprise Workload Management
• High Performance Data Capture
• Network for Earthquake Engineering Simulation
• Earth System grid (ESG)
• Open Science Grid
• Biomedical Informatics Research Network (BIRN)

6
Requirements

• Standardized Mechanism and Interfaces
• Service-oriented architecture and Web services
• GT4 Web service containers
• Infrastructure
• Computers, storage systems, and instrumentation
• GRAM, GTCP, and GridFTP(no WS)
• Monitoring and Discovery
• Based on WSRF/WS-Notification specification
• XML-based resource properties
• Three kinds Aggregator services
• Registry (MDS-Index), archiver(MDS-Archive),
  event-driven data filter (MDS-Trigger)

7
Requirements - cont.

• Security
• X.509 public key credential
• Data
• GridFTP
• Globus Replica Location Service (RLS)
• Globus Data Access and Integration (DAI)
• Choreography coordinate activities
• Indirect support for coordination itself
• DAGman, MPICH-G2

8
Service-oriented architecture

• GT4 is a set of components that implement Web
  services for building distributed systems
• A web service is
• Web Service Implementations
• Web Services Specifications
• XML, SOAP, WSDL
• Ws-Security, SAML
• WS-Addressing, WSRF, Ws-Notification
• Others
• GridFTP, OGSA-DAI, WS-I Profile, WSDM, WS-CIM,
  OGSA

9
SOA vs. SOI

• Web services can be used to build both
  Service-Oriented Applications and
  Service-Oriented Infrastructure
• Service-oriented application is constructed via
  the composition of components defined by service
  interfaces
• Financial or biological database, an options
  pricing routine or a biological sequence analyzer
• Service-oriented infrastructure to denote the
  resource management and provisioning mechanisms
  used to meet quality of service goals for
  components and applications
• GT4 are concerned with enabling the construction
  of Service-oriented infrastructure

10
GT4 Architecture
11
Predefined GT4 Services

• Job management (GRAM)
• Reliable File Transfer (RFT)
• Delegation
• MDS-Index, MDS-Trigger, MDS-Archive
• Community Authorization (CAS)
• OGSA-DAI
• GTCP
• GRAM and MDS, pre-WS legacy implementations are
  also provided
• Three additional GT4 services (not-WS)
• GridFTP
• Replica location service
• MyProxy

12
Command Line Programs
13
GT4 Containers

• GT4 Container
• Implements SOAP over HTTP as message transport
  protocol
• Both transport-level and WS-Security
  message-level security for all communications
• Implements WS-Addressing, WSRF, and
  WS-Notification
• Supports logging via Log4j
• Defines WSRF WS-Resources
• Thus, GT 4 container can
• Host custom Web services
• Host custom WSRF Web services
• Host advanced services provided by GT4
• GRAM, MDS, and RFT (GT4 WSRF Web services)
• Clients can
• Use GT4 container registry interfaces
• GT4 containers support three different languages
  (Java, C, Python)

14
Other issues

• Security
• Deploying GT4 Web Services
• Developing GT4 Web Services
• See GT4 Programmers Tutorial

15
Execution Management

• You want to
• Make a program available as a network service
• Dispatch
• Run an executable on a remote computer
• Run an parallel program across multiple
  distributed computers
• Run a set of loosely coupled tasks
• Steer a computation
• These task all fall within the purview of
  execution management

16
Execution Management Tools

• Grid Resource allocation Management Services
• Submission, Monitoring, and control of jobs on
  computers
• Interfaces to Unix shell (fork), or schedulers
• PBS, LSF, Condor
• Include support for MPICH-G2 jobs
• Multi-job submission, process coordination in a
  job, subjob coordination in a multi-job
• Associated tools
• GRAM-enabled schedulers
• Condor, OpenPBS, Torque, PBSPro, SGE, LSF
• Programming models using GRAM to dispatch tasks
  to remote coms.
• Condor-G, DAGman, MPICH-G2, GriPhyN VDS, Nimrod-G
• Meta-Schedulers
• CSF, MAUI

17
GRAM

• Basic concept of remote execution
• Define a Web service operation that execute your
  program
• E.g.,) Foo(a,b) to request that foo be executed
  with a as input and returning b
• Additional Requirements
• State, User executables, Staging of input and
  output
• Streaming output, Control, Schedulers
• Monitoring
• State transitions
• The GRAM and its client software address these
  issues
• GRAM is meant to be used in situations where the
  ability to run arbitrary programs achieve
  reliable operation, perform stateful monitoring,
  manage credentials, stage files, and interact
  with schedulers are important

18
GRAM

• GRAM create a stateful entity
• A ManagedJob
• lifetime similar to that of the associated job
• Submit operation return handle for the new
  Managed job
• WS-Addressing endpoint reference
• Various operations can be performed via handle
• Kill, query the status, attach
• GRAM is not a resource scheduler, but rather a
  protocol engine for communicating with a range of
  different local resource schedulers

19
globusrun-ws

• A command line client to submit a job
• Basic job submission
• Command
• globus-ws submit job-command /bin/touch
  touched_it
• Support XML-based job description language
• Target GRAM server is specified by an EPR
• Constructing an EPR automatically from a
  specified URL

20
globusrun-ws

• Interacting with a submitted job
• Job Status and Lifecycle
• File Staging
• Credential Delegation
• Options
• Modify timeout values
• Job resubmission for exactly-once job submission
  semantics
• Lifetime management
• Parallel jobs
• Multiple copies of the specified executable
• Multijobs
• execute different programs
• Job and process rendezvous
• Synchronization between processes in a
  multiprocess job

21
WS Randezvous

• http//www-unix.globus.org/toolkit/docs/developmen
  t/3.9.5/execution/wsrendezvous/

22
Related Software and Tools

• DAGman, Condor-G, GriPhyN
• GT2 compliant
• Nimrod-G v3.0
• (v3.0) GT4 compliant
• Released
• MPICH-G2
• GT2 compliant (Pre-WS)
• Ninf-G
• (v4.0) GT4 compliant
• Alpha version

23
How GRAM Works
24
Data Management
25
Monitoring and Discovery
26
Security
27
Security
28
User Interface
29
Packaging and Distribution
30
Misc.
31
Conclusion

• GT4 implements WSRF
• GT4 provides services for construction of SOI and
  SDKs for supporting development of SOA
• WS-GRAM is more powerful than we think
• Job Management
• Support multijobs
• WS-Randezvous
• Some programming models already support GT4
• Maybe MPICH-G4 is coming

32

• Checklist
• Are we going to use MPI as a High-level Language
  ?
• Are we going to follow component-based
  architecture ?
• Loosely coupled Grid
• Are we going to follow service-oriented
  architecture ?
• Are we going to use WSRF ?
• Are we going to use GT4?
• Functional requirements
• Discovery
• Migration
• Authentication
• Process Generation
• Co-allocation and Sync.
• What we have to do from now?
• Finish the checklist
• Define functional requirements to run MPI
  applications
• Design framework