Job Execution

1
Job Execution

• Standards for Interoperable Grids Experience
  from NextGRID and OMII-Europe
• Clive Davenhall
• National e-Science Centre, University of Edinburgh

2
Job Execution Introduction

• Running jobs on a Grid.
• Topics
• Naming and addressing
• (brief introduction needed by this talk and the
  next)
• OGSA-EMS
• OGSA-BES
• JSDL

3
NextGRID Naming and Addressing

• NextGRID uses WS-Addressing and WS-Naming.
• WS-Addressing
• W3C standard.
• Way of addressing components, services, data
  items etc.
• De facto standard in the Web services community.
• WS-Naming
• OGF Standard.
• Extends WS-Addressing.
• By adding optional naming elements.
• NextGRID
• Has some additional extensions and restrictions.

4
Job Execution

• Topics
• Naming and addressing
• OGSA-EMS
• OGSA-BES
• JSDL

5
OGSA-EMS

• EMS Execution Management Services
• An architecture for initiating, managing and
  controlling the execution of activities or
  units of work.
• Units of work might be
• OGSA applications.
• Legacy applications running in an OGSA harness.
• An architecture not a standard
• So in the OGSA informational rather than
  recommendation track.

6
Role and Status of OGSA-EMS

• OGSA uses EMS to identify and position standards
  that address part of the activity management
  problem.
• Tool or mechanism for identifying necessary
  standards and their interrelations.
• This effort continues.
• Not all the necessary standards are yet in place.
• Initial focus on fundamental job execution.
• Where possible OGSA adopts or adapts existing
  standards,
• Rather than inventing new ones ab initio.

7
Problems addressed by EMS

• Finding possible locations where activities can
  execute.
• Selecting the execution location for an activity.
• Preparing for execution.
• Initiating the execution.
• Etc

8
EMS Standards

• BES
• Basic Execution Service
• JSDL
• Job Submission Description Language
• CDDLM Deployment API
• Configuration, Description, Deployment and
  Lifecycle Management Deployment API
• CDDLM CDL
• Configuration, Description, Deployment and
  Lifecycle Management Configuration Description
  Language
• ACS
• Applications Content Service
• RSS
• Resource Selection Services

9
EMS Standards

• BES
• Basic Execution Service
• JSDL
• Job Submission Description Language
• CDDLM Deployment API
• Configuration, Description, Deployment and
  Lifecycle Management Deployment API
• CDDLM CDL
• Configuration, Description, Deployment and
  Lifecycle Management Configuration Description
  Language
• ACS
• Applications Content Service
• RSS
• Resource Selection Services

10
Job Execution

• Topics
• Naming and addressing
• OGSA-EMS
• OGSA-BES
• JSDL

11
OGSA-BES

• BES Basic Execution Service
• Defines Web service interfaces for creating,
  monitoring and controlling computational
  entities
• Units of work.
• Activities.
• Individual activities to be performed are defined
  in JSDL.
• Activities execute on a suitable computational
  resource.
• Agnostic about the underlying environment on
  which the activity executes.
• BES interface is extensible.

12
BES Port-types

• BES-Management
• Allows clients to monitor and control a BES
  instance.
• Intended for system administrators.
• BES-Factory
• Allows clients to create, monitor and control
  sets of activities.
• BES-Activity
• Allows clients to create, monitor and control
  individual activities.

13
Extensibility

• State Model
• A set of defined states and the transitions
  between them.
• BES defines a basic mode and allows extensions.
• Information Model
• Allows clients to examine aspects of a BES
  instance and its applications.
• Allows mandatory, optional and user-defined
  attributes.
• Resource Model
• Specifies basic operations, but allows additional
  ones.
• Specifically allows various OGSA interaction
  modes.

14
Basic State Model
Terminated
Request to terminate activity
Finished
Pending
Running
Activity terminates successfully
System error/failure event
Failed
15
Basic State Model

• Allows extensibility by specialisation.

Terminated
Request to terminate activity
Finished
Pending
Running
Activity terminates successfully
System error/failure event
Failed
16
BES-Factory Attributes

• IsAcceptingNewActivities
• CommonName
• LongDescription
• TotalNumberOfActivities
• ActivityReference
• TotalNumberOfContainedResources
• ContainedResource
• NamingProfile
• BESExtension
• LocalResourceManagerType
• ResourceName
• OperatingSystem
• CPUArchitecture
• CPUCount
• CPUSpeed
• PhysicalMemory
• VirtualMemory

17
Operations Provided by Port-types

• BES-Management
• StopAcceptingNewActivities
• StartAcceptingNewActivities
• BES-Factory
• CreateActivity
• GetActivityStatuses
• TerminateActivities
• GetActivityDocuments
• GetFactoryAttributesDocument

18
Operations Provided by Port-types

• BES-Management
• StopAcceptingNewActivities
• StartAcceptingNewActivities
• BES-Factory
• CreateActivity
• GetActivityStatuses
• TerminateActivities
• GetActivityDocuments
• GetFactoryAttributesDocument

19
BES-Factory CreateActivity

• Used to request the creation of a new activity.
• In its simplest form
• ltbesActivityDocumentgt
• ltjsdlJobDefinitiongt
• lt/jsdlJobDefinitiongt
• ltxsdany/gt
• lt/besActivityDocumentgt

20
BES-Factory CreateActivity

• Used to request the creation of a new activity.
• In its simplest form
• ltbesActivityDocumentgt
• ltjsdlJobDefinitiongt
• lt/jsdlJobDefinitiongt
• ltxsdany/gt
• lt/besActivityDocumentgt

21
Job Execution

• Topics
• Naming and addressing
• OGSA-EMS
• OGSA-BES
• JSDL

22
JSDL

• JSDL Job Submission Description Language.
• Not an OGSA standard but adopted by OGSA.
• Abstract description of the requirements for a
  job, activity or unit of work.
• Can contain
• Generic job identification information.
• Application description.
• Resource requirements (main focus is on
  computational jobs).
• Description of any data files required.
• Describes what the job is to do, not how to do
  it.

23
JSDL and BES

• JSDL is an abstract description of a job or
  activity.
• It deliberately does not define a job submission
  interface.
• And it is therefore independent of any submission
  interface.
• BES provides a submission interface.

24
Extensions

• An extension for applications executing on a
  POSIX-compliant operating system is included in
  the basic specification.
• Two additional extensions have subsequently been
  defined
• JSDL HPC (High Performance Computing) extension.
• JSDL SPMD (Single Program Multiple Data)
  extension.
• Additional extensions are anticipated.

25
JSDL Document Structure

• ltJobDefinitiongt
• ltJobDescriptiongt
• ltJobIdentification ... /gt?
• ltApplication ... /gt? ltResources... /gt?
• ltDataStaging ... /gt
• lt/JobDescriptiongt
• ltxsdanyother/gt
• lt/JobDefinitiongt

• Note
• None 1..1
• ? 0..1
• 0..n
• 1..n

26
JSDL Document Structure

• ltJobDefinitiongt
• ltJobDescriptiongt
• ltJobIdentification ... /gt?
• ltApplication ... /gt? ltResources... /gt?
• ltDataStaging ... /gt
• lt/JobDescriptiongt
• ltxsdanyother/gt
• lt/JobDefinitiongt

• Job Identification
• Application description
• Resource requirements
• Data files required

• Note
• None 1..1
• ? 0..1
• 0..n
• 1..n

27
Job Identification Element

• ltJobDefinitiongt
• ltJobDescriptiongt
• ltJobIdentification ... /gt?
• ltApplication ... /gt? ltResources... /gt?
• ltDataStaging ... /gt
• lt/JobDescriptiongt
• ltxsdanyother/gt
• lt/JobDefinitiongt

• ltJobIdentificationgt
• ltJobName ... /gt?
• ltDescription ... /gt?
• ltJobAnnotation ... /gt?
• ltJobProject ... /gt?
• ltxsdanyother/gt
• lt/JobIdentificationgt

28
Job Identification Example

• ltJobDefinitiongt
• ltJobDescriptiongt
• ltJobIdentification ... /gt?
• ltApplication ... /gt? ltResources... /gt?
• ltDataStaging ... /gt
• lt/JobDescriptiongt
• ltxsdanyother/gt
• lt/JobDefinitiongt

• ltJobIdentificationgt
• ltJobNamegt
• My Job
• lt/JobNamegt
• ltDescriptiongt
• Simple application
• lt/Descriptiongt
• lt/JobIdentificationgt

29
Hello World in JSDL

• lt?xml version"1.0" encoding"UTF-8"?gt
• ltjsdlJobDefinition
• xmlnsjsdlhttp//schemas.ggf.org/2005/11/jsd
  l
• xmlnsjsdl-posix
• http//schemas.ggf.org/jsdl/2005/11
  /jsdl-posixgt
• ltjsdlJobDescriptiongt
• ltjsdlApplicationgt
• ltjsdl-posixPOSIXApplicationgt
• ltjsdl-posixExecutablegt
• /bin/echo
• ltjsdl-posixExecutablegt
• ltjsdl-posixArgumentgthellolt/jsdl-posix
  Argumentgt
• ltjsdl-posixArgumentgtworldlt/jsdl-posix
  Argumentgt
• lt/jsdl-posixPOSIXApplicationgt
• lt/jsdlApplicationgt
• lt/jsdlJobDescriptiongt
• lt/jsdlJobDefinitiongt

30
NextGRID Recommendations

• BES
• Reiterates that extensions to WS-Naming are
  permitted, specifically the ones required by
  NextGRID naming.
• JSDL
• Defines extensions to support data-staging which
  is compatible with NextGRID data transfer.