COMP3019 Coursework: The OMII Development Kit

1
COMP3019 CourseworkThe OMII Development Kit

• Dr Stephen CrouchOMII-UK
• s.crouch_at_omii.ac.uk
• 12/03/09

2
Contents

• The Open Middleware Infrastructure Institute
  (OMII-UK)
• Background Grid, Web Services
• The OMII Development Kit
• Job Submission using GridSAM
• Service Discovery using Grimoires

3
OMII-UK

• OMII-UK is an open-source organisation that
  empowers the UK research community by providing
  software for use in all disciplines of research
• Collaboration between
• School of ECS at University of Southampton
• School of Computer Science at University of
  Manchester
• National e-Science Centre, Edinburgh
• Promote Open Source
• User-focused software development
• Development process, testing documentation
• Focus on the user experience
• Easy to install use
• Utilise existing software and standards
• Use commonly used components
• Promote and support interoperability through
  standards

4
Where does our Software come from?

• Open Source Community
• Tomcat, Axis, etc.,
• Software Repository
• Accept software contributions
• Software deployed, tested evaluated to provide
  feedback
• Commissioned Software
• Fill gaps to build a solid enabling
  infrastructure
• Projects to bring research software to production
  quality
• Components delivered to integrate into the release

5
Grid Architecture Today

• The best way of designing grids
• Loosely coupled services
• Message based exchange
• The best way of running grids
• Interoperability between versions grids
• Standards for infrastructure services
• The best way of building grids
• Leverage existing infrastructure standards
• Use Web Services

6

• Web Services

7
Web Service Definition

• A service is the logical manifestation of some
  physical or logical resources (databases,
  programs, devices, humans, etc) and/or some
  application logic that is exposed to the network
• Service interaction is facilitated by message
  exchanges

8
Web Services (WS) I

• XML Platform neutral mechanism to describe data
• SOAP Mechanism to describe message exchange
• Simple Object Access Protocol
• Not simple and nothing to do with Objects!
• Service Oriented Access Protocol
• Re-engineering of acronym to fit current use!
• WSDL Defines the service interface
• Web Services Definition Language

9
Web Services (WS) II

• Services have to reside in a supporting
  environment
• Called hosting environment or container
• e.g. Axis on Tomcat
• Marshals requests into and response out of the
  service
• Service can discover local configuration
  parameters
• Provides a standard infrastructure for service
  developers
• Processing incoming requests outgoing responses
• Called Message handlers
• Manipulates elements of the message header
• Primarily the SOAP header
• Handlers can be applied to message traffic into
  or out of the whole container or a specific
  service

10
Interacting with a Web Service

• Remote Procedure Call (RPC)
• Message body specifies the invoked service
  function
• Input XML version of input arguments
• Output XML version returned to caller
• Document model
• Asynchronous model
• XML document consumed by port
• XML document returned to caller

11

• The OMII Development Kit

12
OMII Development Kit Introduction

• OMII Dev Kit acts as a provider of
• Services
• That provide function. e.g.
• GridSAM - job submission
• Grimoires - service registry (for discovery)
• On a number of security levels
• Resources
• Computation (supercomputer, cluster)
• Data
• Provides resources through services
• Termed middleware
• Within a secure environment
• Currently version 3.4.3

13
OMII Development Kit Server

• Comprised of
• Web Services Container
• Tomcat/Axis - Web Services provider/container
  with optional WS-Security - handles
  authentication
• Data persistence managed by Postgres database
• Optionally downloadable and installable
  Commissioned Software server components
• GridSAM Job submission and monitoring
• Grimoires UDDI metadata registry
• OGSA-DAI Database abstraction and federation

14
OMII Development Kit Client

• Used to communicate with an OMII Server
• Comprised of
• Effectively a client security container.
• Contains Configuration for security (including
  security credentials)
• Used by client applications to interact with
  services on OMII server
• Optionally downloadable and installable
  Commissioned Software client components

15
Architecture
ExampleService
WS-Security
Client
16

• GridSAM
• Job Submission and Monitoring

17
GridSAM Overview

• What is GridSAM?
• A Job Submission and Monitoring Web Service
• Allows compute resources to be accessible
  securely over the internet
• Uses Job Submission Description Language (JSDL)
  to describe requirements for jobs
• Funded by the OMII-UK Commissioned Software
  programme

18
GridSAM Overview

• What is GridSAM to the resource owners?
• A Web Service to expose heterogeneous execution
  resources uniformly
• Single machine through Forking or SSH
• Condor
• Portable Batch Scheduler (PBS)
• Sun Grid Engine
• Globus
• Acts as a client to these resources

19
GridSAM Overview

• What is GridSAM to end-users?
• A set of end-user tools and client-side APIs to
  interact with GridSAM Web Services
• Submit and start jobs
• Monitor jobs
• Terminate jobs
• File transfer
• Client-side submission scripting
• Client-side Java API

20
GridSAM Publications Enabled Activities
21
GridSAM Architecture

• A staged event-driven architecture
• Submission pipeline is constructed as a network
  of stages connected by event queues
• Each stage performs a specific action upon
  incoming events

22
Scenario Condor Pool
23
Example using File Staging

• Objectives submit simple job with data input and
  output requirements and monitor progress

submit JSDL
OMII GridSAM Client
OMII GridSAM Server
monitor
2 input files
OMII GridSAM FTP Server
1 output file
24
JSDL Example

• ltomii_client_homegt/gridsam/data/examples/remotecat
  -staging.jsdl
• ltJobDescriptiongt
• ltJobIdentificationgt lt/JobIdentificationgt
• ltApplicationgt
• ltPOSIXApplication xmlns"http//schemas.ggf.or
  g/jsdl/2005/06/jsdl-posix"gt
• ltExecutablegtbin/concatlt/Executablegt
• ltArgumentgtdir2/subdir1/file2.txtlt/Argumentgt
  
• ltOutputgtstdout.txtlt/Outputgt
• ltErrorgtstderr.txtlt/Errorgt
• ltEnvironment name"FIRST_INPUT"gtdir1/file1.t
  xtlt/Environmentgt
• lt/POSIXApplicationgt
• lt/Applicationgt

25
JSDL Example

• ltDataStaginggt
• ltFileNamegtbin/concatlt/FileNamegt
• ltCreationFlaggtoverwritelt/CreationFlaggt
• ltSourcegt
• ltURIgtftp//ftp.do55521/concat.shlt/URIgt
• lt/Sourcegt
• lt/DataStaginggt
• ltDataStaginggt
• ltFileNamegtdir1/file1.txtlt/FileNamegt
• ltCreationFlaggtoverwritelt/CreationFlaggt
• ltSourcegt
• ltURIgtftp//ftp.do55521/input1.txtlt/URIgt
• lt/Sourcegt
• lt/DataStaginggt

ltDataStaginggt ltFileNamegtdir2/subdir1/file2.txtlt/
FileNamegt ltCreationFlaggtoverwritelt/CreationFlag
gt ltSourcegt ltURIgtftp//ftp.do55521/input2.t
xtlt/URIgt lt/Sourcegt lt/DataStaginggt ltDataStaginggt
ltFileNamegtstdout.txtlt/FileNamegt
ltCreationFlaggtoverwritelt/CreationFlaggt
ltDeleteOnTerminationgttruelt/DeleteOnTerminationgt
ltTargetgt ltURIgtftp//ftp.do55521/stdout.txt
lt/URIgt lt/Targetgt lt/DataStaginggt lt/JobDescriptio
ngt lt/JobDefinitiongt
26

• Grimoires
• Service Discovery

27
Grimoires Overview

• Grid RegIstry with Metadata Oriented Interface
  Robustness, Efficiency, Security
• A Grimoire is a magician's manual for invoking
  demons (Oxford English Dictionary)
• Pronunciation grimwar / grimwars
• The Grimoires registry hosts descriptions of
  services (and workflows), which can be published,
  discovered and used by clients
• Like a yellow pages for web services

28
Grimoires Overview

• What is Grimoires to the resource owners?
• A Web Service to expose locations of
  heterogeneous web services uniformly
• e.g. GridSAM services, other Grimoires services
• Acts as a client managing these resources

29
Grimoires Overview

• What is Grimoires to end-users?
• An end-user shell and client-side APIs to
  interact with GridSAM Web Services
• For publishing entries
• For searching for entries
• Can attach arbitrary metadata to entries

30
UDDI

• Grimoires service is an UDDIv2 interface
• Service-Oriented Architecture
• Universal Description, Discovery and Integration
• The service publication and discovery
  specification adopted by WS community
• The latest version is v3 (v2 is still widely
  used)
• The UDDI standard defines
• The data model to describe services
• The API to publish/discover services

31
UDDI Data Model