GT 3 Base Services Inside Information Services

1
GT 3 Base Services Inside- Information Services

• KISTI Supercomputing Center
• Grid Technology Research Department
• Eunsung Kim
• eskim74_at_kisti.re.kr

2
Contents

• Overview
• Components
• Service Data Provider Components
• Service Data Aggregation Components
• Registry Components
• Applications
• GT3 Index Service
• GRAM RIPS
• MMJFS
• ContainerRegistry and VORegistry

3
Overview
4
GT3 Information Services

• What is GT3 Information Services?
• Grid service which provides information about
  Grid resources
• modular Java component framework for OGSA
• service developers can use to implement various
  information management solutions for
  GT3-compatible OGSA Services and Service Data

5
GT2 vs. GT3 Information Services

• Components
• MDS ? Index Service, Service Data Providers and
  Aggregators, Query and Notification Framework
• GRAM Reporter ? Resource Information Provider
  Service
• Data Format
• LDIF ? XML
• Data Source
• GLUE providers ? GLUE providers, Service Instance
• Query Mechanisms
• LDAP ? XPath, XQuery

6
Service Data

• A Grid service instance maintains a set of
  service data elements (SDE)
• Declared via an extended XSD element declaration,
  placed in a WSDL portType
• Values carried in a portType or in an instance
• Includes basic introspection information,
  interface-specific data, and application state
• Pull and push models for information query
• GridServiceFindServiceData operation
• Pull queries this information via extensible
  query language
• NotificationSourceSubscribe
• Push Subscribe to notification of changes to
  information

7
Why Service Data?

• Discovery often requires instance-specific,
  perhaps dynamic information
• Service data offers a general solution
• Every service must support some common service
  data, and may support any additional service data
  desired
• Not just meta-data, but also instance state
• Part of GT MDS-2 model contained in OGSI
• Defines standard data model, and query op
• Complements soft-state registration and
  notification

8
OGSI ServiceData Model

• ServiceData for self-description
• Model service with properties!
• Fine-grained view of resource functionality
• Data scoped by service instance
• Domain-dependent state
• Service discovery/monitoring information
• Stateful properties of service, e.g. cpu-load
• Infrastructure state
• Introspection for client tooling, e.g. handles
• Parameters of generic features
• ServiceDataNames introspects available data

9
Components
10
Basic Components

• Service Data Provider Components
• Service Data Aggregation Components
• Registry Components

11
Service Data Provider Components

• Service Data Provider components
• provide a standard mechanism for dynamic
  generation of service data via external programs
• External provider programs
• can be the core providers that are part of GT3 or
  
• can be user-created, custom providers

12
In Detail Service Data Providers

• Service Data Provider interfaces are designed to
  support execution in either
• synchronous (pull) mode
• asynchronous (push) mode
• A valid provider
• is composed of any Java class which implements at
  least one of three predefined Java Interfaces
• generates a compatible form of XML output as the
  result of its execution

13
In Detail Service Data Providers

• Provider Interfaces
• SimpleDataProvider
• synchronous provider which produces XML output in
  the form of a Java OutputStream
• DOMDataProvider
• synchronous extension of SimpleDataProvider which
  can also produce XML output in the form of a Java
  org.w3c.dom.Document
• AsyncDataProvider
• asynchronous version of SimpleDataProvider
  sending the output to the specified callback
  Object, which is assumed that the provider
  implementer and the provider caller have both
  agreed on the callback interface at compile-time

14
In Detail Provider Interfaces

• SimpleDataProvider
• Basic interface which all service data providers
  must implement

15
In Detail Provider Interfaces

• DOMDataProvider
• Generic interface for XML service data providers
  that are capable of emitting a org.w3c.dom.Documen
  t object at runtime

16
In Detail Provider Interfaces

• AsyncDataProvider
• Asynchronous version of provider interface

17
In Detail GT3 Providers

• AsyncDocumentProvider
• An asynchronous version of a generic XML document
  provider
• ScriptExecutionProvider
• ServiceDataProvider that provides a generic way
  to execute scripts which produce XML documents
• HostScriptProvider
• Constructs Host service data from the output of
  multiple scripts

18
In Detail GT3 Providers

• ForkInfoProvider
• ServiceDataProvider which monitors local system
  PIDs
• PBSInfoProvider
• ServiceDataProvider which queries PBS for queue
  information
• SimpleSystemInformationProvider
• Basic MDS GRIS-sytle sensor which emits system
  information in XML, with state managed directly
  as an XML Document using JDOM - JDK 1.3 compatible

19
In Detail Provider Manager

• Provider execution
• is handled by the ServiceDataProviderManager
  class, which schedules and manages provider
  execution as Java TimerTasks
• ServiceDataProviderManager
• uses an XML-based configuration file to load and
  link installed Service Data Providers during
  runtime through standard Java reflection methods
• Configuration file
• GLOBUS_LOCATION/etc/indexservice.providers
• GLOBUS_LOCATION/etc/rips.providers

20
In Detail Provider Manager

• Configuration entry for the provider in a
  configuration file
• enables your provider for execution by the
  Provider Manager
• publishes the existence of your provider to
  clients
• Required attribute in the configuration entry
• the class attribute, which is simply the fully
  qualified Java class name

21
In Detail Custom Data Handlers

• The default data processing behavior of the
  Provider Manager
• take the logical XML document result of a
  providers execution
• wrap it in a new SDE
• and then add it to the Services ServiceDataSet
• We can override the default data processing logic
  in the Provider Manager
• by specifying the handler attribute in the
  Providers configuration file

22
In Detail Custom Data Handlers

• Custom callback routine
• provides a mechanism for facilitating sorting,
  merging or other data preprocessing and
  transformation
• allows the developer complete control for output
  data
• Use case GRAM RIPS
• uses a custom callback handler for the Fork and
  PBS providers
• to search, sort and atomize child Job elements
  from the logical result document into new SDEs

23
In Detail ProviderExecutionPortType

• enumProviders

24
In Detail ProviderExecutionPortType

• executeProvider

25
In Detail Mechanisms
Service or User
Information Providers
SimpleDataProvider
enumProvider
Provider Manager
DomDataProvider
executeProvider
AsyncDataProvider
Custom Data Handler
26
Basic Components

• Service Data Provider Components
• Service Data Aggregation Components
• Registry Components

27
Service Data Aggregation Components

• ServiceDataAggregator components
• provide a reusable mechanism for handling
  subscription, notification, and updating of
• locally stored copies of service data which is
  generated by other services
• By using the ServiceDataAggregator class in your
  service code
• service Data from both locally executing
  information providers and other OGSA service
  instances can be aggregated into any given service

28
In Detail Service Data Aggregator

• ServiceDataAggregator component
• is used to perform server-side notification
  subscription management similar to the
  SinkListener sample
• Key additional feature of ServiceDataAggregator
• notification data that is processed by the
  deliverNotification() function
• is actually copied and stored locally as a SDE,
  which includes creation timestamp, TTL and source
  metadata

29
In Detail Service Data Aggregator

• Aggregated SDEs are
• organized by SDE QName
• stored in a array which is returned as a
  org.gridforum.ogsa.ServiceDataSetType to
  FindServiceData name queries
• Originator field (GSH type) of the OGSA
  ServiceDataType
• is used as the primary key
• to differentiate like-named entries from each
  other
• to identify the source of the data itself

30
In Detail AggregatorPortType

• addSubscription and removeSubscription

31
In Detail Mechanisms
Service or User
Grid Services
addSubscription
Aggregator
deliverNotification

removeSubscription
32
Basic Components

• Service Data Provider Components
• Service Data Aggregation Components
• Registry Components

33
Registry Components

• Registry components
• maintain a set of available peer Grid Service
  Handles
• provides soft-state cataloging of a set of Grid
  Services
• i.e., the registry of services is periodically
  updated with existence notification messages and
  any existing entries which fail to refresh within
  the timeout period are eventually expired
• Registries
• can be used to support query or other operations
  that may apply to one or more services in a set

34
In Detail RegistrationPortType

• registerService and unregisterService

35
In Detail Mechanisms
Grid Services
registerService
Registry

unregisterService
36
Applications
37
Applications Putting it All Together

• Various GT3 Information Services components
• provide a convenient way to address common and
  recurring tasks with Service Data in the GT3 OGSA
  environment
• One or all of the components
• can be used to manage data depending on
  application needs

38
Applications Putting it All Together

• Current GT3 OGSA services using Information
  Services components
• GT3 Index Service
• ServiceDataProviderManager, ServiceDataAggregator,
  and Registry components
• GRAM Resource Information Provider Service (RIPS)
• ServiceDataProviderManager components
• GRAM Master Managed Job Factory service (MMJFS)
• ServiceDataAggregator components
• ContainerRegistry and VORegistry
• Registry components

39
Use Case GT3 Index Service

• GT3 Index Service
• combines the ServiceDataProviderManager
  components along with the ServiceDataAggregator
  and Registry components
• creates a dynamic data-generating and indexing
  node, similar in concept to GT2 MDS GRIS/GIIS
• is implemented as a ServiceSkeleton so that
  developers can easily customize and extend its
  functionality
• Registry and ServiceDataAggregator components
• can interconnect multiple GT3 Index Services
  together into arbitrarily complex topologies to
  form both Virtual Organizations and Virtual
  ServiceDataSets

40
Use Case GT3 Index Service

• Key capabilities
• An interface for connecting external information
  provider programs to service instances
• A generic framework for aggregation of service
  data
• A Registry of Grid services
• Java package org.globus.ogsa.impl.base.index

41
Use Case GT3 Index Service
User
Service Data
Service Data
GSH
Index Service
Aggregator Mechanism(caching here)
Provider Mechanism
Registry Mechanism
Java Provider
Collective Layer
Existence Notification Message
Notification Message
Service Data
Service Data
Service Data
Grid Service
Grid Service
Grid Service
Resource Layer
42
Use Case GRAM RIPS

• GRAM Resource Information Provider service
• uses the ServiceDataProviderManager components,
  along with custom providers and the OGSA core
  component NotificationServiceSkeleton
• monitors forked processes, PBS scheduler queues,
  and local host system statistics
• polls the host system periodically for this
  information since changes to the local hosting
  environment are inherently non-deterministic
• Java package org.globus.ogsa.impl.base.gram.rips

43
Use Case GRAM RIPS

• Execution frequency of this polling
• is specified on a per-provider-instance basis
• is specified by the client when the provider is
  executed the first time via the
  ProviderExecutionPortType executeProvider()
  method
• Sweeper thread
• periodically traverses the services internal SDE
  list and invalidates any entries which have not
  received updates to the timestamp metadata

44
Use Case GRAM RIPS

• Special processing in its custom jobDataHandler
  function
• breaking out specific child Job elements from
  the logical result document
• creating individual SDEs for each, assigning
  appropriate TTL metadata to new entries and
  updating the TTL data on existing SDEs
• In order to detect whether a notification is
  warranted based on a state change
• examines the state of each Job element during the
  callback

45
Use Case GRAM RIPS

• RIPS tuning
• global configuration parameters that can be set
  in the RIPS deployment descriptor in
  server-config.wsdd

46
Use Case GRAM RIPS

• RIPS tuning (cont.)
• The sweeperDelay value
• is the amount of time in milliseconds for the
  Sweeper thread to wait between execution cycles
• should generally be at least as long as the
  average amount of time spent in the
  jobDataHandler callback
• The gracePeriod
• is the amount of time in milliseconds to add to
  the entries GoodUntil TTL value to generate its
  AvailableUntil value
• If the sweeper encounters an entry whose
  AvailableUntil value is expired, it deletes the
  entry
• The GoodUntil TTL value is set by default to be
  the provider execution frequency in milliseconds

47
Use Case MMJFS

• GRAM Master Managed Job Factory service
• exposes aggregated service data to GRAM clients
  by using the ServiceDataAggregator
• to subscribe to one or more RIPS installations
  for information on running jobs and queue status
• Java package org.globus.ogsa.impl.base.gram.mmjfs

48
Use CaseContainerRegistry and VORegistry

• ContainerRegistryImpl
• This class implements a registry for all services
  in a container
• Java package org.globus.ogsa.impl.base.registry
• VORegistryImpl
• This registry implements a repository of services
  within a virtual orginization
• It supports local as well as remote service
  publication and lookup
• Java package org.globus.ogsa.impl.samples.registr
  y

49
Q A

• Thanks for Coming!