Java and Python

1

• Java and Python
• CoG Kits

2
Chapter Outline

• Introduction to Grids
• What is a Grid?
• What is the Globus Toolkit?
• What is a Commodity Grid Kit?
• Using and Programming Grids with the JavaTM and
  Python CoG Kits

3
Integration of High-end Resources
Supercomputers
Software catalogs
Sensor nets
Colleagues
Data archives
On-demand creation of powerful virtual computing
systems
4
Grid Computing and Existing Technologies

• Commonalities between Grid computing and major
  industrial thrusts
• Business-to-business, peer-to-peer, application
  service providers, storage service providers,
  distributed computing, Internet computing
• Differences between Grid computing and existing
  technologies
• Complicated requirements Run program X at site
  Y subject to community policy P, providing access
  to data at Z according to policy Q
• High performance unique demands of advanced
  high-performance systems

5
What challenges do we have to solve?

• Authenticate once
• Specify simulation (code, resources, etc.)
• Locate resources
• Negotiate authorization, acceptable use, etc.
• Acquire resources
• Initiate computation
• Steer computation
• Access remote datasets
• Collaborate on results
• Account for usage

Domain 1
Domain 2
6
An Example ApplicationAn Advanced Scientific
Instrument
Virtual Reality Cave
Advanced Photon Source
Scientist
Avatar
Supercomputer
Electronic Library and Databases
Computing Portal Clients
7
The Globus Toolkit

• The Globus Toolkit provides a range of basic Grid
  Services
• Security, information, fault detection,
  communication, resource management
• These services are simple and orthogonal
• Independently use mix and match
• Programming model independence
• For each service there is generally a
  well-defined API
• Standards are used extensively
• E.g. LDAP, GSS-API, X.509

8
Globus Approach

• A toolkit and collection of services addressing
  key technical problems
• Modular bag of services model
• Not a vertically integrated solution
• General infrastructure tools (aka middleware)
  that can be applied to many application domains
• Interdomain issues, rather than clustering
• Integration of intradomain solutions
• Distinction between local and global services

9
Globus Hourglass

• Focus on architecture issues
• Propose set of core services as basic
  infrastructure
• Use to construct high-level, domain-specific
  solutions
• Design principles
• Keep participation cost low
• Enable local control
• Support adaptation
• IP hourglass model

A p p l i c a t i o n s
Diverse global services
Core Globus services
Local OS
10
Layered Grid Architecture
Applica- tion
Application
Internet Protocol Architecture
Collective
Resource
Connectivity
Transport
Internet
Link
Fabric
11
Production Grids Testbeds
GUSTO Testbed
The Alliance National Technology Grid
NASAs Information Power Grid
12
Key Grid Services

• Resource Discovery
• List all of the solaris machines with at least 16
  processors and 2 Gigs of memory
• Resource Acquisition
• Submit my job to this machine
• Data Management/Movement
• Locate, or create, a replica of this data set
• Move this data set to this host
• Security
• Authentication
• Authorization

13
Grid Security Infrastructure

• Standards based infrastructure that provides
• Single sign-on
• Authentication data integrity/confidentiality
• Delegation
• Based on
• X.509 Proxy certificates (draft ietf pkix
  standard)
• TLS
• GSS-API

14
Delegation in GSI
Client
GSI Proxy
3) Submit Job
5) Retrieve Data Set
Compute Resource
Tertiary Storage
6) Run computation
15
Definition CoG Kit

• A Commodity Grid (CoG) Kit defines and implements
  a set of general components that map Grid
  functionality into a commodity environment.
• Web/CGI, Java, Python, CORBA, DCOM, ....
• CoG Kits help us build applications, Problem
  Solving Environments, and Portals.
• CoG Kits take the good things from framework
  Grid

16
To further elaborate

• Computational Grids
• Globus Infrastructure
• Commodity Technologies
• CORBA, Java, Python, Perl, Web technologies, ...
• Science Application
• Computing Portal
• Problem Solving Environments
• Workbenches

17
Motivation Java Python CoG Kits

• Use and leverage existing technologies for Grid
  programming
• - The capabilities of the framework onto which
  Grid Services are mapped can be exploited
• Objects, Events, Exceptions, JNDITM, ...
• - Objects like jobs/tasks can be defined.
• - XML support is provided.
• - GUI's, ...., IDE's can be used (Forte, BOA
  Constructor)
• Maximize software flexibility, extensibility, and
  reusability
• Provide foundations for application developer
  teams that are familiar to develop applications
  in this framework
• - Reduce development and maintenance cost
• Use as glue for many technologies
• Python is well suited to tying together many
  different languages/technologies.

18
What is the Java CoG Kit ?

• The Java CoG Kit provides a mapping between Java
  and the Globus Toolkit. It extends the use of
  Globus by enabling to access advanced Java
  features such as events and objects for Grid
  programming.
• The Java CoG Kit is implemented in pure Java. It
  speaks the Grid protocols.
• It is not a wrapper of the C Globus Toolkit
• This allows integration within applets.
• Mostly client side support

19
What is the Python CoG Kit?

• Similarly the Python CoG Kit provides a mapping
  between Python and the Globus Toolkit. It extends
  the use of Globus by enabling to access advanced
  Python features such as events and objects for
  Grid programming.
• The Python CoG Kit is implemented as a series of
  Python extension modules that wrap the Globus C
  code.
• Uses SWIG (http//www.swig.org) to help generate
  the interfaces.

20
Status Java CoG Kit

• Modified core Globus components (Protocols)
• Basic services are provided accessing
• Security
  (GSI)
• Remote job submission and monitoring (GRAM)
• Quality of service
  (GARA)
• Remote Data Access (GSIFTP)
• Information Service Access (MDS)
• Certificate store
  (myProxy)
• Current 100 client side components includes
• Reusable Grid GUI components

21
Status Python CoG Kit

• Basic services are provided accessing
• Security
  (security)
• Remote job submission and monitoring
  (gramClient)
• Gass Server (gassServerEZ)
• Secure high-performance network IO (io)
• Protocol independent data transfers (gassCopy)
• High performance Grid FTP transfers (ftpClient)
• Support for building Grid FTP servers (ftpControl)
  
• Remote file IO (gassFile)
• Replica Catalog (replicaCatalog)
• Replica Management (replicaManagement)
• GSI enabled SOAP (GSISOAP)

22
Python CoG Kit basics

• Everything is contained in the pyGlobus package.
• The low-level C-wrappers are all named modulec,
  e.g., ftpClientc, gramClientc
• Shouldnt need to access these directly, instead
  use the Python wrappers (ftpClient, gramClient)
• Exceptions are used to handle error conditions
• pyGlobus.util.GlobusException is the base class
  for all pyGlobus exceptions. It extends the base
  Python exceptions.Exception class
• The Python wrappers classes manage the underlying
  memory

23
Building the Python CoG Kit

• Uses standard GNU tools (autoconf, make)
• Download the latest source from
  http//www-itg.lbl.gov/Grid/projects/pyGlobus/
• Tar zxvf pyGlobus-VERSION.tar.gz
• cd pyGlobus mkdir build cd build
• ../configure
• Use with-flavorltGlobus flavor stringgt to
  override the default flavor
• Use with-pythonltpath to pythongt to override the
  default python installation
• make make install
• Installs into your site-packages directory (No
  need to edit PYTHONPATH)

24
Many more CoGs

• Perl
• JSP
• CORBA
• .
• GUIs
• Applications

25
Java CoG Requirements

• JDK 1.2
• Security
• Needs security package such as IAIK
• We are able to replace security packages based on
  a couple of common functions with public domain
  package.
• We hope that these package will reach the
  maturity of iaik

26
Python CoG Requirements

• Python 2.0
• The GT2-beta (or better) release of Globus.
• Support for dynamic libraries.
• Currently pyGlobus is tested on
• Linux
• Solaris
• Should compile
• Will support win32 when Globus is ported

27
References

• Java CoG Kit
• http//www.globus.org/cog.
• Python CoG Kit
• http//www-itg.lbl.gov/Grid/projects/pyGlobus
• Globus
• http//www.globus.org
• SWIG
• http//www.swig.org
• krjackson_at_lbl.gov
• gregor_at_mcs.anl.gov

28
Acknowledgements

• This research is supported by the Mathematical,
  Information, and Computational Science Division
  subprogram of the Office of Advanced Scientific
  Computing Research, U.S. Department of Energy,
  under Contract W-31-109-Eng-38 and under Contract
  DE-AC03-76SF00098 with the University of
  California. Globus research and development is
  supported by DARPA, DOE, and NSF.