GridPort 3

1
GridPort 3

• Maytal Dahan
• maytal_at_tacc.utexas.edu
• Distributed and Grid Computing Group
• Texas Advanced Computing Center
• The University of Texas at Austin
• Monterey Grid Computing Conference
• August 10th 2004

2
Overview

• GridPort and Grid Computing
• What is Globus?
• What is GridPort 3?
• GridPort Components Informational Services,
  Interactive Services
• Interfaces to GridPort Portals and More
• GridPort Demonstration Portal

3
GridPort and Grid Computing
4
GridPort and Grid Computing

• Grid computing must provide basic functions
• resource discovery and information collection
  publishing
• data management on and between resources
• process management on and between resources
• common security mechanism underlying the above
• process and session recording/accounting
• Current grid computing tools such as Globus
  provide most of the above at some level
• It is incomplete
• New web service based-standard will help current
  tools become interoperable.

5
GridPort and Grid Computing

• GridPorts Role in Grid Computing
• GridPort is middleware which aids developers by
  easing use of low level grid tools
• GridPort aggregates grid services from grid
  software packages
• Provides grid capabilities while presenting a
  simple, consistent API for portal and
  applications developers.
• Simplifies portal development which
• lowers the barrier of entry for grid users
• Assists users in accessing resources on the grid
• Aggregates different services into one interface
• event-based model and assists in automating
  computing tasks

6
What is Globus?
7
What is Globus Toolkit 3 (GT3) ?

• Open source toolkit for building grids
• Low-level Middleware and grid services
• OGSI Based migrating to WSRF
• What is OGSI?
• Open Grid Services Infrastructure
• Set of specifications for grid Computing
• GT3 is an implementation of OGSI (grid services)
• This is being replaced by Web Services Resource
  Framework (WSRF)
• Similar functionality to OGSI but adhering to web
  services standards
• Includes Globus 2 services (Pre-OGSI)

8
GT3 Components

• 4 Major components to GT3
• Security (GSI)
• Data management (GridFTP, RFT)
• Resource Management (MMJFS)
• Information Services (Index Services)
• More information can be found at
  http//www.globus.org

9
Security

• Globus uses Grid Security Infrastructure (GSI)
• GSI Authentication has the following components
• Certificate Authority (CA)
• Certificate
• Private Key
• Proxy

10
Security Certificate and Key

• Grid certificate is like a passport
• Allows single sign on authentication
• Certificate /.globus/usercert.pem
• Key /.globus/userkey.pem
• Globus uses these certificates to authenticate
  users

11
Security Proxies

• Proxy gives you single sign on capabilities for
  12 hours by default
• Log on to any grid resource without re-entering a
  password
• Users have a Distinguished Name (DN) which is
  their certificate identity
• The DN must be in a Globus Grid Mapfiles on the
  remote resources users want to access

12
Data Management

• GridFTP
• GridFTP is a high-performance, secure, reliable
  data transfer protocol optimized for
  high-bandwidth wide-area networks.
• GSI Security on control channels and data
  channels
• Built on FTP
• Put, Get, Third Party Transfer
• Reliable File Transfer (RFT) is a grid service
  implementation of GridFTP.

13
Resource Management

• Master Managed Job Factory Service (MMJFS)
• Job Submission
• Batch Job submission submit a job to a local
  scheduler and return a job handle
• Job Handles URLs known as Grid Service Handles
  (GSH)
• Interactive Job Submission execute a command
  and wait for results
• Job Status query a jobs status using the GSH

14
Information Services

• Data is gathered using informational grid
  services
• GT3 uses Index Services to aggregate data from
  informational grid services
• Individual grid services can register with an
  index service
• Index services can be arranged hierarchically

15
GridPort and Globus

• GridPort uses GT 3.0.2 and currently upgrading to
  GT 3.2
• Will evolve to WSRF when GT4 comes out at the end
  of January 2005
• Resources for Globus 3.0
• GT 3 Quickstart by IBM http//publib-b.boulder.ib
  m.com/Redbooks.nsf/RedpaperAbstracts/redp3697.html
  ?Open

16
What is GridPort 3?
17
What is GridPort?

• High-Level middleware introduced between the
  low-level grid services offered by systems such
  as Globus and the interface layer such as
  portals.
• This layer can fundamentally transform the ease
  and speed with which user-interface developers
  can bridge the gap between end-users and a grid.

18
GridPort 3 Status

• Current Release GridPort 3.0
• Incremental GridPort release (GP3.0.2) this week.
  It has some minor bug fixes and improvements.
• http//www.gridport.net - download, project
  description, user guide, mailing list, etc.
• GridPort is in the current NMI Grids release (NMI
  R5) and NPACKage

19
GridPort Architecture
20
GridPort Components

• Informational Services
• GridPort Information Repository (GPIR)
• VO and resource based information repository
• Administration client manages resources and VOs
• Interactive Services
• Authentication
• Batch Job Submission
• Command Execution
• File Transfer
• Advanced Features
• Job Sequencer
• Platform Community Scheduling Framework (CSF) Job
  Submission
• Demonstration Portal

21
GridPort Components Informational Services,
Interactive Services
22
Informational Services GPIR

• Clients, such as portals, will frequently need
  information regarding the state of a grid
• GridPort Information Repository (GPIR) is used to
  store and retrieve information about the grid
• GPIR is a relational database (PostgreSQL) based
  system
• Query and Ingestion functionality is accessible
  via a Web Services interface

23
GPIR

• Stores two broad types of data
• Dynamic grid status data
• Load, Node, Resource Status, Jobs, NWS, MOTD
• Human-centric data
• Virtual organizations (VO), Resource data,
  Contacts, Sites, Institutions, Departments

24
GPIR Interfaces

• Web Services
• Query
• Allows WS-based queries of grid data
• Reads from the database
• Ingester
• WS-based input of dynamic grid data (e.g. Job
  Data)
• Writes to the database
• Administration Client
• Web-based GUI for the maintenance of
  human-centric grid data

25
Getting Information into GPIR

• In order to take advantage of GPIR, it must be
  populated with grid data
• Ingester Web Service
• Dynamic grid data such as machine load will be
  populated via the Ingester web service
• The Ingester is written to via Information
  Providers that run on or poll the resources
  themselves
• Administration Client
• For infrequently changing data such as resource
  names, use the web-based GPIR Admin client

26
Administration Client

• Go to the web client and edit data directly

27
Information Providers and the Ingester WS

• Sends XML data (job, node, motd, etc.) to GPIR
  web services
• Run as cron jobs at the desired periodicity
• Sample information providers are included with
  the GridPort download in the examples directory
• The IP address of any provider writing to the
  Ingester Web Service must be registered with GPIR
  (called a client) for a given resource in order
  to provide data for that resource
• This registration can be achieved through the
  Administration Client.

28
GPIR Web Services
Information Providers
Resources
Clients
dB
Portals
Perl Client
Portlets
edu.tacc. gridport.gpir
Java Client
Ingester WS
Query WS
MDS
GPIR PostgreSQL
Other Middleware
OGSA (Future)
Web Scraping
SOAP-XML
HTTP
Other
JDBC
29
Interactive Services

• Authentication
• Using the GridPort repository
• Using Myproxy
• File Management
• File Listing
• File Transfer
• Job Submission
• Command Execution
• Batch Job Submission
• CSF Job Submission
• Sequencer

30
Interactive Services - Authentication

• What does logging in mean in GridPort?
• GridPort can get a proxy to authenticate the user
  during a session
• GridPort creates a session for the user
• 2 ways to authenticate a user in GridPort
• GridPort Repository
• Myproxy

31
Interactive Services - Authentication

• Using the GridPort Repository
• Repository has 2 main tasks
• Store a users credentials
• These credential files are stored according to
  GridPort username.
• Example ltusernamegt_cert.pem and
  ltusernamegt_key.pem
• User logs in to GridPort with username and
  private key password
• Maintain state of a user
• When a user logs in a session ID is created
• The proxy for the session is named by the session
  ID
• A session file for maintaining state is created
• The session file is named with the session ID

32
Interactive Services - Authentication

• Using MyProxy (http//grid.ncsa.uiuc.edu/myproxy/)
  
• Myproxy is an online credential repository
• User stores a proxy on a Myproxy server for a 7
  day period
• User can log in to GridPort to by retrieving
  credentials stored in MyProxy
• It is up to the user to store credentials in
  Myproxy
• This is for more grid savvy users who dont want
  to give GridPort a copy of their credentials
• What happens when a user logs in to GridPort
  with Myproxy?
• GridPort checks out credentials from Myproxy
• Generates a session file for this user

33
Interactive Services File Management

• File Listing
• Used to list the files in a users home directory
  or any subdirectory
• File Transfer
• User can execute a GridFTP put transfers a file
  from the GridPort server to a remote resource
• User can execute a GridFTP get transfers a file
  from a remote resource to the GridPort server
• User can execute a GridFTP third party transfer
  transfers a file between two remote resources

34
Interactive Services Job Submission

• Command execution
• Allows a user to execute a command on a remote
  resource
• Returns results immediately
• Batch job submission
• Allows a user to submit a batch job to a
  scheduler on a remote resource
• Returns a job handle which can be used to obtain
• Job status
• Job results

35
Interactive Services - CSF

• What is CSF?
• Platform Computing developed an Open Source
  Community Scheduler Framework (CSF)
• CSF is a meta-scheduling framework implemented as
  an OGSI grid service which schedules jobs across
  a grid
• The CSF meta-scheduler schedules the job on a
  particular resource based on a scheduling
  plug-in.
• How to obtain CSF
• Via Platform computing at http//www.platform.com/
  products/globus/
• Via GT3 source download in the scheduler
  directory http//www-unix.globus.org/toolkit
• GridPort can submit batch jobs to CSF which will
  pick an appropriate resource and submit the job

36
Interactive Services - Sequencer

• Allows a user to construct a sequence of steps
• File transfer
• CSF job submission
• File transfer from the host CSF chose (if this
  step occurs directly after a CSF job submission)
• Interface exposed as a web service
• Uses a Sequencer session to maintain state and
  authentication of the sequence even after the
  user has logged out of the portal
• Tentative plans to add batch job submission and
  command execution
• This requires a working instance of GPIR

37
Interfaces to GridPort Portals and More
38
Why are GridPort Interfaces Useful?

• Need a common method for exploiting the
  functionalities of GridPort
• A user can interact with multiple remote
  services, devices etc. through one common
  interface that ties them all together
• Can have multiple types of interfaces to access
  the same functionality but in different ways
• A well designed interface should make complex
  tasks easier to understand and execute

39
Who Uses the GridPort Interfaces?

• Users of the GridPort interfaces include
• machine operator who monitors the operational
  status of resources
• scientific researcher who needs to submit jobs
  and move files between multiple resources
• public relations staff member who needs an
  simple, visible example of how complex backend
  software applications work
• And many others.

40
Types of Interfaces
41
Portal

• Easily accessible by anyone with a Web browser
• Low resource overhead for the user
• Access from different types of devices (desktop,
  laptop, PDA, cell phone)
• Centrally managed by portal development team
• Different types of portals User Portal,
  Application Portal

42
Portal Interface Example
43
Web Service

• SOAP over HTTP provides implementation-agnostic
  access to GridPort functionality through the Web
  Service interfaces
• Web services allow applications to communicate
  with each other in a platform- and programming
  language-independent manner.
• Web services and clients must both speak SOAP

44
Other Interfaces

• Shell
• Interact with grid services from the command line
  interface
• A grid shell environment can be implemented to
  utilize GridPort APIs
• Desktop Application GUI
• Client interface that runs in a users desktop
  environment and uses GridPort API to get
  information about and interact with grid services
  and resources.
• Can provide a more complex interactive features
  than other types of interfaces but requires
  action on the part of the user to install and
  configure the client

45
GridPort Demonstration Portal
46
Portal Example GridPort Demo Portal

• Demo portal based on Jetspeed Portlet Framework
• Portlet Pluggable, portal component interfaces
  accessible through a web browser
• Easy aggregation of different grid interactive
  and informational services into a portal
  interface
• Configurable parameters that are unique to each
  users session
• Customizable look and feel
• A portal is made up of multiple portlets which
  can be added or removed depending on user
  customization

47
Portal Components

• Demo Portlets
• GPIR
• GPIR Historical
• Authentication
• File Management
• Command Execution
• CSF Job Submission
• Job Sequencer

48
Demonstration Portlets

• GPIR Portlet
• Displays informational data in GridPort according
  to VO
• 3 Main Views
• Systems View
• Summarizes static and dynamic data for resources
  in a VO
• Grid View
• Presents information about a VO as a unit,
  presenting total capabilities on a grid, network
  bandwidth and latency
• Machine View
• Machine specific information

49
(No Transcript)
50
Demonstration Portlets

• GPIR Historical Portlet
• Presents a visual summary of archived data in
  GPIR
• User can view graphs of data such as machine
  load, jobs run, node information, and status
• Can specify range of dates to display data
• Example can display a graph of machine load for
  resource X over the past month

51
Demonstration Portlets

• Authentication Portlet
• Handles Login/Logout
• Includes Myproxy Authentication
• Extends the authentication already in Jetspeed
  and integrated GridPort authentication
• File Management Portlet
• Allows a user to manage files on remote resources
• Capabilities
• List Remote Files and Directories
• Upload Download files to and from a resource
• Transfer files between resources

52
Demonstration Portlets

• Command Execution
• Enables the user to execute a command on a remote
  resource
• Has the option for command line arguments
• Batch Job Submission
• Enables user to submit a batch job to a remote
  resource and check job status
• Keeps track of previous job submissions

53
Demonstration Portlets

• Grid Job Submission Portlet
• Allows users to submit jobs to CSF
• Keeps track of submitted jobs and job status
• Job Sequencer Portlet
• Allows construction of a sequence of tasks
• Currently allows CSF Job submission and File
  Transfer
• Tracks previous sequences submitted and the
  status of current sequences

54
Conclusion

• What does GridPort offer to this community ?
• For developers
• Ease portal development by using GridPort API to
  complex Grid software
• Rapid development of user and application based
  portals for various communities
• For Users
• Portal interface to lower the barrier of entry to
  grid computing
• Provide benefits over traditional scientific
  computing by using capabilities that portals
  offer job tracking, data management, workflow,
  event based tasks, visualization in one single
  interface
• Please contact us, we would be interested in
  gathering requirements from this community.

55
More About GridPort

• For documentation, release information, training,
  etc. http//www.gridport.net
• To receive project announcements subscribe to
• gridport-announce_at_tacc.utexas.edu
• Maytal Dahan, Project Manager
• maytal_at_tacc.utexas.edu
• More About TACC
• Texas Advanced Computing Center
• http//www.tacc.utexas.edu
• info_at_tacc.utexas.edu