OGCE Overview: Services, Gadgets, and Tags

1
OGCE Overview Services, Gadgets, and Tags

• Marlon Pierce, Suresh Marru, Gregor von
  Laszewski, Mary Thomas, Maytal Dahan, Gopi
  Kandaswamy, Rion Dooley, and Wenjun Wu

2
Links for More Information

• Website http//www.collab-ogce.org
• News/RSS/Blog http//collab-ogce.blogspot.com
• Email discuss_at_ogce.org
• SVN Code Repository http//ogce.svn.sourceforge.n
  et/viewvc/ogce/
• And get tar.gzs
• Source Forge http//sourceforge.net/projects/ogce

3
Project Description Status
Grid Portal Packaged, one step installation of Grid portlets into TomcatGridsphere Stable, V2.4 for SC08
Axis Services One step install of Resource Discovery and Prediction Services. More services to come. Dev. Release for SC08.
Workflow Suite Support for scientific workflows, based on LEAD tools. Suresh will describe next. Dev. Release for E-Science
GTLAB JSF Tag Libraries for building Grid portlets and gadgets Dev. Release for SC08
JavaScript COG (Cyberaide) JavaScript libraries for building Grid gateways. Dev. Preview for SC08
Gadget Container Open Social Inspired container for iGoogle gadget Dev. Preview for SC08
Incubator projects Pylons portals, metascheduling services, bio-gadgets, etc. SVN if you dare
Services, Workflow, GTLAB, Cyberaide, and Gadget
container are part of an emerging stack
4
OGCE and Gateways

• We develop and package software for use by
  Gateways.
• A lot of this comes from active Gateways.
• GPIR TeraGrid User Portal
• Workflow tools LEAD
• Resource Discovery Service, File Browser Applet
  TGUPGridChem

Gateways
OGCE Software
5
GPIR for Facebook, iGoogle

• GPIR GridPort Information Repository, stores
  dynamic and static resource information for VOs
• Facebook
• Mini TeraGrid system monitor and full size system
  monitor
• Mini http//apps.facebook.com/mini_tgsysmon/
• Full Size http//apps.facebook.com/tgsysmon/
• Google
• Mini System monitor for iGoogle page
• To add visit
• http//www.google.com/ig/adde?moduleurlhttp//iol
  a.tacc.utexas.edu22080/gpir-app/minigpir.xmlnoca
  che1sourceimag

GPIR Facebook App
6
Open Life Science Gateway Bio-gadgets
Job Submission History gadget
ClustalW gadget
Blast gadget
7
OGCE Portal
8
Components for Science Portals

• OGCE is founded on the principal that portals
  should be built out of reusable parts.
• Key standard in our first phase the JSR 168
  portlet specification.
• Portlets can run in multiple containers
• uPortal, Sakai, GridSphere, LifeRay, etc.
• Allows us to build Grid specific components and
  deploy along side other goodies Sakai
  collaboration tools, contributed portlets, etc.
• Future Open Social compliant Google Gadgets

9
Downloading and Installing OGCE

• Portal installation is described in greater
  detail later in the tutorial.
• Key point for current purposes you can have a
  basic portal up and running in a few minutes.
• Installation consists of
• Updating properties in pom.xml to match your
  Grid.
• Running mvn clean install
• Thats it
• Well now look at some features.

10
OGCE GPIR portlet can interoperate with TeraGrid
and your own GPIR services.
11
Manage TeraGrid MyProxy credentials with the OGCE
ProxyManager portlets.
12
OGCE file management client portlets interact
with TeraGrid GridFTP servers.
13
General purpose batch and interactive job
submission to GRAM, WS-GRAM is supported.
14
Dashboard Portlet
The dashboard portlet allows users to track jobs
on the selected resource. The user can view
either his own set of jobs or get information on
all submitted jobs.
14
15
(No Transcript)
16
Queue forecasting portlets work with the NWS
QBETS to predict wait times and deadlines.
17
PURSe portlets manage user requests for portal
accounts and Grid credentials.
18
Condor and Condor-G
19
OGCE IFrame Portlet can be used to integrate
external sites.
20
Under the Hood Building Components

• Portlets can be built with our Apache Velocity
  bridge.
• Template or steal this code approach.
• OGCE portlets use the Java COG Abstraction Layer
  to access Grid services.
• Provides a buffer over different Grid toolkits
  and versions.
• Provides ability to compose graph-based
  workflows.
• Tutorial in itself
• We have also developed Grid Tag Libraries and
  Beans (GTLAB) to simplify this process.

21
Some OGCE Services

• New things available for download

22
Resource Discovery Service

• Extracted from the TeraGrid User Portal
• Contributed by Rion Dooley
• Axis2 Web service
• Integrates information from INCA, GPIR services.
• What machines are in my GridHHHH Cloud?
• What are the service names?
• Which ones are actually running?
• Works with TeraGrid deployed services, but you
  can use it with your own information service
  installations.

23
Resource Prediction Service (RPS)

• Predicts an optimal set of resources for running
  scientific applications
• Predicts a set of resources on which the sum of
  data transfer time, queue wait time and compute
  time for the application will be minimum
• Data transfer time Uses input data size and NWS
  info
• Queue wait time Uses QBETS info
• Compute time Uses application performance info
• Axis2 web service with a MySQL backend
• RPS is a subset of the Fault Tolerance and
  Recovery service used in LEAD and VGrADS

24
Packaging Services

• Everything builds with one command
• mvn clean install
• We use Apache Maven 2 for builds.
• Structured, modular builds and consistent layouts
• Build a single module
• mvn clean install f somemod/pom.xml
• We include Apache Tomcat in the download, so all
  you need is Java and MySQL.
• Suitable also for the NMI build and test system.
• Want the Javadoc? mvn javadocjavadoc

25
Upcoming Services

• XMC-CAT metadata cataloging service with JSDL
  sample implementation.
• See also our demo on Wednesday.
• Swarm Web Service for mass job submission and
  management.
• See also Sangmi Pallickaras talk Friday at 200
• http//escience2008.iu.edu/sessions/SWARM.shtml
• GPIR Web Service for information on your Grid.
  This is being repackaged older packaged version
  (but same code) is already available.
• Workflow suite services will be integrated in the
  future.

26
GTLAB

• Tag Libraries for building gadgets and portlets
  out of reusable parts.

27
Users Browser JavaScript
OGCE Gadget Container
iGoogle, Facebook
GTLAB Gadgets
TeraGrid Services
Other Services
RPS, RDS, GPIR Services
GTLABs Relationship to Other Components
28
MyProxy Example
lthform id"form"gt lthoutputText
value"Username"/gt lthinputText
value"resource.username" required"true"/gt
lthoutputText value"Password "/gt
lthinputSecret value"resource.password"
required"true"/gt lthoutputText
value"MyProxy Server "/gt lthinputText
value"resource.myproxyHostname"
required"true"/gt ltosubmit id"submit"
value"Submit" binding"builder.body"
action"builder.junkAction" gt
ltomultitask id"multi" persistent"true"
taskname"resource.taskname"gt
ltomyproxy id"mypr" hostname"resource.myproxyH
ostname" lifetime"2" password"resource.pa
ssword" port"7512" username"resource.user
name"/gt lt/omultitaskgt lt/osubmitgt
lt/hformgt
29
Executing Multiple Task
ltosubmit id"submit" value"Submit"
binding"builder.body" action"builder.j
unkAction" gt ltomultitask id"multi"
persistent"true" taskname"resource.tasknam
e"gt ltojobsubmit id"js" myfaces"true"
arguments"resource.arguments"
executable"resource.executable"
hostname"resource.hostname"
provider"resource.provider"
stdout"resource.stdout"/gt
ltofiletransfer id"ft" myfaces"true"
from"resource.from" to"resource.to"
/gt ltodependency id"dep" task"ft"
dependsOn"js"/gt lt/omultitaskgt lt/osubmitgt
30
RRRPS Client with GTLAB
ltosubmit id"submit" value"Submit"
binding"builder.body"
action"builder.junkAction"gt
ltomultitask id"multi" persistent"true"
taskname"resources.tasknamegt
ltorpsOptimal resource"task1-resources"
id"rpsoptimal-exec"
myfaces"true"/gt
lt/omultitaskgt lt/osubmitgt
31
Example with Dependency
ltosubmit id"submit gt ltomultitask
id"multi gt ltorpsAddModel
resource"task1-resources"
id"rpsAddModel-exec"
myfaces"true"/gt ltorpsManager
resource"task1-resources"
id"rpsManager-exec"
myfaces"true"/gt ltodependency
id"deps-rpsstuff task"rpsManager-exec"
dependsOn"rpsAddModel-exec"/gt
lt/omultitaskgt lt/osubmitgt
32
GTLAB Features

• Extends Java Server Faces.
• Tag components wrap major COG Abstraction Layer
  features
• And Web Service clients, GridShib, etc.
• Supports tag inter-dependencies.
• Allows you to do standalone development.
• Use JSF portlet bridge to convert into portlets
• No new coding, just add/modify XML config files
  and jars.
• Or forget about portlets and
• Convert into Google Gadgets
• Develop as a Facebook application
• Etc.

33
Installing GTLAB

• Download the release code from www.collab-ogce.org
  or check out from SVN
• svn checkout https//ogce.svn.sourceforge.net/svnr
  oot/ogce/GTLAB
• SourceForge also supports tar.gz downloads
• Tagged release in http//ogce.svn.sourceforge.net/
  viewvc/ogce/tags/
• Run mvn clean install in GTLAB directory.
• Download includes a web server.
• Start with ./startup.sh, go to
  http//localhost8080/GTLAB

34
Whats New with GTLAB?

• Completely revised dependency graph processing.
• Removed dependence on COG graph processing (still
  use the COG)
• We can now support graphs of any tag you care to
  write.
• Only support pipelines currently, not full DAGs
• Redesigned to simplify writing new tags.
• Extensive use of factories.
• You no longer have to change core code to add a
  tag
• Simplified process of creating new tags
• 4 Java files and 3 XML config files
• Now can be done in one (mvn) step
• Many new tags

35
Levels of Abstraction in GTLAB code
You can use prepackaged gadgets/portlets.
You can develop new applications using existing tag libraries.
You can make new tags for your services
You can hack or steal the code

• Note JSF is compatible with JSP, so you can mix
  and match.
• And you can embed JavaScript
• YUI, Scriptaculous user interface goodies.
• Google and Facebook APIs.

36
Tag Description
MyProxy Gets a proxy credential (COG)
JobSubmit Runs remote commands (COG)
FileOperation Create, delete, list remote files and dirs (COG)
FileTransfer GridFTP downloads, 3rd party transfers (COG)
XSubmissionOutput Format outputs of job and file operations
ResourceDiscovery Clients for the RDS
ResourcePrediction Clients for the RPS
MyCommunityProxy Get and decorate TG Community credentials with Grid Shib
Community Log Log use of TG Community credentials
37
GTLAB Example Description
MyProxyExample Get a credential and store in OGCE global proxy store.
JobSubmitExamples Various examples showing how to submit a grid job and get output.
Dashboard Examples Runs showq on Big Red, Cobalt, and Mercury and formats output.
DirectoryBrowser Displays and downloads files on a remote host via GridFTP
MultipleTaskExample Shows how to couple file transfers, operations, and
These will work as gadgets, portlets, or
standalone applications
38
GTLAB gadgets can also run as portlets with no
code changes by using the JSF portlet bridge.
iGoogle Client Aggregator
Tomcat Web Server Portlet Container
Tomcat Web Server
Compile and deploy into server
GTLAB Gadgets Code Base
GTLAB Portlets Code Base
Decorate with bridge and container jars, xml
config files.
39
Making GTLAB Google Gadgets

• A Gadget is a standalone application that is
  integrated into your personal iGoogle display.
• They can run on your Web server.
• You can use SSL/HTTPS
• Google Gadgets have two versions
• http these use Googles JavaScript APIs
• url these wrap external applications with
  IFrames.
• Limitations
• Not much real estate (but location, location,
  location)
• Cookies manage state on gadgets from the same
  server but we need a way to integrate multiple
  independent servers.

40
Example Gadget Config File
lt?xml version"1.0" encoding"UTF-8" ?gt
ltModulegt ltModulePrefs title"MyProxy Gadget
Example" scrolling"true"
height"500"/gt ltContent type"url"
href"http///8080/GTLAB/examples/MyProxyExamp
le.jsf"gt lt/Contentgt lt/Modulegt
Save this as MyProxy.xml and place in a Web
accessible place.
41
Gadget containers aggregate content from multiple
providers. Content is aggregated on the client
by the user. Nearly any web application can be a
simple gadget (as Iframes)
Other Gadgets Providers
Tomcat GTLAB Gadgets
Other Gadgets Providers
RSS Feed, Cloud, etc Services
Grid and Web Services (TeraGrid, OSG, etc)
Social Network Services (Orkut, LinkedIn,etc)
42
Click Add stuff and then Add feed or gadget.
Type the URL of your gadget.
43
GTLAB Applications as Google Gadgets MOAB
dashboard, remote directory browser, and proxy
management.
44
Making GTLAB Gadgets into Portlets

• Do all development in jsf_standalone.
• When you are ready to convert
• cd GTLAB/transition
• mvn clean process-resources
• cd GTLAB/portlets
• mvn clean install
• You will need to manually edit portlet.xml,
  group.xml, and layout.xml in GTLAB/portlet/src/mai
  n/webapp/WEB-INF/
• Thats it.

45
Common science gateway architecture. Aggregation
is in the portlet container. Users have limited
selections of components.
HTML/HTTP
Tomcat Portlets and Container
SOAP/HTTP
Grid and Web Services (TeraGrid, OSG, etc)
Grid and Web Services (TeraGrid, OSG, etc)
Grid and Web Services (TeraGrid, OSG, etc)
46
Various GTLAB applications deployed as portlets
Remote directory browsing, proxy management, and
LoadLeveler queues.
47
Whats next for GTLAB?

• OpenID and RSS client support
• Many other client libraries
• Google, Facebook, Amazon
• Support for more complicated graphs
• JavaScript/AJAX actions as well as HTML FORM
  actions.
• But really, were looking for interested
  developers and users.