Open Grid Computing Environments/Gateways Tutorial

1
Open Grid Computing Environments/Gateways Tutorial

• Marlon Pierce, Suresh Marru, Gopi Kandaswamy,
  Gregor von Laszewski, and Tom Scavo

2
JM8TBevr myproxy.ncsa.uiuc.edu
3
Tutorial Overview

• Part 1 OGCE Portal (Marlon), 830-930 am
• Introduction and overview
• Portal Demo
• How to build Grid portlets and gadgets
• Part 2 OGCE Workflow Tools (Suresh and Gopi)
• Part A Demo and overview, 930-1000 am
• Part B Hands on demo, deploying new
  applications, 1030-1115 am
• Part 3 Web 2.0 for Grid Portals (Gregor)
• Part 4 GridShib and Gateway security (Tom)

4
Last Part of Tutorial

• Both Gregor and Tom had flight cancellations, so
  this opens up 45 minutes of the program.
• We can do any or all of the following
• More hands on with portal and workflow stuff.
• Building stuff from scratch
• Wrapping applications with GFAC
• Discuss Gateway issues, requirements, etc
• I also have Web 2.0 tutorial material in hand if
  there is any interest.

5
Goals of Tutorial

• Goals
• Review portal and gateway technologies old and
  new.
• Demonstrate how to execute workflows and wrap
  TeraGrid applications.
• Discuss security
• Hear from TG Gateway community what do you want?
• We have a lot of material to cover so we will
  sample many things.
• You will have enough information to get going as
  a homework assignment.
• Please also check with us at TG08
• Please see Nancys Gateway Parade this
  afternoon.

6
Slides and Demo Site

• Tutorial slides are available from
  http//www.collab-ogce.org/ogce/index.php/Tutorial
  s
• We run a permanent demo portal at
  https//community.ucs.indiana.edu8443/gridsphere/
  
• Also aliased as https//ogceportal.iu.teragrid.org
  8443/gridsphere
• Accounts train30-train49 have been created for
  the workshop. Password is the same as the
  account name.
• We also have training accounts with same names
  that can be used to retrieve TG proxy
  credentials.
• These should be active all week.

7
Portal and Gadget Demos

• Demonstration of basic capabilities

8
Getting the Code
9
What You Can Download

• OGCE Portal
• Various portlet samplers
• Web and Grid Services
• XRegistry Web Service Registry
• GFAC Application Factory Service
• Grid Portal Information Repository
• Programming Libraries
• Java and JavaScript COG
• Grid Tag Libraries and Beans (GTLAB)
• Information Service Client APIs

10
Project Release Schedule

• We have two major releases per year with
  intermediate updates.
• TeraGrid (June)
• Supercomputing (November)
• Focus of this release cycle the workflow suite
  tools shown in Part 2.
• We do not currently package services and portlets
  together.
• These are separate downloads.

11
Project Build and Test

• Source Code Version Control and Building We use
  Maven 2 and SourceForges SVN
• Anonymous checkouts
• We are Open Source
• Nightly Build Testing We use the NMI Build and
  Test facility at UW
• Mac and Linux flavors
• System Testing We use Apache JMeter to verify
  portal builds function correctly.
• You can download this from us
• You can also view the nightly dashboard

12
Project Information

• Web site http//www.collab-ogce.org
• Announcement Blog http//collab-ogce.blogspot.com
  
• RSS/Atom Feeds http//collab-ogce.blogspot.com/at
  om.xml
• This is also embedded in the wiki as News
• Contact us discuss_at_ogce.org

13
Building the Portal
14
Choosing a Host Computer

• Select your host computer.
• You will need a real IP address for some
  COG-based portlets
• VPN also OK (?)
• Know general Globus et al firewall issues.
• I recommend turning firewalls and SELinux off for
  development.
• Operating Systems
• We test with Mac OS X and many Linux flavors
• Other Unix should be fine (assuming Java works)
• We have not had any requests for Windows.
• Software Environment
• The portal download comes with everything you
  need EXCEPT Java. You need the JDK, not the JRE.
• We test with Java 1.5. Dont use Java 1.4.
• The more memory, the better.

15
Downloading and Building the Portal

• Download the portal from http//www.collab-ogce.or
  g.
• Unpack in HOME.
• You should also download the Maven repository.
• This is optional but recommended
• Edit ogce-portal-home/pom.xml
• Run the command mvn clean install.
• Use mvn o clean install to build offline.
• Build will typically take 5 minutes (new Mac) to
  30 minutes (old Linux PC).

You can also check out the latest code from SVN.
Check the Nightly Build dashboard to verify.
16
Practical Considerations

• Increase your JVM Memory
• export JAVA_OPTS"-server -Xms512m -Xmx1024m
  -XXMaxPermSize256m
• Xmx2048m, 4096m, etc if you can.
• Monitor catalina.out for problems
• tail f catalina.out
• Make sure that Tomcat shuts down
• Kill if necessary
• NFS can really slow down building
• Use /usr/local/ if you have problems

17
Editing pom.xml
ltpropertiesgt ltportal.server.ipgt
156.56.104.143 lt/portal.server.ipgt
lthost.base.urlgt http//portal.server.ip80
80/ lt/host.base.urlgt ltgridftp.host.namesgt
. lt/gridftp.host.namesgt
ltgram.host.namesgt . lt/gram.host.namesgt
ltproject.homegt env.HOME/ogce-portal-only
lt/project.homegt lt/propertiesgt

• You only need to edit the top level
  ogce-portal-only/pom.xml.
• You only need to change ltpropertiesgt for local
  settings.
• You MUST update the IP address.
• You may want to add or remove GRAM and GridFTP
  hosts.
• Update project.home if you are building someplace
  besides HOME.

18
Rebuilding the Portal

• Running mvn clean install will completely wipe
  out the existing installation.
• You may not want to do this.
• If you want to rebuild individual components,
  just run the following command from the
  ogce-portal-only directory.
• mvn clean install f portlets/myportlet/pom.xml
• You can do this for all ltmodulesgt in
  ogce-portal-deploy/pom.xml

19
OGCE Portal Project Layout

• We follow standard Maven 2 conventions for
  project layouts for each component.
• moduleName/src/main/webapp
• moduleName/src/main/java
• Modules are listed on next slide
• See ogce-portal-only/pom.xml.
• Each module is a separate directory and has its
  own pom.xml.

20
(No Transcript)
21
Developing Grid User Interface Components

• Portlets, gadgets, etc.

22
Under the Hood Building Components

• OGCE portlets use the Java COG Abstraction Layer
  to access Grid services.
• Provides a buffer over different Grid toolkits
  and versions.
• Provides ability to compose graph-based
  workflows.
• Tutorial in itself
• Portlets can be built with our Apache Velocity
  bridge.
• Template or steal this code approach.
• We have also developed Grid Tag Libraries and
  Beans (GTLAB) to simplify this process.

23
GTLAB Features

• Extends Java Server Faces.
• Tag components wrap major COG Abstraction Layer
  features
• Allows you to do standalone development.
• Use JSF portlet bridge to convert into portlets
• No new coding, just change, add config files and
  jars.
• Or forget about portlets and
• Convert into Google Gadgets
• Develop as a Facebook application
• Etc.
• Status GTLAB is under active development, can be
  used to build simple things, but still has many
  loose ends.
• You need to be technically adept to work with it
  now.
• But you are welcome to give feedback or
  contribute.

24
Installing GTLAB

• You need to install the OGCE portal
• GTLAB depends on many shared jars.
• Download the release code from www.collab-ogce.org
  or check out from SVN
• svn checkout https//ogce.svn.sourceforge.net/svnr
  oot/ogce/GTLAB
• Edit GTLAB/jsf_standalone/pom.xml
• ltcatalina.homegt should point to your OGCE portal
  server.
• Edit GTLAB/jsf_standalone/src/main/resources/resou
  rces.properties.
• Run mvn clean install in GTLAB/jsf_standalone/

25
Levels of Abstraction in GTLAB code

• Note JSF is compatible with JSP, so you can mix
  and match.
• And you can embed JavaScript
• YUI, Scriptaculous user interface goodies.
• Google and Facebook APIs.

26
MyProxy Example
lthform id"form"gt lthoutputText
value"Username"/gt lthinputText
value"resource.username" required"true"/gt
lthoutputText value"Password "/gt
lthinputSecret value"resource.password"
required"true"/gt lthoutputText
value"MyProxy Server "/gt lthinputText
value"resource.myproxyHostname"
required"true"/gt ltosubmit id"submit"
value"Submit" binding"builder.body"
action"builder.junkAction" gt
ltomultitask id"multi" persistent"true"
taskname"resource.taskname"gt
ltomyproxy id"mypr" hostname"resource.myproxyH
ostname" lifetime"2" password"resource.pa
ssword" port"7512" username"resource.user
name"/gt lt/omultitaskgt lt/osubmitgt
lt/hformgt
27
Show the Proxy Credential
lthdataTable value"myproxy.storedProxyList"
border"1" binding"myproxy.stored
ProxyTable" var"proxyEntry"
id"proxyTable"gt lthcolumngt
ltffacet name"header"gt lthoutputFormat
id"subject" escape"false value"Subject"/gt
lt/ffacetgt lthoutputText
id"proxySubject" value"proxyEntry.subject"/gt
lt/hcolumngt lthcolumngt
ltffacet name"header"gt lthoutputFormat
id"Remove" escape"false value"Delete"/gt
lt/ffacetgt lthcommandButton
id"deleteProxy" value"Remove
actionListener"myproxy.deleteCredential"/
gt lt/hcolumngt lt/hdataTablegt
28
Executing Multiple Task
ltosubmit id"submit" value"Submit"
binding"builder.body" action"builder.j
unkAction" gt ltomultitask id"multi"
persistent"true" taskname"resource.tasknam
e"gt ltojobsubmit id"js" myfaces"true"
arguments"resource.arguments"
executable"resource.executable"
hostname"resource.hostname"
provider"resource.provider"
stdout"resource.stdout"/gt
ltofiletransfer id"ft" myfaces"true"
from"resource.from" to"resource.to"
/gt ltodependency id"dep" task"ft"
dependsOn"js"/gt lt/omultitaskgt lt/osubmitgt
29
These will work as gadgets, portlets, or
standalone applications
30
Making GTLAB Gadgets into Portlets

• Do all development in jsf_standalone.
• When you are ready to convert
• cd GTLAB/transition
• mvn clean process-resources
• cd GTLAB/portlets
• mvn clean install
• You will need to manually edit portlet.xml,
  group.xml, and layout.xml in GTLAB/portlet/src/mai
  n/webapp/WEB-INF/
• Thats it.

31
Making GTLAB Google Gadgets

• A Gadget is a standalone application that is
  integrated into your personal iGoogle display.
• They can run on your Web server.
• You can use SSL/HTTPS
• Google Gadgets have two versions
• http these use Googles JavaScript APIs
• url these wrap external applications with
  IFrames.
• Limitations
• Not much real estate (but location, location,
  location)
• Cookies manage state on gadgets from the same
  server but we need a way to integrate multiple
  independent servers.

32
Some GTLAB Gadgets
33
Example Gadget Config File
lt?xml version"1.0" encoding"UTF-8" ?gt
ltModulegt ltModulePrefs title"MyProxy Gadget
Example" scrolling"true"
height"500"/gt ltContent type"url"
href"http///8080/GTLAB/examples/MyProxyExamp
le.jsf"gt lt/Contentgt lt/Modulegt
Save this as MyProxy.xml and place in a Web
accessible place.
34
Some Web 2.0 Resources

• See recent tutorials by Geoffrey Fox and me
• http//grids.ucs.indiana.edu/ptliupages/presentati
  ons/Web2.0Tutorial.ppt
• http//grids.ucs.indiana.edu/ptliupages/presentati
  ons/CTSTechFutures_May19-08.pptx
• Includes material on Google Gdata, Facebook API,
  Open Social, Microformats, etc.
• See also full examples at http//communitygrids.bl
  ogspot.com

35
TeraGrid Information Services Portlets and
Gadget Libraries

• Slides courtesy of Maytal Dahan

36
TG Information Services

• What are TG Information Services?
• Services that publish metadata about TeraGridd
• TG Software, TG resources, TG Services
• Software and applications can query TG Info
  services for latest information
• Existing TG Info Services
• Currently we are relying on the REST interface of
  TG Info Services
• TG Also has WebMDS interfaces
• Service includes CTSS4 information

37
Libraries for TG Info Services

• Using the REST Interfaces we are
• Developing JavaScript Libraries that
  automatically query and return service
  information
• Handling all the querying and xml parsing
  necessary and return objects through libraries
• API will include the ability to (and more)
• getTGSites() Get a list of all TG sites
• getResoures() Get list of resources at a site
• getServices() Get list of TG resources
• Late summer availability
• Demo portlets available now from SVN
• svn co https//ogce.svn.sourceforge.net/svnroot/og
  ce/ogce-portal-only/portlets/tg-infoservices-portl
  et/
• Not yet integrated with full OGCE build, so you
  will need to use Maven 1.
• Check with Maytal if you are interested.

38
Talks, Demos, and BOFs
39
Sample Portlet Interfaces
40
Future Work SC08

• GTLAB enhancements generalized tasks, more
  gadgets, Facebook and Open Social examples,
  GridShib support.
• Workflow Tools portal-based workflow client,
  WS-Messaging, GRAM Auditing, GridShib integration
• JavaScript Libraries TG Info Services, support
  for Grid Services.

41
Finally

• GCE08 workshop will be held at SC08 in Austin
• More details to come.
• E-Science 2008 conference is in Indianapolis in
  December.
• Papers due July 20