Web Services Based Access to Scientific Applications

1
Web Services Based Access to Scientific
Applications

• Sriram Krishnan, Ph.D.
• sriram_at_sdsc.edu

2
Agenda

• Basic introduction to Web services 30-45 mins
• Introduction to the Opal Toolkit 30-45 mins
• Break 15 mins
• Using the Opal Toolkit to deploy a real
  scientific application as a Web service
  (hands-on) 90 mins

3
An Introduction to Web Services
4
Acknowledgements

• This presentation is based on a prior Web
  services tutorial presented at the Joint ACM Java
  Grande-ISCOPE 2002 Conference, Seattle, WA, Nov
  2002 by Madhusudhan Govindaraju, Aleksander
  Slominski, Sriram Krishnan, and Dennis Gannon

5
Web Services Definition

• Many different definitions are available
• IBM (Gottschalk, et al) A web service is an
  interface that describes a collection of
  operations that are network accessible through
  standardized XML messaging.
• Microsoft (on MSDN) A programmable application
  logic accessible using standard Internet
  protocols.
• Another Web Service definition
• A network service that provides a programmatic
  interface to remote clients

6
Web Services Requirements

• Accessible through standard Web protocols
• Interoperability is important
• Standard description language
• Publishable to a registry of services
• Discoverable via standard mechanisms

7
Web Services Features

• Independent of programming language and OS
• Not bound to any single platform or specific
  protocol
• All information required to contact a service is
  captured by the Web Service Description
• Web Services Description encapsulates an
  interface definition, data types being used, and
  the protocol information

8
Web Services Architecture
Service Registry
Lookup
Publish
Service Requestor
Service Provider
Interact
9
Web Services Stack
10
Need to know three things about WS

• What does the service do?
• How is the service accessed?
• Where is the service located?

11
WSDL

• Web Services Definition Language (WSDL)
• Standard to describe the invocation syntax of a
  Web Service
• Authors IBM, Microsoft and others
• WSDL is an XML document that describes
• Interface types for each port
• Content of messages it receives and sends
• Bindings of interfaces to protocols
• SOAP is default, others possible
• Describes the access points (host/port) for the
  protocol

12
WSDL Elements

• PortTypes
• Message
• Types
• Binding
• Port
• Service
• SoapStruct echoStruct(SoapStruct ss)

13
WSDL Types

• Types collection of all data types used in the
  Web service (any schema language can be used,
  typically XML Schemas)
• referenced by messages
• lttypesgt
• ltschema xmlns"http//www.w3.org/2001/XMLSchema"gt
  
• ltcomplexType name"SOAPStruct"gt
• ltallgt
• ltelement name"varString"
  type"string"/gt
• ltelement name"varInt" type"int"/gt
• lt/allgt
• lt/complexTypegt
• lt/schemagt
• lt/typesgt

14
WSDL Message

• Message abstract, typed definition of the data
  being sent
• ltmessage name"echoStructRequest"gt
• ltpart name"inputStruct type"sSOAPStruct"/gt
• lt/messagegt
• ltmessage name"echoStructResponse"gt
• ltpart name"return" type"sSOAPStruct"/gt
• lt/messagegt

15
WSDL PortType

• portType an abstract set of operations supported
  by one or more endpoints
• ltportType name"TestPortType"gt
• ltoperation name"echoStruct"gt
• ltinput message"tnsechoStructRequest"/gt
• ltoutput message"tnsechoStructResponse"/gt
• lt/operationgt
• lt/portTypegt

16
WSDL Binding

• Protocol Binding details of how elements in
  PortType are converted into concrete
  representations
• ltbinding name"TestSoapBinding"
  type"tnsTestPortType"gt
• ltsoapbinding style"rpc transport"http//sc
  hemas.xmlsoap.org/soap/http"/gt
• ltoperation name"echoStruct"gt
• ltsoapoperation soapAction"http//test.org/
  "/gt
• ltinputgt
• ltsoapbody use"encoded"
  namespace"http//test.org/"
• encodingStyle"http//schemas.xmlsoap.org/soap/
  encoding/"/gt
• lt/inputgt
• ltoutputgt
• ltsoapbody use"encoded"
  namespace"http//test.org/"
• encodingStyle"http//schemas.xmlsoap.org/soap/
  encoding/"/gt
• lt/outputgt
• lt/operationgt
• lt/bindinggt

17
WSDL Service and Port

• Service a named collection of ports
• Port how a binding is deployed to a particular
  endpoint
• ltservice name"TestService"gt
• ltport binding"tnsTestSoapBinding
• name"echo"gt
• ltsoapaddress
• location"http//test.org5049/serv
  /echo"/gt
• lt/portgt
• lt/servicegt

18
Three properties of a Web service

• PortType What does the service do?
• Binding How is the service accessed?
• Service Where is the service located?

19
WSDL Information Model

• Separation between
• Abstract specification
• Concrete implementation of specification
• In WSDL
• Abstract portType messages
• Concrete service port binding

20
SOAP

• SOAP HTTP XML
• XML
• De facto standard for representing data in a
  platform independent way
• HTTP
• Simple universally supported protocol
• Interoperability
• Easily understood Network Protocol HTTP
• Common Data Format XML

21
Characteristics of SOAP

• XML based
• Internet-based
• Independent of
• Programming language
• Transport mechanism
• Can be used with protocols other than HTTP
• SMTP Simple Mail Transfer Protocol
• JMS Java Message Service

22
Example of SOAP request

• POST /serv/echo HTTP/1.1
• Host www.test.org
• Content-Type text/xml
• Content-Length 357
• SoapAction http//test.org/
• ltSOAP-ENVEnvelope
• xmlnsSOAP-ENV http//schemasenvelope
• SOAP-ENVencodingStylehttp///encodinggt
  
• ltSOAP-ENVBodygt
• ltmechoStructRequest xmlnsmhttp//test.org/gt
  
• ltinputStructgt
• ltvarStringgtStay Classy, San Diego!lt/varStringgt
• ltvarIntgt2006lt/varIntgt
• lt/inputStructgt
• lt/mechoStructRequestgt
• lt/SOAP-ENVBodygt
• lt/SOAP-ENVEnvelopegt

23
Example of SOAP response

• HTTP/1.1 200 OK
• Content-Type text/xml
• Content-Length 343
• ltSOAP-ENVEnvelope
• xmlnsSOAP-ENVhttp///envelope/
• SOAP-ENVencodinghttp///encoding/gt
• ltSOAP-ENVBodygt
• ltmechoStructResponse xmlnshttp//../gt
• ltreturngt
• ltvarStringgtI am Ron Burgundy?lt/varStringgt
• ltvarIntgt2006lt/varIntgt
• lt/returngt
• lt/mechoStructResponsegt
• lt/SOAP-ENVBodygt
• lt/SOAP-EnvEnvelopegt

24
Web Services Stack
25
Web Service Toolkits

• Apache Axis http//ws.apache.org/axis/
• One of the most popular toolkits
• User by several projects here like NBCR, GEON,
  CAMERA, GLEON
• Microsoft .NET http//msdn.microsoft.com/webservi
  ces/
• SUN http//java.sun.com/webservices/
• Several other lightweight implementations XSUL,
  ZSI (Python), SOAPLite (Perl)

26
Writing a Web Service Apache Axis

• Define a Web service interface
• Use WSDL to define port types, messages, and data
  types
• Use a stub compiler to generate Java sources
• WSDL2Java generates client and server side
  bindings from the WSDL
• Also generates Web Services Deployment
  Descriptors (WSDD)

27
Writing a Web Service Apache Axis (contd.)

• Implement the service implementation
  (server-side)
• Deploy the service into a Container
• Jakarta Tomcat http//tomcat.apache.org/
• Hosting environment - provides desirable features
  such as reliability, scalability, security, etc
• Write custom clients based on the generated stubs
• Can also write clients in other languages, viz.
  Python, Javascript, Perl, etc

28
Summary

• Hopefully, we have a better idea of what Web
  services are
• What? How? Where?
• Next, we will see how Web services are applicable
  to the scientific community

29
Wrapping Scientific Applications As Web Services
Using The Opal Toolkit
30
Acknowledgements

• Karan Bhatia
• Brent Stearn
• Kim Baldridge
• Wilfred Li
• Peter Arzberger

• PMV Team
• Kepler Team
• Chris Misleh
• Kurt Mueller
• Jerry Greenberg
• Steve Mock

31
Motivation

• Enable access to scientific applications on Grid
  resources
• Seamlessly via a number of user interfaces
• Easily from the perspective of a scientific user
• Enable the creation of scientific workflows
• Possibly with the use of commodity workflow
  toolkits

32
Some Problems

• Access to Grid resources is still very
  complicated
• User account creation
• Management of credentials
• Installation and deployment of scientific
  software
• Interaction with Grid schedulers
• Data management

33
Towards Services Oriented Architectures (SOA)

• Scientific applications wrapped as Web services
• Provision of a SOAP API for programmatic access
• Clients interact with application Web services,
  instead of Grid resources
• Used in practice in NBCR, CAMERA, GLEON, among
  others

34
Big Picture
Gemstone
PMV/Vision
Kepler
State Mgmt
Application Services
Security Services (GAMA)
Globus
Globus
Globus
Condor pool
SGE Cluster
PBS Cluster
35
Scientific SOA Benefits

• Applications are installed once, and used by all
  authorized users
• No need to create accounts for all Grid users
• Use of standards-based Grid security mechanisms
• Users are shielded from the complexities of Grid
  schedulers
• Data management for multiple concurrent job runs
  performed automatically by the Web service
• State management and persistence for long running
  jobs
• Accessibility via a multitude of clients

36
Possible Approaches

• Write application services by hand
• Pros More flexible implementations, stronger
  data typing via custom XML schemas
• Cons Not generic, need to write one wrapper per
  application
• Use a Web services wrapper toolkit, such as Opal
• Pros Generic, rapid deployment of new services
• Cons Less flexible implementation, weak data
  typing due to use of generic XML schemas

37
The Opal Toolkit Overview

• Enables rapid deployment of scientific
  applications as Web services (lt 2 hours)
• Steps
• Application writers create configuration file(s)
  for a scientific application
• Deploy the application as a Web service using
  Opals simple deployment mechanism (via Apache
  Ant)
• Users can now access this application as a Web
  service via a unique URL

38
Opal Architecture
39
Implementation Details

• Service implemented as a single Java class using
  Apache Axis
• Application behavior specified by a configuration
  file
• Configuration passed as a parameter inside the
  deployment descriptor (WSDD)
• Possible to have multiple instances of the same
  class for different applications
• Distinguished by a unique URL for every
  application
• No need to generate sources or WSDL prior to
  deployment

40
Sample Container Properties
the base URL for the tomcat installation
this is required since Java can't figure out the
IP address if there are multiple network
interfaces tomcat.urlhttp//ws.nbcr.net8080
database information database.usefalse database.u
rljdbcpostgresql//localhost/app_db database.use
rltapp_usergt database.passwdltapp_passwdgt
globus information globus.usetrue globus.gatekeep
erws.nbcr.net2119/jobmanager-sge globus.service_
cert/home/apbs_user/certs/apbs_service.cert.pem g
lobus.service_privkey/home/apbs_user/certs/apbs_s
ervice.privkey parallel parameters num.procs16
mpi.run/opt/mpich/gnu/bin/mpirun
41
Sample Application Configuration
ltappConfig xmlns"http//nbcr.sdsc.edu/opal/types"
xmlnsxsd"http//www.w3.org/2001/XMLS
chema"gt ltmetadatagt ltusagegtlt!CDATApsize.py
opts ltfilenamegtgtlt/usagegt ltinfo
xsdtype"xsdstring"gt lt!CDATA
--help Display this text
--CFACltvaluegt Factor by which to expand mol
dims to get coarse
grid dims default
1.7 ... gt lt/infogt
lt/metadatagt ltbinaryLocationgt/homes/apbs_user/bin
/psize.pylt/binaryLocationgt ltdefaultArgsgt--GMEMCE
IL1000lt/defaultArgsgt ltparallelgtfalselt/parallelgt
lt/appConfiggt
42
Application Deployment Undeployment

• To deploy onto a local Tomcat container

ant -f build-opal.xml deploy -DserviceNameltservic
eNamegt -DappConfigltappConfig.xmlgt

• To undeploy a service

ant -f build-opal.xml undeploy -DserviceNameltserv
iceNamegt
43
Service Operations

• Get application metadata Returns metadata
  specified inside the application configuration
• Launch job Accepts list of arguments and input
  files (Base64 encoded), launches the job, and
  returns a jobID
• Query job status Returns status of running job
  using the jobID
• Get job outputs Returns the locations of job
  outputs using the jobID
• Get output as Base64 Returns an output file in
  Base64 encoded form
• Destroy job Uses the jobID to destroy a running
  job

44
MEMEMAST Workflow using Kepler
45
Kepler Opal Web Services Actor
46
Gemstone Access to Molecular Science
47
Future Work

• Opal 2.0
• Currently under way - in design phase
• Use of Axis2 for better performance
• 6-8x performance improvement over Axis1.2
• Pluggable Resource Provider Model
• Easier to integrate access to resources via GRAM,
  DRMAA, CSF4, etc
• State persistence via Hibernate
• Alternate mechanisms for I/O staging - GridFTP,
  RFT, etc

48
Summary

• Opal enables rapidly exposing legacy applications
  as Web services
• Provides features like Job management,
  Scheduling, Security, and Persistence
• More information, downloads, documentation
• http//nbcr.net/services/

49
Using Opal to deploy a real scientific
application as a Web service
50
Goals

• Start off with a clean slate
• Install software prerequisites
• Install the Opal toolkit
• Deploy scientific application as Web service
• Access service via command-line client

51
Before we get started

• Should have JAVA and ANT already installed
• And environment variables JAVA_HOME and ANT_HOME
  set correctly
• Make sure that these can be accessed from the
  command prompt
• Type java
• Type ant
• Watch for error messages
• Ask for help if these dont work

52
Download Software Prerequisites

• Might want to create a new folder called
  Tutorial on your Desktop
• Download and Extract Axis
• http//nbcr.net/services/downloads/tutorial/axis-1
  _2_1.zip
• Extract inside Tutorial directory
• Download and Extract Tomcat
• http//nbcr.net/services/downloads/tutorial/jakart
  a-tomcat-5_0_30.zip
• Extract inside Tutorial directory
• Set the environment variable CATALINA_HOME
• Control Panel -gt System -gt Advanced

53
Download Opal

• Download and Extract Opal
• http//nbcr.net/services/downloads/tutorial/opal-w
  s-1.0RC1.zip
• Extract inside Tutorial directory
• Compile the sources
• On the command prompt, cd to the above location
• Type ant -f build-opal.xml compile
• Watch for error messages
• Ask for help if you see any

54
Deploy Axis inside Tomcat

• Copy axis directory inside axis-1_2_1\webapps
  into CATALINA_HOME\webapps
• Copy activation-1_0_2.jar and
  mailapi-1_3_1.jar from opal-ws-1.0RC1\lib
  into CATALINA_HOME\common\lib
• Start Tomcat from the command prompt
• Type cd CATALINA_HOME\bin
• Type startup.bat
• Test the Axis deployment
• http//localhost8080/axis/happyaxis.jsp
• Watch for error messages, warnings are OK

55
Edit Opal properties

• Using Wordpad, open opal-ws-1.0RC1\etc\opal.prope
  rties
• Look over the various properties being set
• Modify the property tomcat.url by replacing
  localhost with the the IP address of your
  machine
• Type ipconfig /all to get your IP address

56
Opal Properties
parallel parameters num.procs1
mpi.run/Users/sriramkrishnan/Misc/mpich-1.2.7/bin
/mpirun the base URL for the tomcat
installation this is required since Java
can't figure out the IP address if there
are multiple network interfaces
tomcat.urlhttp//localhost8080 database
information database.usefalse
database.urljdbcpostgresql//localhost/app_db
database.userapp_user database.passwdapp_p
asswd globus information
globus.usefalse globus.gatekeeperlocalhost2
119/jobmanager-sge globus.service_cert/Users/
sriramkrishnan/certs/app_service.cert.pem
globus.service_privkey/Users/sriramkrishnan/certs
/app_service.privkey
57
Download Scientific Application

• We will use the application OpenBabel
• Designed to support molecular modeling,
  chemistry, and many related areas, including
  interconversion of file formats and data
• http//openbabel.sourceforge.net/
• Download http//nbcr.net/services/downloads/tutor
  ial/openbabel-2.0.0awins.zip
• Extract inside Tutorial\Babel directory
• Type babel from the above directory to ensure
  that it installed fine

58
Edit Babel Config

• Using Wordpad, open opal-ws-1.0RC1\etc\babel_conf
  ig.xml
• Look over the various elements
• Change the value of the binaryLocation to point
  to your Babel executable

59
Babel Config
ltappConfig xmlns"http//nbcr.sdsc.edu/opal/types"
xmlnsxsd"http//www.w3.org/2001/XMLS
chema"gt ltmetadatagt ltusagegtlt!CDATA./babel
-iltinput-typegt ltnamegt -oltoutput-typegt
ltnamegtgtlt/usagegt ltinfo xsdtype"xsdstring"gt
lt!CDATA ... Currently supported
input types alc -- Alchemy file prep -- Amber
PREP file ... Currently supported output
types ... caccrt -- Cacao Cartesian
file cacint -- Cacao Internal file ...
gt lt/infogt lt/metadatagt
ltbinaryLocationgt/Users/sriramkrishnan/bin/babellt/b
inaryLocationgt ltdefaultArgsgtlt/defaultArgsgt
ltparallelgtfalselt/parallelgt lt/appConfiggt
60
Deploy application using Ant

• On the command prompt, cd to your Opal
  installation
• To deploy application, type the following
• ant -f build-opal.xml deploy -DserviceNameBabelSe
  rvicePort -DappConfigetc\babel_co
  nfig.xml
• Restart Tomcat
• Type cd CATALINA_HOME\bin
• Type shutdown.bat
• Type startup.bat
• Check list of services
• http//localhost8080/axis/services

61
Run command line client

• On the command prompt, cd to your Opal
  installation
• Set classpath by typing classpath.bat
• To run the client, type the following
• java edu.sdsc.nbcr.opal.GenericServiceClient -l
  http//localhost8080/axis/services/BabelServicePo
  rt -a "-ipdb sample.pdb -h -opdb output.pdb" -f
  etc\sample.pdb
• Replace localhost with your neighbors IP address
  to use their Babel service

62
Whats Next?

• To add another service, you just need to add
  another application config
• And deploy using Ant - thats all!
• You can write custom clients for your Web
  services
• We will see how to use Gemstone tomorrow

63
More Information

• http//nbcr.net/services
• Downloads, Papers, Presentations
• More documentation
• Advanced configuration like Database, Globus, and
  Security setup
• Feel free to drop us a note - contacts available
  from our web page