D2K Web Service Crash Course

1
D2K Web Service Crash Course
2
Outline

• D2KWS in a Nutshell
• D2K Components Review
• Web Services Defined
• D2KWS Architecture Interface
• Creating a D2KWS Itinerary
• Creating a D2KWS Client
• D2KWS Client Example Phylomat
• D2KWS Admin Application
• Demo

3
D2K Web Service in a Nutshell

• Provides a platform and programming language
  neutral API for remotely executing D2K
  Itineraries
• Enables any type of client to run D2K Itineraries
• Java
• C
• .Net
• Python
• Etc.
• Manages multiple concurrent users and multiple
  concurrently processing jobs
• Dispatches jobs to high-end compute resources

4
D2K Components

• D2K Infrastructure
• Itinerary Execution engine

5
D2K Components

• D2K Infrastructure
• Itinerary Execution engine
• D2K-Driven Applications
• Applications that make use of the D2K
  Infrastructure
• Toolkit is a D2K-Driven app

6
D2K Components

• D2K Infrastructure
• Itinerary Execution engine
• D2K-Driven Applications
• Applications that make use of the D2K
  Infrastructure
• Toolkit is a D2K-Driven app
• D2K Server
• Special kind of D2K-Driven app
• Wraps the infrastructure to provide remote
  itinerary and module execution
• Used by the Toolkit to distribute module execution

7
D2K Components

• D2K Infrastructure
• Itinerary Execution engine
• D2K-Driven Applications
• Applications that make use of the D2K
  Infrastructure
• Toolkit is a D2K-Driven app
• D2K Server
• Special kind of D2K-Driven app
• Wraps the infrastructure to provide remote
  itinerary and module execution
• Used by the Toolkit to distribute module
  execution
• D2K Web Service
• Provides a generic programmatic interface for
  executing itineraries
• Communicates with D2K Servers over socket
  connections using D2K Specific protocols.

8
Web Services Defined

• A web service is a collection of procedures
  (functions, methods, etc.) available at a
  particular URL, accessible by remote client
  applications using standard web technologies.
• Web services are part of a middleware evolution,
  not a revolution.
• Previous middleware technologies
• Distributed Object Systems (ORB Object Request
  Brokers)
• CORBA (Common Object Request Broker Architecture)
  Object Management Group early 1990s
• DCOM/COM (Distributed Component Object Model)
  Microsoft adopted competitor to CORBA early
  1990s
• RMI-IIOP/EJB (Remote Method Invocation over
  Internet Inter-Orb Protocol) Java CORBA mid
  1990s
• Interoperability among disparate, distributed
  systems is the primary goal of middleware.
• Web services promise this and more.

9
Web Services Standards

• SOAP (Simple Object Access Protocol)
• Service communication
• XML vocabulary for exchanging data over networks
• SOAP elements envelope, header, body
• WSDL (Web Service Description Language)
• Service description
• XML vocabulary for describing the web service
  interface
• WSDL elements types, message, operation, port
  type, binding, service
• Used by code generators for dynamic binding
• UDDI (Universal Description, Discovery and
  Integration)
• Service discovery
• Specification for creating a web service registry
• Searchable for specific types of companies or web
  services
• Electronic yellow pages
• Accessed using SOAP

10
Java EE Web Service Communication
XML Data
11
D2K Web Service Architecture

• D2K Web Service and constituent parts are general
  purpose
• In no way are specific to an itinerary or client
  application
• Single endpoint can service many client
  applications
• D2K Web Service has many roles
• Job broker and monitor (primary role)
• Job processing is delegated to eligible D2K
  Servers using D2K specific protocols over TCP
  socket communications
• Proxy threads running in D2KWS process monitor
  job activity in remote processes and persist
  state changes
• Policy enforcer
• Operations, itineraries, jobs, and server sets
  (servers) are access controlled
• Central storage/archive
• Itinerary definitions, module classes, job
  execution parameters, job results, job resources
• Module class server
• D2K Servers lazily load classes from D2K Web
  Service at job runtime
• Requested classes (and class dependencies!) are
  archived when job finishes execution
• Archive available to clients for deserializing
  results

12
D2K Web Service Architecture

• RDMS maintains state for service
• Accounts, Roles
• Itinerary metadata
• Id, name, description, owner, version info, etc.
• Job metadata
• Id, name, status, owner, submit date, start date,
  etc.
• Server Set metadata
• Id, name, description, constituent servers,
  owner, etc.
• Access policies
• Itineraries, jobs, and server sets
• D2K Servers are job processors
• Head process accepts job requests
• If server is not busy, a worker process is
  started on new port and port number is returned
  to web service
• D2K Web Service connects to new process for job
  execution
• Job hand off is complete once itinerary
  definition has been pushed to worker process
• Resources needed for itinerary execution are
  loaded over the network from D2KWS by worker
  processes

13
D2K Web Service Interface (Java Binding)
14
D2K Web Service Interface (Java Binding)
15
D2K Web Service Interface (Java Binding)
16
D2K Web Service Interface (WSDL)
17
D2K Web Service Interface (WSDL)
18
Creating an Itinerary for the D2K Web Service

• Do not use subclasses of UIModule in your
  itinerary
• UIModules cannot be transported back to the
  client
• Implement your user interfaces within the client
  application instead
• If you must use subclasses of UIModule, set the
  Suppress User Interface Display property to
  true
• Decouple itinerary from local resources
• Allows web service implementation to take
  advantage of distributed server resources
• Avoid Runtime.exec()
• Creates server dependency
• Implement functionality as module if possible
• Avoid loading local data
• Creates server dependency
• Load data from a central location, accessible
  from anywhere
• To produce generic results (Data), modules
  should implement ncsa.d2k.core.modules.DataProduce
  r
• getData() returns a Transferable object
• Transferables should support String based
  DataFlavors
• text/plain, text/xml, text/html

19
Creating a Java Client for the D2K Web Service

• 4 ways to bind with the web service (JAX-RPC
  client APIs)
• Precompiled Stubs
• Generated statically at compile time
• JAX-RPC utility wscompile reads WSDL and
  generates endpoint interface and stub
• Stub implements interface
• Dynamic Proxies
• Proxy class generated dynamically at runtime
• JAX-RPC reads WSDL from URL and generates proxy
• Proxy used in similar way to stub
• Dynamic Invocation Interface
• Fixed API for invoking operations on Web service
  endpoints
• Can be used with or without WSDL
• D2K Service Proxy
• Wraps implementation used to communicate with D2K
  Web Service
• Additional methods provided for working with
  itineraries, modules, properties, etc.
• Included with D2K Web Service source
  distribution. Use Ant task client-package

20
D2KWS Client Example Phylomat

• Calculates the distribution of a particular
  motif among available (fully sequenced) proteomes
• Extracts motif sequences from full protein
  sequences
• Aligns extracted motif sequences and generates a
  neighbor-joining tree, which can be visualized
  with a supplied applet
• Evolutionary trends resulting from protein
  groups can be inferred from these data and
  hypotheses about the evolutionary origin of the
  motif/domain class can be formulated
• Job runs can be named, saved and rerun in the
  future

21
D2KWS Client Example Phylomat
22
D2KWS Client Example Phylomat

• User interacts with web browser to specify job
  execution parameters, then submits job to
  Phylomat web application.
• Phylomat web application contacts the D2K Web
  Service via the D2KServiceProxy class, and
  submits job
• Web service brokers job to D2K Server
• Text results are produced by job process
• Phylomat web application checks for job
  completion. Results are collected and displayed
  when job is finished

23
D2KWS Client Example Phylomat

• Data loaded from central loc.
• Runtime.exec() is used to cluster sequences. Has
  created server dependency
• Tree, Alignments, Sequences, Counts, Summary
  modules all implement
• ncsa.d2k.core.modules.DataProducer
• Data results get saved in web service tier
• Client retrieves results by calling
  getResultModuleData(int jobId, String moduleName)
  on the web service

24
D2KWS Client Example Phylomat
25
D2KWS Admin Application

• Used to configure and manage D2K Web Service
  endpoints
• Add, view, edit itineraries
• Add, view, edit server sets
• View, edit jobs
• Add, view, edit accounts
• Manage resource permissions
• Is itself a D2K Web Service client
• Uses the D2KServiceProxy
• Web application that can run in the same Tomcat
  server as the D2K Web Service, or a different one

26
D2K Web Service Demo
DEMO
27
Q and A
Questions?