Kepler: a scientific workflow management system

1
Kepler a scientific workflow management system

• Zhiming Zhao
• zhiming_at_science.uva.nl
• VL-e SP2.5, University of Amsterdam

2
Outline

• About Kepler
• Kepler in scientific experiments from a
  scientists point of view
• Kepler in VL-e
• Discussion

3
About Kepler

• Kepler is a Scientific Workflow Management System
  
• An open source cross-project collaboration. It is
  led by San Diego Supercomputer Center (SDSC),
  with participation from the SEEK ITR, DOE SciDAC
  SDM, and others.
• Based on the Ptolemy II system.
• State
• Still active
• Current downloadable version (from
  www.kepler-project.org) is alpha 9
• A beta release has been announced, but is not
  available online yet.

4
Basic concepts workflow modeling

• Actors basic workflow components
• It can be atomic
• A set of ports (for input and output)
• A set of parameters (for configuration)
• A fire() function the main functionality
• Or composite
• Contain a set of actors (atomic or composite)
• And one director for interpreting the runtime
  behavior
• Directors
• For interpreting runtime behavior between actors
• Relation links
• Describing the port relations between actors
• A workflow, at an abstract level, is a composite
  actor

5
What does Kepler offer?

• A visual interface
• For composing a workflow
• For annotating components and flows
• A set of components
• Actors
• Directors
• Relation links
• Ports
• Execution control facilitates
• Run/Pause/Stop
• Parameter tuning
• Set breakpoint

6
How to compose a workflow?

1. Select proper components
2. Develop necessary new components
3. Describe flow logic
4. Save/Open a flow description

7
How to compose a workflow (Cont.)

• Select proper actors and directors
• Browse the tree structure of a component library
• Ontology based search (not mature)
• Check the documentation and source code of a
  component through GUI
• Check the functionality of an actor Port,
  parameter, pre-fire(), fire(), and post-fire().
• Check the functionality of a director.
• Develop new components, if needed
• Describe flow logic
• Save/Open a flow description

8
How to compose a workflow (Cont.)

• Select proper components
• Develop necessary new actors
• Wrap a legacy code (as a command line)
• Using CommandExecLine Actor
• Specify location of the command, output file and
  location, and Environment variables
• Wrap R scripts
• Using RExpression Actor
• Provide the script, working directory and Pixels
  for output
• Needs a R engine

9
How to compose a workflow (Cont.)

• Select proper components
• Develop necessary new components
• Import existing web services
• Using Web service Actor
• Provide WSDL URL, method name, authentication,
  and others
• Only one method can be imported per actor
• Interfacing Grid resources
• Authentication
• Globus Job
• Globus Proxy
• Grid FTP
• Parameterized Globus Job
• Run Job Grid Client
• Update Grid FTP

10
How to compose a workflow (Cont.)

• Select proper components
• Develop necessary new components
• Write java code and add a new actor
• write the java code, according to the Kepler
  actor convention such as definition of ports,
  parameters and (pre/post) fire functions
• Write make a actor.xml file describe the
  functionality and class location, component ID,
  ontology, and a MANIFEST.MF file
• Put the code in kepler/src, and compile it
• After compiling the code, you have to ant
  buildkarlib
• And generate new kepler.jar file, you can view
  the source of the new actor
• Remove .kepler and .ptolemy from the users home
  directory, before re-starting kepler

11
How to compose a workflow (Cont.)

• Select proper components
• Develop necessary new components
• Describe flow logics
• Between actors port/port connection.
• A port is for input or output
• In a workflow
• A single port can only be connected by one
  relation link
• A multi-ports can be connected by more than one
  links
• Single ports can linked as more to more, such
  as fan-in or fan-out by using a relation
  component. Fan-out is only supported by few
  directors
• Choose execution mode
• Select an director
• Continuous Time, Discrete Event, Process Net and
  Static Data Flow
• There are more directors in Ptolemy
• Customize it
• Set parameters

12
How to compose a workflow (Cont.)

• Select proper components
• Develop new components, if needed
• Describe flow logics
• Save/Open a flow description
• Modeling Markup Language (MoML)
• in XML dialect

13
An example

• Add a helloworld actor
• Make three web service actors
• Import a web service echoString
• Connect them to output

14
Kepler at Runtime

1. Human in the loop
2. Scheduling mechanisms

15
Kepler at Runtime, Cont.

• Human in the loop
• Run a workflow in the same environment where the
  user compose the workflow.
• Set break point in an actor
• Modify parameters of Actors and directors
• Scheduling mechanisms

16
Kepler at Runtime, Cont.

• Human in the loop
• Scheduling mechanisms
• Parameter sweeping not directly, but can do it
  via a Nimrod actor
• Job farming not directly support

17
Kepler in VL-e

• At application SPs
• In IBU case
• A starting point for the new design of the VL-e
  framework
• Virgil as the user interface
• Developing a RTS (in VLAM) director

18
Discussion

• Outstanding functionality Workflow modeling
• Concept of Director (from Ptolemy) explicitly
  modeling the execution semantics
• Different levels of encapsulation
• a complex actora director actors (or complex
  actors),
• a workflow a complex actor
• Other functionality is provided in the similar
  way as the other systems, e.g., Taverna, Triana
  and VLAM
• GUI drag and draw
• Runtime parameter configuring
• Grid access, Web service support, Parameter
  sweeping
• Weak points
• Triana provides better way to outsource web
  services
• It is still in Alpha release

19
Cont.

• Interoperability with a different workflow
  system
• At the engine level
• Wrap the workflow engine as a kepler actor.
• Do not change the components in that system
• At the actor level
• Or wrap components of the workflow system as
  actors,
• Optional (wrap the engine as a kepler director)
• and use the Kepler engine to control them.

20
References

• Kepler website http//kepler-project.org/