A Workflow Bus for eScience Applications

1
A Workflow Bus for e-Science Applications

• Dr Zhiming Zhao
• Faculty of Science, University of Amsterdam
• VL-e SP 2.5

2
Outline

• Introduction
• A workflow bus and generic e-Science framework
• Prototype and experiment results
• Discussion
• Conclusions
• Future work

3
Scientific workflow in e-Science

4
Scientific Workflows in e-Science
Experiment processes

• A SWMS is able to
• Automate experiment routines
• Rapid prototype experimental computing systems
• Hide integration details between resources
• Manage experiment lifecycle

workflows for administration, e.g., AAA, and
other issues.
Abstract workflows
Executable (concrete workflows)
5
Insight a Scientific Workflow Management System

• In our view, a SWMS at least implements
• A model for describing workflows
• An engine for executing/managing workflows
• Different levels of support for a user to
  compose, execute and control a workflow.

Workflow (based on certain model)
Composition
A SWMS
User support
Engine level control
Engine
Resource level control
resources
6
Diversity in SWMS

• Taverna
• Web services based language Scufl
• FreeFluo engine
• Graphical viz of workflow

• Triana
• Components
• Task graph
• Data/control flow

• Kepler
• Actor,director
• MoML
• Execution models

• Pegasus
• Based on DAGMan
• VDL
• DAG

• DAGMan
• Computing tasks
• DAG

7
Research context

• Different levels of abstraction
• Workflow services
• Short term
• Long term a generic and effective workflow
  management service

8
Mission

• Effectively reuse existing workflow managements
  systems, and provide a generic e-Science
  framework for different application domains.
• A generic framework can
• Improve the reuse of workflow components and the
  workflows for different experiments
• Reduce the learning cost for different systems
• Allow application users to work on a consistent
  environment when underlying infrastructure
  changed

9
Possible options

• Abstract approach
• Extend approach
• Aggregate approach

SWMS1
SWMS2
SWMS3
SWMSG
SWMS1
SWMS2
SWMS3
SWMSG
SWMS1
SWMS2
SWMS3
SWMS1
SWMS2
SWMS3
SWMS G
10
Why we choose an aggregation approach?

• Abstract approach
• Build a perfect system
• Difficult to find a set of systems cover all the
  required generic functionality it requires
  re-implementation of existing things
• Extend approach
• Incrementally development
• The solution depends on a specific system
• Aggregate approach
• Maximize the reuse of the existing workflow
  systems
• Has to handle interoperability issues provide
  customized interface existing workflow system

11
A workflow bus paradigm
Workflow
Sub workflow 1
Sub workflow 2
Sub workflow 3
Triana
Taverna
Kepler
Workflow bus
A workflow bus is a special workflow system for
executing meta workflows, in which sub workflows
will be executed by different engines.
12
Architecture

• Terminology
• The execution of a workflow is one study, and the
  execution of a sub-workflow is called a
  sub-study, or a scenario
• Basic idea
• Study manager schedules sub workflows
• Scenario managers interface third party workflow
  engines and reacts to the Study manager
• A user interface for composition and execution
  control.

13
Requirements

• A distributed framework for study and scenario
  managers
• Data input/output of a sub-workflow, description
  of the workflow can be described and recognized
  by study and scenario managers
• Handle the user interactions which are needed in
  scenarios
• The engine can be decoupled from a SWMS
• Be fault tolerant

14
Considerations

• From integration point view study and scenario
  managers can be coupled by
• Web services
• Object oriented middleware (CORBA, HLA, etc.)
• Agent based middleware
• Or an existing workflow system (Kepler, Taverna,
  Triana or others)
• The description of meta workflow
• The execution model of the meta workflow

15
A JADE/Ptolemy based prototype
Scenario Mnger
Scenario Mnger
Scenario Mnger
Study Mnger
Ptolemy
Actor
Actor
Director
Actor
User interface
16
How it works

• In user front end a user defines meta workflow,
  each actor represents a sub workflow
• At runtime, each actor initiates a scenario
  agent, and passes the workflow description to the
  scenario manager
• A scenario manager controls an engine and execute
  the sub-workflow

17
Prototype
18
Experiment results

• Message delay

19
Cont.

• Overhead

1020 performance improvement.
20
1020 performance improvement.
21
Discussion

• Challenges in supporting scientific workflows
• Requirements on domain specific experiments
• Generic workflow support and domain specific
  applications
• Existing workflow management systems are diverse
  in functionality, design and user support
• Related work
• Interoperability among workflow systems (sister
  Link project)
• Resource level e.g., Kepler invokes Tavernas
  resources

22
Applications of workflow bus

• Use case 1
• A user has workflow in Taverna
• Some functionality is missing in Taverna but can
  be provided by Triana
• He can develop the workflow in two systems, and
  run it via the workflow bus
• Use case 2
• A user wants to execute a Taverna or Triana
  workflow in multiple instances with different
  input data

23
Conclusions

• A workflow bus is a feasible approach to realize
  generic e-Science framework
• Multi agent technology provides a distributed
  environment for decomposing and encapsulating
  control intelligence
• Ptolemy II provides different computing paradigms
  which give user freedom to execute workflows

24
Future work

• Working on developing a scenario manager for
  Kepler engine.
• Synchronized data flow is currently used more
  computing modes will be evaluated.
• Data provenance for workflow bus.

25

• Referneces
• Z. Zhao A. Belloum H. Yakali P.M.A. Sloot and
  L.O. Hertzberger Dynamic Workflow in a Grid
  Enabled Problem Solving Environment, in
  Proceedings of the 5th International Conference
  on Computer and Information Technology (CIT2005),
  pp. 339-345 . IEEE Computer Society Press,
  Shanghai, China, September 2005.
• Z. Zhao A. Belloum A. Wibisono F. Terpstra
  P.T. de Boer P.M.A. Sloot and L.O. Hertzberger
  Scientific workflow management between
  generality and applicability, in Proceedings of
  the International Workshop on Grid and
  Peer-to-Peer based Workflows in conjunction with
  the 5th International Conference on Quality
  Software, pp. 357-364. IEEE Computer Society
  Press, Melbourne, Australia , September 19th-21st
  2005.
• Z. Zhao A. Belloum P.M.A. Sloot and L.O.
  Hertzberger Agent Technology and Generic
  Workflow Management in an e-Science Environment,
  in Hai Zhuge and G.C. Fox, editors, Grid and
  Cooperative Computing - GCC 2005 4th
  International Conference, Beijing, China, in
  series Lecture Notes in Computer Science, vol.
  3795, pp. 480-485. Springer, November 2005. ISBN
  3-540-30510-6. (DOI 10.1007/11590354_61)
• Z. Zhao A. Belloum P.M.A. Sloot and L.O.
  Hertzberger Agent technology and scientific
  workflow management in an e-Science environment,
  in Proceedings of the 17th IEEE International
  conference on Tools with Artificial Intelligence
  (ICTAI05), pp. 19-23. IEEE Computer Society
  Press, Hongkong, China, November 14th-16th 2005.
• Acknowledgement
• Suresh Booms
• All the members in VL-e SP2.5