LQCD Workflow Project

1
LQCD Workflow Project

• L. Piccoli
• October 02, 2006

2
Triana
3
Triana

• public class Job extends Unit
• // parameter data type definitions
• private String queue
• private int nodes
• /
• Called whenever there is data for the unit to
  process
• /
• public void process() throws Exception
  
• // Implementation

4
Triana

• Integration with existing system
• Control and data dependencies represented
• Substitute for current run makefile tool
• Creation of toolboxes for generic jobs
• Not as flexible
• Workflows can be turned into command line
  applications
• No Web Services or Grid capabilities used

5
Triana

• Requirements
• Workflow representation
• XML containing Triana specific tags
• Interaction with external scheduler
• Possible via task-by-task schedule
• As dependencies are resolved, the execution
  request is passed to campaign scheduler
• Schedule of complete workflow requires
  significant changes
• Campaign scheduler is in charge of resolving
  dependencies

6
Triana

• Requirements
• Scheduling (one or more workflows)
• Not present
• Modifications
• Submit workflows to an external scheduler
• Update workflow progress based on external
  monitoring information
• Fault Management
• Not present
• Triana services may be used for adding these
  features
• Need further investigation

7
Triana Summary
Language Java
GUI Yes
Script Yes (Generated through the GUI)
WF Represetation XML (Triana tags)
Scheduling No
External Scheduler No (Triana Services?)
Fault Tolerance No (Triana Services?)
Data Dependencies Yes
Control Dependencies Yes
GUI Execution Engine Coupling Yes (Plans to decouple)
Non-DAG Workflows Yes (Control loops)
8
System Architecture
9
This workflow represents the dependencies found
in the charm and bottom 2-pt run file (available
at http//whcdf03.fnal.gov/lqcd/RunWorkflowFramewo
rk). There are pseudo dependencies in this graph
(e.g. hQ923 do not depend on hQ127), forcing
the system to execute tasks in an order that fits
better to the available hardware. This
optimization is manual and requires knowledge of
the system by the user. The workflow scheduler
should be able to handle this situation
automatically.