GridFlow: Workflow Management for Grid Computing

1
GridFlow Workflow Management for Grid Computing

• Junwei Cao (???)
• CC Research Labs, NEC Europe Ltd., Germany
• Stephen A. Jarvis and Graham R. Nudd
• Dept. of Computer Science, Univ. of Warwick, UK
• Subhash Saini
• NASA Ames Research Center, USA

2
Outline

• Background Grid Workflow
• System Architecture
• GridFlow User Portal
• Global Grid Workflow Management
• Local Grid Sub-workflow Scheduling
• Fuzzy Timing Techniques
• Summary
• Ongoing and Future Work

3
Background Grid Workflow

• Workflow Definition
• WPDL, BPEL4WS, GSFL, ASCI Grid,
• Workflow Systems
• WebFlow, Symphony, GridAnt, BPWS4J, TENT,
• Component-based Systems
• CCA/XCAT, SCIRun, CXML,
• Other Systems
• Condor DAGMan, UNICORE, MyGrid, GEMSS, GridLab,
  BioOpera, USC Grid failure handling,

4
Grid Resources

• Grid Resource A particular grid resource is a
  high-end computing or storage resource that can
  be accessed remotely.
• Local Grid A local grid consists of multiple
  grid resources that belong to one organization.
• Global Grid The global grid includes all grid
  resources that belong to different organizations
  within a virtual organization.

5
Grid Tasks

• Task Tasks are the smallest elements in a grid
  workflow, e.g. MPI PVM programs.
• Sub-workflow A sub-workflow is a flow of closely
  related tasks that is to be executed in a
  predefined sequence on grid resources of a local
  grid (within one organization).
• Workflow A grid application can be represented
  as a flow of several different activities, each
  activity represented by a sub-workflow.

6
Grid Management

• Mapping grid workflows to the global grid
• Mapping grid sub-workflows to local grids
• Mapping grid tasks to grid resources

7
System Architecture
8
PACE Performance Prediction
Application Tools
Evaluation Engine
Resource Tools
9
Titan Resource Scheduling

• Heuristic
• Evolutionary
• Near-optimal
• Makespan
• Idletime
• Deadlines

10
ARMS Grid Management

• Agent structure
• Communication layer
• Decision-making layer
• Local management layer
• Agent hierarchy
• Service advertisement
• Service discovery
• Agent Capability Tables

A
User
11
GridFlow User Portal
12
Global Grid Workflow Management
/ 5 / 5
/ 7 / 12
13
Local Grid Sub-workflow Scheduling

• Scheduling a flow of tasks onto grid resources
  within a local grid is very similar to the
  process that schedules a workflow onto different
  local grids. There are two challenges
• It is a difficult task to provide an accurate
  prediction on task/workflow start, execution and
  end times.
• Multiple tasks from different sub-workflows may
  require the same grid resource at the same time.

14
Fuzzy Timing Techniques

• Turning the prediction accuracy into a fuzzy
  concept that is represented using fuzzy numbers.

p1(t)0.5(0,2,6,7)
p2(t)(2,4,4,6)
15
Fuzzy Number Operations
latest
earliest
min
sum
max
16
Resource Conflict Solving I

• The start time of a task cannot be configured
  with the latest end time of its pre-tasks
  directly, since other tasks exists that may use
  the same resource at the same time.

• A first-come possibly-first-serve policy is
  adopted. This does not order the conflictive
  tasks explicitly, but adds some information on
  degrees of possibilities of task start times.

17
Resource Conflict Solving II

• All possible start sequences are considered and
  are combined to provide an estimation of the end
  time.

18
Summary

• GridFlow is a prototype grid workflow management
  system, focusing on grid workflow simulation and
  scheduling.
• GridFlow is based on a specific grid resource
  management infrastructure implemented using
  agent-based methodologies and performance-driven
  scheduling technologies.
• Making grid workflow management a reality also
  requires to address general grid computing
  challenges openness, standards, security and QoS
  support.

19
Ongoing Work Applications

• Developing grid enabled medical simulation
  services (GEMSS) using GT3
• Developing grid performance services based on
  historical information analysis
• Developing medical application workflows using
  BPEL4WS

20
Future Work Agile Computing
Efficiency (Cheap, Large-scale, Pervasive,
Continuous, Massive)
Grid Computing P2P Computing Internet Computing
Agile (Grid) Computing
Cluster Computing HPC Supercomputing
Flexibility (Performance, Adaptation, QoS,
Individualization)

• Workflow techniques one of keys for next
  generation agile (grid) computing

21
For More Information

• http//www.dcs.warwick.ac.uk/hpsg/
• http//www.ccrl-nece.de/cao/
• http//www.ccrl-nece.de/gemss/
• http//www.agilecomputing.org/
• mailtocao_at_ccrl-nece.de