Resource Allocation and Scheduling for Workflows

1
Resource Allocation and Scheduling for Workflows

• Gurmeet Singh, Carl Kesselman, Ewa Deelman

2
Problem

• Given a workflow and a Grid, how to allocate
  resources and schedule the workflow in order to
  optimize performance.
• Which resources should be provisioned
• When they should be provisioned
• How much capacity should be provisioned
• Scheduling on provisioned resources

3
Previous Work

• No allocation in case of time sharing systems.
• Combined allocation and scheduling in case of
  queuing based systems.
• Scheduling of data is coupled with scheduling of
  computations.
• Operations Research (inventory control, MRP)

4
Formal Model

• An allocation plan consisting of allocation
  requests and cost
• AP ar1, ar2, , arn ar
  lts,t,n,d,hgt
• CA(AP) f(ar1, ar2, arn) e.g. utilization,
  wait time
• A policy, S, for scheduling partial workflows
  and associated cost, CSCH
• costs can be makespan, lateness, reliability etc.
  
• The goal is to minimize the total cost of
  allocation and scheduling.
• CT g(CA(AP), CSCH)

5
Relation to previous work

• The AP defines the Virtual Grid.
• Can be used to explain previous scheduling
  approaches based on the appropriate cost
  formulations.
• Dedicated resources imply constant allocation
  cost
• Others imply the cost of allocation is the wait
  time associated with it.

6
Approaches to Allocation

• Deterministic (using reservations)
• Stochastic (using predictions)
• A combination of both
• The space of possible allocation plans is likely
  to be exponential even with reservations. Search
  based heuristics might be useful.

7
Execution stack for SCEC, Montage workflows
TG/OSG
Glidein
AP
Condor-G
Grid
S
Provisioned Resource
8
Preliminary Work

• Java based Grid Simulator
• A parametric task graph generator
• 4 Grid sites with background load
• Backfill based FCFS scheduling policy for Grid
  resources.
• Resources can be queried for earliest start time
  of a request.

9
Preliminary Work

• Preliminary results show good results with very
  simple allocation strategy and small workflows.

10
Current Research

• Modeling resource availability in Grids
• Heuristics for creating an allocation plan
• Greedy approach
• Search based metaheuristics
• Experiments to evaluate the performance.
• Implementation to be integrated with current
  planning and execution systems such as Pegasus.