Marco Mililotti

1
Genetic Algorithms for GridComputing
schedulingwith multiple constraints

• Marco Mililotti
• Caspur

... Genetic Algorithms use random choice as a
tool, but do not generate their results
randomly.
2
Grid Computing

• Power Grid metaphor
• Resource utilization optimization
• Coordinated and integrated use of resources

3
A few problematics

• Resource management distributed
  heterogenous distinct access policies
  dynamical change of availability and load.
• Scheduling
• fundamental for optimization of applications
  efficiency
• utilization of predictive models for evaluation
  of the applications efficency
• centralized / distributed

4
Our approach

• Problem
• Superscheduler (metascheduler) for job allocation
  using informations on single nodes of the Grid
  and informations on single jobs to be allocated.
• Objective
• A flexible system for (sub)optimum solution
  research minimizing the total execution time at
  each Grid node (maximizing computing resource
  utilization). The study of optimization
  parameters through
• an ad hoc simulator.
• Characterizing
• Resources (computing and network resources)
• Jobs

5
Superscheduler
job pool
Grid Level
superscheduler
Local Level
scheduler
scheduler
scheduler
Not suitable for scheduling of jobs with small
granularity with respect to resources
6
Resource classification

• SHMEM SMP, traditional supercomputer (vector
  processor)
• DMEM cluster of SMP on proprietary switch
• PCC low cost cluster, NOW, Beowulf

Networking total connected topology each node
has specified a given bandwidth.
7
Resource definition

• Computing nodes represented by set of machines
• Machines characterized by
• architecture
• number or flops
• network connection

8
Job definition

• Performance on each architecture defined by
• flops
• time (an evaluation of computing complexity)
• efficiency (an evaluation of scalability on
  different architectures)
• An example
• arch dmem flops 70 time 100 effic 100
• Data position on the Grid expressed by
• the node hosting data
• the fraction of data available at the specified
  node
• An example
• id 3 fract 80

9
The simulator
10
Architecture of the simulator...
'Resource Discovery' 'Resource Allocation' Others
Grid interfaces
11
Architecture of the simulator...
Envl
Node simulator
Data for the Simulation
'Resource Discovery' 'Resource Allocation' Others
Grid interfaces
Scheduler
Job
Environment
12
Architettura simulatore...
Schedule Output
CmdLine
Genetic Algorithm
Timing for a job (Trelease, Tdata...)
Node simulator
Data for the Simulation
'Resource Discovery' 'Resource Allocation' Others
Grid interfaces
Scheduler
Job
Environment
13
Results
14
Results...
Single node, single machine example with 9 jobs.
Optimum and sub-optimum allocation results
15
Results...
Single node, two machines example with 12 jobs.
Optimum allocation result
16
Results...
Two nodes, two machines example with 13 jobs.
Optimum allocation result with balancing of
resources allocated.
17
Results...
Two nodes, two machines Example with 24
jobs. Optimum and sub-optimum allocation results
18
Results...
Optimum allocation
Tmax 130
19
Results...
Sub-optimum allocation
Tmax 150
20
Results...
Two nodes, two machines with data access
Example with 13 jobs and data access
constraints. Optimum allocation results.
21
Results...
Optimum allocation with data access
Tmax 100 Tmax with data 102
22
Results...
Optimum allocation with constraints on execution
architecture of a few jobs. Tmax 170 Njobs
32 Jobs in red must be run on DMEM
23
Results
Sub-optimum allocation. In green, resource not
allocated. Tmax 180 Njobs 32 Jobs in red
are correctly allocated on DMEM
24
Conclusions and future developments

• Good quality of sub-optimum solutions
• in the 60 of solutions we have a downgrade of
  10 with respect to the optimum
• in the 90 of solutions we have a downgrade of
  20 with respect to the optimum
• Flexibles optimization techniques. New approachs
  may be possibles
• History log for job and user characterization
• New constraints (i.e. whished deadline for job
  completion).
• More parameters for job description
• Integration of different metaheuristics,
  multi-objective optimization

• Buona qualità delle soluzioni sub-ottime trovate
• nel 60 dei casi si ha un peggioramento del 10
  sull'ottimo
• nel 90 dei casi si ha un peggioramento del 20
  sull'ottimo
• Tecniche di ottimizzazione flessibili. Possibili
  nuovi approcci
• 'History log' per caratterizzare utenti e job
• Nuovi vincoli (es. 'deadline' desiderata per il
  completamento del job)
• Ulteriori parametri per il job
• Integrazione metaeuristiche, ottimizzazione
  multi-obiettivo

25
Thank you Marco Mililotti m.mililotti_at_caspur.it