Vladimir Korkhov

1
Resource Management issues in Virtual Laboratory
environment

• Vladimir Korkhov
• University of Amsterdam, Faculty of Science, CAPS
  group
• vkorkhov_at_science.uva.nl

2
Virtual Laboratory for e-Science

• A collaborative analysis environment for applied
  experimental science
• Offers effective and transparent utilization of
  distributed resources
• Support scientists in building applications by
  providing ready-to-use application components and
  means for developers to create such components

3
Virtual Lab Objectives

• Designing middleware to bridge the gap between
  fundamental services of Grid and application
  layer
• Enable VL users to define, execute and monitor
  their experiments on Grid resources transparently
• Provide VL users
• location independent experimentation
• easy to use experimentation environment
• assistance during experiment
• Rapid development of application prototypes to
  check ideas and to learn

4
Virtual Laboratory Overview
http//www.vl-e.nl
5
VL experiment

• PFT is a formalized abstraction of common data
  and processing steps that are typically involved
  in a certain type of scientific experiment
• Study is the instantiation of PFT consisting of
  descriptions of steps involved in the PFT
• Topology is composed of computational processes
  in a study representing a data flow graph

Source data definition
Experiment Topology
Processing
Data inspection
Visual control
Additional dataset
Postprocessing
Reference background
Data image storage description
Experiment PFT
6
VL experiment topology

• Driven by dataflow
• Processing components on the Grid modules -
  submitted to Globus GRAM
• Support of data streaming, not only batch
  processing of data files
• Connections between modules typed I/O ports

7
VL module structure

• Module experiment building block independent
  data processing entity with specialized
  functionality, e.g. FFT module
• I/O Ports to exchange typed or untyped data
• Parameters controlled by Run-Time System during
  experiment execution
• Development
• API for module developers (C/Java)
• Control interface (CORBA) used by Run Time
  System to connect modules, to start processing,
  to set parameters

DATA
DATA
DATA
8
VL Grid core Run-Time System (RTS)

• Support data flow experiment topologies
• Tasks
• Submit experiment topology to resource manager
  and get mapping of modules to available Grid
  resources
• Instantiate modules, submit them to Grid
  resources
• Connect submitted modules using I/O ports
• Start experiment when all modules are submitted
  and ready to run, control experiment execution
• Control module parameters during execution
• Redirect graphical modules output to a virtual
  desktop

9
Resource Management goals

• Maximizing application performance
• Minimizing idle time of computing systems,
  increasing number of jobs performed in a time
  unit
• Eliminating conflicts between applications during
  runtime (e.g. advanced reservation)

10
Grid Resource Management issues

• Resources
• Dynamic the set of available resources and their
  state is varied in time
• Shared influence of other users' applications
• Heterogeneous various platforms
• No centralized control over resources
• Resources are in different administrative domains

11
Resource Manager

• Analyze application requirements
• Analyze resource information from information
  services
• Discover, select and locate suitable resources
• Map modules to resources according to application
  model and performance metric to achieve most
  efficient execution
• Resource state monitoring and prediction,
  rescheduling and task migration, advanced
  resource reservation

12
Resource Management system
13
VL Application info
14
Application requirements
15
Resource description
Mds-Computer-platform i686 Mds-Cpu-Cache-l2kB
512 Mds-Cpu-speedMHz 2386 Mds-Cpu-Total-count
1 Mds-Cpu-Total-Free-15minX100
041 Mds-Cpu-Total-Free-1minX100
091 Mds-Cpu-Total-Free-5minX100
055 Mds-Memory-Ram-Total-sizeMB
501 Mds-Memory-Ram-Total-freeMB
188 Mds-Memory-Vm-Total-sizeMB
964 Mds-Memory-Vm-Total-freeMB
612 Mds-Fs-sizeMB 8454 Mds-Fs-freeMB 3319
16
Module costs estimation
17
Application model
18

• Task minimize target function
• Heuristic algorithms
• Random search algorithms (simulated annealing)
• Application model is used to estimate generated
  schedule

19
Heuristic algorithms
20
Simulated annealing
21
Pseudo code for SA
22
Problems to solve
23
Related projects
24
VL applications

• Materials Analysis of Complex Surfaces (MACS)
• Magnetic Resonance Imaging Scanner (MRI Scanner)
• DNA Array genome expression

25
(No Transcript)
26
Towards service oriented Grids
27

• Application performance prediction, module
  templates
• Module migration
• Move to WS-RF
• Represent modules as separated processing and
  state which can help in resource management and
  load balancing
• Enhance fault tolerance by dividing data streams
  to elementary data blocks that can be
  re-processed automatically in case of failure
• Automated module deployment
• Fault tolerance in data-streaming applications
• Decentralized RM

28
(No Transcript)