NimrodG GRID Resource Broker and Computational Economy

1
Nimrod/G GRID Resource Broker and Computational
Economy
David Abramson, Rajkumar Buyya, Jon Giddy School
of Computer Science and Software
Engineering Monash University, Melbourne,
Australia Email davida, rajkumar,
jon_at_csse.monash.edu.au
2
Computing Power (HPC) Drivers

• Solving grand challenge applications using
  computer modeling, simulation and analysis

Life Sciences
Aerospace
Digital Biology
CAD/CAM
Military Applications
Military Applications
Military Applications
3
Computing Platforms Evolution Breaking
Administrative Barriers
?
PERFORMANCE
Administrative Barriers
Individual Group Department Campus State National
Globe Inter Planet Universe
Desktop (Single Processor?)
SMPs or SuperComputers
Local Cluster
Enterprise Cluster/Grid
Global Cluster/Grid
Inter Planet Cluster/Grid ??
4
Global Computational Grids
5
Grid Resource Management Challenging Issues

• Authentication (once)
• Specify simulation (code,
  resources, etc.)
• Discover resources
• Negotiate authorization, acceptable use, Cost,
  etc.
• Acquire resources
• Schedule Jobs
• Initiate computation
• Steer computation
• Access remote data-sets
• Collaborate on results
• Account for usage

Domain 1
Domain 2
Ack. globus..
6
Grid Components
Applications and Portals
Grid Apps.

Prob. Solving Env.
Collaboration
Engineering
Web enabled Apps
Scientific
Development Environments and Tools
Grid Tools

Libraries
Web tools
Languages
Resource Brokers
Monitoring
Debuggers
Distributed Resources Coupling Services
Grid Middleware

QoS
Comm.
Sign on Security
Information
Process
Data Access
Local Resource Managers
TCP/IP UDP

Operating Systems
Queuing Systems
Libraries App Kernels
Grid Fabric
Networked Resources across Organisations

Clusters
Data Sources
Storage Systems
Scientific Instruments
Computers
7
Computational Market Model for Grid Resource
Management
Grid Information Server(s)
Info ?
Health Monitor

Grid Explorer

Application
Job Control Agent
Grid Node1
Schedule Advisor
Trading
Trade Server
Charging Alg.
Trade Manager
Accounting
Resource Reservation
Other services

Deployment Agent
Jobs
Resource Allocation
Grid User
Grid Resource Broker

R1
R2
Rm
Grid Resource/Control Domains
Grid Middleware
8
What is Nimrod/G ?

• A global scheduler for managing and steering task
  farming (parametric simulation) applications on
  computational grid based on deadline and
  computational economy.
• Key Features
• A single window to manage control experiment
• Resource Discovery
• Trade for Resources
• Scheduling
• Steering data management
• It allows to study the behaviour of some of the
  output variables against a range of different
  input scenarios.

9
Nimrod/G Grid Resource Broker Architecture
Nimrod/G Client
Nimrod/G Client
Nimrod/G Client
Nimrod/G Engine
Schedule Advisor
Grid Bookkeeper
Trading Manager
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS
Globus,Legion, Condor-g,, Ninf,etc.
GE GIS
Grid Information Server(s)
RM TS
RM TS
RM TS
L
N
G
G
Ninf enabled node.
Globus enabled node.
L
C
RM Local Resource Manager, TS Trade Server
Condor enabled node.
10
Nimrod/G Interactions
Resource location
Grid Info servers
Scheduler
Trade Server
Resource allocation (local)
Prmtc.. Engine
Queuing System
Job Wrapper
Dispatcher
Process server
Root node
Gatekeeper node
Computational node
11
A Nimrod/G Client
Deadline
Legion hosts
Globus Hosts
Bezek is in both Globus and Legion Domains
12
Change deadline/budget Monitor activities
13
Adaptive Scheduling algorithms ...
Locate more Machines
Locate Machines
Establish Rates
Re-distribute Jobs
Meet requirements ? Deadlines and Budget
Distribute Jobs
14
(No Transcript)
15
Nimrod/O Automatic Design Optimization

• Search parameter space rather than exploring all
  options

Active Sheets - Excell cell func()s execution on
the Grid
16
Related Work Further Info.

• AppLeS (UC. San Diego)
• application level scheduling templates
  case-by-case for different Apps, soon PST.
• NetSolve (UTK/ORNL) -- API for creating farms
• SETI _at_ Home, Distributed.net, .
• Millennium (UC. Berkeley)
• remote execution environment on clusters and
  supports computational economy
• CODINE/GRD (Genias/Gridware)
• meets deadline by dominating over others share.
• Mariposa- Distributed Database system (UC,
  Berkeley)
• query with budget, creates sub-query
  dividesbudget, trades with (remote) servers
• More Info -- www.csse.monash.edu.au/davida/nimrod
  .html