Grid Computing Research @ Melbourne: Gridbus Perspective

1
Grid Computing Research _at_ Melbourne Gridbus
Perspective

• Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS)
LaboratoryDept. of Computer Science and Software
EngineeringThe University of Melbourne,
Australiawww.gridbus.org
Gridbus Sponsors
2
Grids _at_ Various Schools in MelbourneTechnology
Creation -gt Applications

• Faculty of Engineering CS/EE/Geomatics
• GRIDS Lab and the Gridbus Project technology
  focus
• SensorWeb (NICTA)
• Natural Language Processing
• Virtual organisation for tsunami research
• Faculty of Science - Physics
• Belle and LHC Grid
• Australian Virtual Observatory
• Faculty of Medicine HFI, Anatomy
• MRI Grid
• Distributed Kidney Modelling
• Social Sciences/Humanities
• The development of a collaborative distributed
  environment for humanities research projects
  whose primary data takes the form of ethnographic
  audiovisual media.
• A Grid-enabled technology for the sharing of
  graphics, audio, video and text between social
  science research groups working internationally.
• Information Tech. Services Division
• MARC (Melbourne Advanced Research Computing)/HPC

3
Classes of Grid Services

• Computational Services CPU cycles
• Pooling computing power SETI_at_Home, TeraGrid,
  AusGrid, ChinaGrid, IndiaGrid, UK Grid,
• Data Services
• Collaborative data sharing generated by
  instruments, sensors, persons LHC Grid, Napster
• Application Services
• Access to remote software/libraries and license
  managementNetSolve
• Interaction Services
• eLearning, Virtual Tables, Group Communication
  (Access Grid), Gaming
• Knowledge Services
• The way knowledge is acquired, processed and
  manageddata mining.
• Utility Computing Services
• Towards a market-based Grid computing Leasing
  and delivering Grid services as ICT utilities.

Utility Grid
Users
Knowledge Grid
Interaction Grid
ASP Grid
Data Grid
infrastructure
Computational Grid
4
Grid Challenges
5
GRIDS Lab _at_ Melbourne
Education
R D

• Youngest and one of the rapidly growing research
  labs in our School/University
• Founded in 2002
• Houses
• Research Fellows/PostDocs
• Research Programmers
• PhD candidates
• Honours/Masters students
• Funding
• National and International organizations
• Australian Research Council DEST
• Many industries (Sun, StorageTek, Microsoft, IBM,
  Microsoft)
• University-wide collaboration
• Faculties of Science, Engineering, and Medicine
• Many national and international collaborations.
• Academics
• Industries
• Software
• Widely in academic and industrial users.
• Publication

Community Services e.g., IEEE TC for Scalable
Computing
6
Selected Funded Projects 2004-2006

• Australian Research Council
• Discovery Projects
• "The Development of Data Grids for High Energy
  Physics"
• "Economic Scheduling for Efficient Management of
  Clusters and their Cooperative Federation"
• Application Projects
• "Development of e-Research Tools for an MRI Grid
  Computing Facility"
• "e-Research Grid Environment for Integration of
  Distributed Kidney Models and Resources

7
The Gridbus Project _at_ MelbourneEnable Leasing
of ICT Services on Demand
WWG
Gridbus
Pushes Grid computing into mainstream computing
8
(No Transcript)
9
What does Grid players require?

• They need tools and technologies that help them
  in value expression, value translation, and value
  enforcement.
• Grid Service Consumers (GSCs)
• How do I express QoS requirements ?
• How do I trade between timeframe cost ?
• How do I map jobs to resources to meet my QoS
  needs?
• How do I manage Grid dynamics and get my work
  done?
• Grid Service Providers (GSPs)
• How do I decide service pricing models ?
• How do I specify them ?
• How do I translate them into resource allocations
  ?
• How do I enforce them ?
• How do I advertise attract consumers ?
• How do I do accounting and handle payments?

10
Solution 1 Service Oriented Architecture (SOA)

• A SOA is a contractual architecture for offering
  and consuming software as services.
• There are four entities that make up an SOA
• service provider,
• service registry, and
• service consumer (also known as service
  requestor).
• The functions or tasks that the service provider
  offers, along with other functional and technical
  information required for consumption, are defined
  in
• the service definition or contract.

registry
contract
provider
consumer
11
Solution 2 Market-Oriented Grid Computing - (a)
Sustained Resourced Sharing and (b) Effective
Management of Shared Resources
Grid Economy
12
Service-Oriented Grid Architecture
Data Catalogue
Grid Bank
Information Service
Grid Market Services
Sign-on
HealthMonitor
Info ?
Grid Node N

Grid Explorer

Secure
ProgrammingEnvironments
Job Control Agent
Grid Node1
Applications
Schedule Advisor
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource Reservation
Misc. services

Deployment Agent
JobExec
Resource Allocation
Storage
Grid Resource Broker

R1
R2
Rm
Core Middleware Services
Grid Service Consumer
Grid Service Providers
13
Gridbus and Complementary Technologies
realizing Utility Grid

Grid Applications
Portals
Science
Commerce
Engineering
Collaboratories

X-Parameter Sweep Lang.
Workflow
ExcellGrid
Gridscape
MPI
User-LevelMiddleware

Grid Brokers
Gridbus Data Broker
Workflow Engine
Nimrod-G
Grid Exchange Federation
Grid MarketDirectory
Globus
Unicore
Grid Storage Economy
GridBank

Core Grid Middleware
Alchemi
NorduGrid
XGrid
Grid Economy
.NET
JVM
Condor
SGE
Tomcat
PBS
Libra
Grid Fabric Software
Mac
AIX
Solaris
Windows
Linux
IRIX
OSF1
Grid Fabric Hardware
Worldwide Grid
14
On Demand Assembly of Services Putting Them All
Together
15
(No Transcript)
16
Alchemi .NET-based Enterprise Grid Platform
Web Services
Alchemi Manager
Web Services
Internet
Alchemi Users
Internet

• SETI_at_Home like Model
• General Purpose
• Dedicated/Non-dedicate workers
• Role-based Security
• .NET and Web Services
• C Implementation
• GridThread and Job Model Programming
• Easy to setup and use
• Widely in use!

Alchemi Worker Agents
17
Some Users of Alchemi
Tier Technologies, USA Large scale document
processing using Alchemi framework
Satyam Computers Applied Research Laboratory,
India Micro-array data processing using Alchemi
framework
CSIRO, Australia Natural Resource Modeling
The University of Sao Paulo, Brazil The Alchemi
Executor as a Windows Service
stochastix GmbH, Germany Serving clients in
International Banking/Finance sector
The Friedrich Miescher Institute (FMI) for
Biomedical Research, Switzerland Patterns of
transcription factors in mammalian genes
Many users in Universities See next for an
example.
18
Agenda

• Introduction
• Utility Networks and Grid Computing
• Application Drivers and Various Types of Grid
  Services
• Global Grids and Challenges
• Security, resource management, pricing models,
• Service-Oriented Grid Architecture and Gridbus
  Solutions
• Market-based Management, GMD, Grid Bank, Alchemi
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Summary and Conclusion

19
Grid Service Broker (GSB)

• A resource broker for scheduling task farming
  data Grid applications with static or dynamic
  parameter sweeps on global Grids.
• It uses computational economy paradigm for
  optimal selection of computational and data
  services depending on their quality, cost, and
  availability, and users QoS requirements
  (deadline, budget, T/C optimisation)
• Key Features
• A single window to manage control experiment
• Programmable Task Farming Engine
• Resource Discovery and Resource Trading
• Optimal Data Source Discovery
• Scheduling Predications
• Generic Dispatcher Grid Agents
• Transportation of data sharing of results
• Accounting

20
Gridbus Broker Architecture
Gridbus Client
Gridbus Client
Gribus Client
(Bag of Tasks Applications)
App, T, , Opt
(Data Grid Scheduler)
Gridbus Farming Engine
Schedule Advisor
Trading Manager
RecordKeeper
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS

GE GIS, NWS
Grid Info Server
RM TS
G

Data Catalog
Data Node
C

U
G
Unicore enabled node.
Globus enabled node.
L
A
RM Local Resource Manager, TS Trade Server
Alchemi enabled node.
21
Gridbus Broker Separating applications from
different remote service access enablers and
schedulers
Application Development Interface
Single-sign on security
Alogorithm1
SchedulingInterfaces
AlogorithmN
Plugin Actuators
Data Store
Access Technology
SRB
Grid FTP
22
Gridbus Services for eScience applications

• Application Development Environment
• XML-based language for composition of task
  farming (legacy) applications as parameter sweep
  applications.
• Task Farming APIs for new applications.
• Web APIs (e.g., Portlets) for Grid portal
  development.
• Threads-based Programming Interface
• Workflow interface and Gridbus-enabled workflow
  engine.
• Resource Allocation and Scheduling
• Dynamic discovery of optional computational and
  data nodes that meet user QoS requirements.
• Hide Low-Level Grid Middleware interfaces
• Globus (v2, v4), SRB, Alchemi, Unicore, and
  ssh-based access to local/remote resources
  managed by XGrid, Condor, SGE.

23
Click Here for Demo
Figure 3 Logging into the portal.
Drug Design Made Easy!
24
Deadline (D) and Budget (B) Constrained
Scheduling Algorithms
25
(No Transcript)
26
GridSim A Toolkit for Modeling and Simulation of
Various Types of Grids
Application, User, Grid Scenarios Input and
Results
. . .
Application Configuration
Resource Configuration
Visual Modeler
Grid Scenario
Output
Grid Resource Brokers or Schedulerss Simulation
GridSim Toolkit
Application Modeling
Information Services
Resource Allocation
Statistics
Resource Entities
AuctionFramework
Add your own policy for resource allocation
Resource Modeling and Simulation (with Time and
Space shared schedulers)
Clusters
Single CPU
Reservation
SMPs
Load Pattern
Network
Basic Discrete Event Simulation Infrastructure
SimJava
Distributed SimJava
Virtual Machine (Java, cJVM, RMI)
Distributed Resources
PCs
Workstations
Clusters
SMPs
27
Selected Funded Projects 2006-2009

• Australian Research Council
• Discovery Project
• QoS-based Scheduling of e-Research Application
  Workflows on Global Grids
• QoS-based Workflow Scheduling
• SLA-based Resource Allocation
• Dept. of Science, Training and Education
• International Science Linkage
• The Utility Grid Project Autonomic and
  Utility-oriented Global Grids for Powering
  Emerging E-Research Applications
• Links Gridbus Efforts to Many International
  Efforts

28
(No Transcript)
29
SLA-based Allocation of Cluster Resources
30
InterGrid Internetworking of Islands of Grids
5th International Conference on Grid and
Cooperative Computing Changsha, China, October
2006

• Rajkumar Buyya
• Grid Computing and Distributed Systems
• (GRIDS) Laboratory Dept. of Computer Science and
  Software EngineeringThe University of Melbourne,
  Australia
• www.gridbus.org/gridsim

31
The Outcomes of Grid Computing thus Far Islands
of Grids
32
InterGrid An architecture for Internetworking of
islands of Grids
33
The InterGrid How to get there?
34
New Research Challenges

• Decentralised service discovery from
  gateways-based to leaf nodes
• Protocols and Business models for Peering
  between Grids
• Application models that support growing and
  shrinking of application
• Resource management and scheduling
• SLA based resource allocation.

35
Summary and Conclusion

• Grids have emerged as enabler for
  Cyberinfrastructure that powers e-Science and
  e-Business applications.
• Delivering ICT services as computing utilities.
• Australia is actively developing Grid
  technologies and applications
• Recently Australian Govt. is pushing Grids
  heavily under e-Research banner.
• Melbourne is leading a number of Grid projects
  nationally.
• SOA Market-based Grid Management Utility
  Grids
• Grids allow users to dynamically lease Grid
  services at runtime based on their quality, cost,
  availability, and users QoS requirements.
• Grids offer enormous opportunities for realizing
  e-Science and e-Business at global level.

36
Thanks for your attention!
We Welcome Cooperation in Research and
Development! http/www.gridbus.org