Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5th Utility

1
Gridbus Middleware and Utility GridsBuilding
Autonomic and Market-Oriented Global Grids for
Delivering IT Services as the 5th Utility

• Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS)
LaboratoryDept. of Computer Science and Software
EngineeringThe University of Melbourne,
Australiawww.gridbus.org
Gridbus Sponsors
2
GRIDS Lab _at_ Melbourne
Education
R D

• Youngest and one of the rapidly growing research
  labs in our School/University
• Founded in 2002
• Houses
• Research Fellows/PostDocs
• Research Programmers
• PhD candidates
• Honours/Masters students
• Funding
• National and International organizations
• Australian Research Council DEST
• Many industries (Sun, StorageTek, Microsoft, IBM,
  Microsoft)
• University-wide collaboration
• Faculties of Science, Engineering, and Medicine
• Many national and international collaborations.
• Academics
• Industries
• Software
• Widely in academic and industrial users.
• Publication

Community Services e.g., IEEE TC for Scalable
Computing
3
(some) Books Co-authored edited
4
Agenda

• Introduction
• Utility Networks and Grid Computing
• Application Drivers and Various Types of Grid
  Services
• Global Grids and Challenges
• Security, resource management, pricing models,
• Service-Oriented Grid Architecture and Gridbus
  Solutions
• Market-based Management, GMD, Grid Bank, Alchemi
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Summary and Conclusion

5
4 Essential Utilities and Delivery Networks
6
(5) Computing Grid Delivering IT services as the
5th utility (Power Grid inspiration)
eScience eBusiness eGovernment eHealth Multilingua
l eEducation
7
Power Grid Inspiration Seamlessly delivering
electricity as a utility to users
8
Grid-like Vision

• In 1969, Leonard Kleinrock, one of the chief
  scientists of the original ARPA project which
  seeded the Internet, wrote
• "As of now, computer networks are still in their
  infancy, but as they grow up and become
  sophisticated, we will probably see the spread of
  "computer utilities", which, like present
  electric and telephone utilities, will service
  individual homes and offices across the country
• Despite major advances in hardware and software
  systems over the past 35 years, we are yet to
  realize this vision. How far are we still from
  delivering computing as a utility?
• Let us look into the ICT evolution and project
  the future.

9
Why Grid Computing Now?Let us look at the
Evolution of ICT
10
Computing and Communication Technologies
Evolution 1960-2010!

HTC

P2P

PDAs
Minicomputers


PCs

Workstations

Mainframes

Grids
COMPUTING

PC Clusters
Computing as Utility

Crays

MPPs

WS Clusters

XEROX PARC worm
e-Science
e-Business
IETF
W3C
TCP/IP
Ethernet
Communication
Mosaic
HTML
Web Services
Email
Sputnik
SocialNet
Internet Era
WWW Era
XML
ARPANET
1960
1970
1975
1980
1985
1990
1995
2000
2010
Control
Centralised
Decentralised
11
Computing is Scaling Towards
Inter-Planetary Level
SERV ICES PERFORMANCE
Administrative Barriers

• Individual
• Group
• Department
• Campus
• State
• National
• Globe
• Inter Planet
• Universe

Personal Device
SMPs or SuperComputers
Local Cluster
Global Grid
Inter Planet Grid
Enterprise Cluster/Grid
12
What is Grid? (It means different things to
different people)

• IBM
• On Demand Computing
• Microsoft
• .NET
• Oracle
• 10g
• Sun
• N1 Sun Grid Engine
• HP
• Adaptive Enterprise
• United Devices and related companies
• Harvesting Unused Desktop resources

13
What is Grid?Buyya et. al.

• A type of parallel and distributed system that
  enables the sharing, exchange, selection,
  aggregation of geographically distributed
  autonomous resources
• Computers PCs, workstations, clusters,
  supercomputers, laptops, notebooks, mobile
  devices, PDA, etc
• Software e.g., ASPs renting expensive special
  purpose applications on demand
• Catalogued data and databases e.g. transparent
  access to human genome database
• Special devices/instruments e.g., radio
  telescope SETI_at_Home searching for life in
  galaxy.
• People/collaborators.
• depending on their availability, capability,
  cost, and user QoS requirements.

Widearea
14
How does Grids look like?A Bird Eye View of a
Global Grid
Grid Information Service
Grid Resource Broker
Application
R2
R3
R4
R5
RN
Grid Resource Broker
R6
R1
Resource Broker
Grid Information Service
15
Classes of Grid Services / Types of Grids

• Computational Services CPU cycles
• Pooling computing power SETI_at_Home, TeraGrid,
  AusGrid, ChinaGrid, IndiaGrid, UK Grid,
• Data Services
• Collaborative data sharing generated by
  instruments, sensors, persons LHC Grid, Napster
• Application Services
• Access to remote software/libraries and license
  managementNetSolve
• Interaction Services
• eLearning, Virtual Tables, Group Communication
  (Access Grid), Gaming
• Knowledge Services
• The way knowledge is acquired, processed and
  manageddata mining.
• Utility Computing Services
• Towards a market-based Grid computing Leasing
  and delivering Grid services as ICT utilities.

Utility Grid
Users
Knowledge Grid
Interaction Grid
ASP Grid
Data Grid
infrastructure
Computational Grid
16
How Are Grids Used?
Utility computing
High-performance computing
Collaborative design
Financial modeling
High-energy physics
E-Business
Drug discovery
Life sciences
Data center automation
E-Science
Natural language processing Data Mining
Collaborative data-sharing
17
e-Science Environment Supporting Collaborative
Science

E-Scientist
Peers sharing ideas and collaborative

interpretation of data/results



Cyberinfrastructure
Distributed data
Remote
Visualization

2100

2100

2100

2100

Distributed computation

2100

21
00

2100

2100

Distributed instruments

Data Compute Service

18
Agenda

• Introduction
• Utility Networks and Grid Computing
• Application Drivers and Various Types of Grid
  Services
• Global Grids and Challenges
• Security, resource management, pricing models,
• Service-Oriented Grid Architecture and Gridbus
  Solutions
• Market-based Management, GMD, Grid Bank, Alchemi
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Summary and Conclusion

19
Grid Challenges
20
Some Grid Initiatives Worldwide

• Australia
• Nimrod-G
• Gridbus
• DISCWorld
• GrangeNet.
• APACGrid
• ARC eResearch
• Brazil
• OurGrid, EasyGrid
• LNCC-Grid many others
• China
• ChinaGrid Education
• CNGrid - application
• Europe
• UK eScience
• EU Grids..
• and many more...
• India
• Garuda

• USA
• Globus
• GridSec
• AccessGrid
• TeraGrid
• Cyberinfrasture
• and many more...
• Industry Initiatives
• IBM On Demand Computing
• HP Adaptive Computing
• Sun N1
• Microsoft - .NET
• Oracle 10g
• Infosys Enterprise Grid
• Satyam Business Grid
• StorageTek Grid..
• and many more
• Public Forums
• Global Grid Forum

27 million
1.3 billion 3 yrs
2? billion
120million 5 yrs
450million 5 yrs
486million 5 yrs
1.3 billion (Rs)
1 billion 5 yrs
http//www.gridcomputing.com
21
Open-Source Grid Middleware Projects
22
The Gridbus Project _at_ MelbourneEnable Leasing
of ICT Services on Demand
WWG
Gridbus
Pushes Grid computing into mainstream computing
23
(No Transcript)
24
Agenda

• Introduction
• Utility Networks and Grid Computing
• Application Drivers and Various Types of Grid
  Services
• Global Grids and Challenges
• Security, resource management, pricing models,
• Service-Oriented Grid Architecture and Gridbus
  Solutions
• Market-based Management, GMD, Grid Bank, Alchemi
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Summary and Conclusion

25
What do Grid players want?

• Grid Consumers
• Execute jobs for solving varying problem size and
  complexity
• Benefit by utilizing distributed resources wisely
• Tradeoff timeframe and cost
• Strategy minimise expenses
• Grid Providers
• Contribute resources for executing consumer jobs
• Benefit by maximizing resource utilisation
• Tradeoff local requirements market opportunity
• Strategy maximise return on investment

26
What do Grid players require?

• They need tools and technologies that help them
  in value expression, value translation, and value
  enforcement.
• Grid Service Consumers (GSCs)
• How do I express QoS requirements ?
• How do I trade between timeframe cost ?
• How do I map jobs to resources to meet my QoS
  needs?
• How do I manage Grid dynamics and get my work
  done?
• Grid Service Providers (GSPs)
• How do I decide service pricing models ?
• How do I specify them ?
• How do I translate them into resource allocations
  ?
• How do I enforce them ?
• How do I advertise attract consumers ?
• How do I do accounting and handle payments?

27
Principle 1 Service Oriented Architecture (SOA)

• A SOA is a contractual architecture for offering
  and consuming software as services.
• There are four entities that make up an SOA
• service provider,
• service registry, and
• service consumer (also known as service
  requestor).
• The functions or tasks that the service provider
  offers, along with other functional and technical
  information required for consumption, are defined
  in
• the service definition or contract.

registry
contract
provider
consumer
28
Principle 2 Market-Oriented (Grid) Computing-
(a) Sustained Resourced Sharing and (b)
Effective Management of Shared Resources
Grid Economy
29
Market-based Systems Self-managed and
Self-regulated systems.

• Complexity present in Grid systems is similar to
  one present in human economies.

30
Service-Oriented Grid Architecture
Data Catalogue
Grid Bank
Information Service
Grid Market Services
Sign-on
HealthMonitor
Info ?
Grid Node N

Grid Explorer

Secure
ProgrammingEnvironments
Job Control Agent
Grid Node1
Applications
Schedule Advisor
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource Reservation
Misc. services

Deployment Agent
JobExec
Resource Allocation
Storage
Grid Resource Broker

R1
R2
Rm
Core Middleware Services
Grid Service Consumer
Grid Service Providers
31
Gridbus and Complementary Technologies
realizing Utility Grid

Grid Applications
Portals
Science
Commerce
Engineering
Collaboratories

X-Parameter Sweep Lang.
Workflow
ExcellGrid
Gridscape
MPI
User-LevelMiddleware

Grid Brokers
Gridbus Data Broker
Workflow Engine
Nimrod-G
Grid Exchange Federation
Grid MarketDirectory
Globus
Unicore
Grid Storage Economy
GridBank

Core Grid Middleware
Alchemi
NorduGrid
XGrid
Grid Economy
.NET
JVM
Condor
SGE
Tomcat
PBS
Libra
Grid Fabric Software
Mac
AIX
Solaris
Windows
Linux
IRIX
OSF1
Grid Fabric Hardware
Worldwide Grid
32
On Demand Assembly of Services Putting Them All
Together
33
On Demand Assembly of Services Putting Them All
Together
34
Alchemi .NET-based Enterprise Grid Platform
Web Services
Alchemi Manager
Web Services
Internet
Alchemi Users
Internet

• SETI_at_Home like Model
• General Purpose
• Dedicated/Non-dedicate workers
• Role-based Security
• .NET and Web Services
• C Implementation
• GridThread and Job Model Programming
• Easy to setup and use
• Widely in use!

Alchemi Worker Agents
35
Some Users of Alchemi
Tier Technologies, USA Large scale document
processing using Alchemi framework
Satyam Computers Applied Research Laboratory,
India Micro-array data processing using Alchemi
framework
CSIRO, Australia Natural Resource Modeling
The University of Sao Paulo, Brazil The Alchemi
Executor as a Windows Service
stochastix GmbH, Germany Serving clients in
International Banking/Finance sector
The Friedrich Miescher Institute (FMI) for
Biomedical Research, Switzerland Patterns of
transcription factors in mammalian genes
Many users in Universities See next for an
example.
36
(No Transcript)
37
On Demand Assembly of Services Putting Them All
Together
38
Agenda

• Introduction
• Utility Networks and Grid Computing
• Application Drivers and Various Types of Grid
  Services
• Global Grids and Challenges
• Security, resource management, pricing models,
• Service-Oriented Grid Architecture and Gridbus
  Solutions
• Market-based Management, GMD, Grid Bank, Alchemi
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Summary and Conclusion

39
Grid Service Broker (GSB)

• A resource broker for scheduling task farming
  data Grid applications with static or dynamic
  parameter sweeps on global Grids.
• It uses computational economy paradigm for
  optimal selection of computational and data
  services depending on their quality, cost, and
  availability, and users QoS requirements
  (deadline, budget, T/C optimisation)
• Key Features
• A single window to manage control experiment
• Programmable Task Farming Engine
• Resource Discovery and Resource Trading
• Optimal Data Source Discovery
• Scheduling Predications
• Generic Dispatcher Grid Agents
• Transportation of data sharing of results
• Accounting

40
Gridbus Broker Architecture
Gridbus Client
Gridbus Client
Gribus Client
(Bag of Tasks Applications)
App, T, , Opt
(Data Grid Scheduler)
Gridbus Farming Engine
Schedule Advisor
Trading Manager
RecordKeeper
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS

GE GIS, NWS
Grid Info Server
RM TS
G

Data Catalog
Data Node
C

U
G
Unicore enabled node.
Globus enabled node.
L
A
RM Local Resource Manager, TS Trade Server
Alchemi enabled node.
41
Gridbus Broker Separating applications from
different remote service access enablers and
schedulers
Application Development Interface
Single-sign on security
Alogorithm1
SchedulingInterfaces
AlogorithmN
Plugin Actuators
Data Store
Access Technology
SRB
Grid FTP
42
Gridbus Services for eScience applications

• Application Development Environment
• XML-based language for composition of task
  farming (legacy) applications as parameter sweep
  applications.
• Task Farming APIs for new applications.
• Web APIs (e.g., Portlets) for Grid portal
  development.
• Threads-based Programming Interface
• Workflow interface and Gridbus-enabled workflow
  engine.
• Resource Allocation and Scheduling
• Dynamic discovery of optional computational and
  data nodes that meet user QoS requirements.
• Hide Low-Level Grid Middleware interfaces
• Globus (v2, v4), SRB, Alchemi, Unicore, and
  ssh-based access to local/remote resources
  managed by XGrid, Condor, SGE.

43
Click Here for Demo
Figure 3 Logging into the portal.
Drug Design Made Easy!
44
Excel Plugin to Access Gridbus Services
45
Adaptive Scheduling Steps
Discover More Resources
Discover Resources
Establish Rates
Evaluate Reschedule
Compose Schedule
Meet requirements ? Remaining Jobs, Deadline,
Budget ?
Distribute Jobs
46
Deadline (D) and Budget (B) Constrained
Scheduling Algorithms
47
(No Transcript)
48
Agenda

• Introduction
• Utility Networks and Grid Computing
• Application Drivers and Various Types of Grid
  Services
• Global Grids and Challenges
• Security, resource management, pricing models,
• Service-Oriented Grid Architecture and Gridbus
  Solutions
• Market-based Management, GMD, Grid Bank, Alchemi
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Summary and Conclusion

49
Case Study High Energy Physics and Data Grid

• The Belle Experiment
• KEK B-Factory, Japan
• Investigating fundamental violation of symmetry
  in nature (Charge Parity) which may help explain
  why do we have more antimatter in the universe
  OR imbalance of matter and antimatter in the
  universe?.
• Collaboration 1000 people, 50 institutes
• 100s TB data currently

50
Case Study Event Simulation and Analysis
B0-gtDD-Ks

• Simulation and Analysis Package - Belle Analysis
  Software Framework (BASF)
• Experiment in 2 parts Generation of Simulated
  Data and Analysis of the distributed data

?Analyzed 100 data files (30MB each) that were
distributed among the five nodes within
Australian Belle DataGrid platform.
51
Australian Belle Data Grid Testbed
VPACMelbourne
52
Belle Data Grid (GSP CPU Service Price G/sec)
G4
NA
G4
G6
VPACMelbourne
G2
Datanode
53
Belle Data Grid (Bandwidth Price G/MB)
32
33
36
G4
31
30
34
NA
38
31
G4
G6
VPACMelbourne
G2
Datanode
54
Deploying Application Scenario

• A data grid scenario with 100 jobs and each
  accessing remote data of 30MB
• Deadline 3hrs.
• Budget G 60K
• Scheduling Optimisation Scenario
• Minimise Time
• Minimise Cost
• Results

55
Observation
56
Agenda

• Introduction
• Utility Networks and Grid Computing
• Application Drivers and Various Types of Grid
  Services
• Global Grids and Challenges
• Security, resource management, pricing models,
• Service-Oriented Grid Architecture and Gridbus
  Solutions
• Market-based Management, GMD, Grid Bank, Alchemi
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Summary and Conclusion

57
Alessandro Volta in Paris in 1801 inside French
National Institute shows the battery while in the
presence of Napoleon I

• Fresco by N. Cianfanelli (1841)
• (Zoological Section "La Specula" of National
  History Museum of Florence University)

58
.and in the future, I imagine a Worldwide Power
(Electrical) Grid ...
Oh, mon Dieu !
What ?!?! This is a mad man
59
2006 - 1801 205 Years(Recent RD Delivering
Internet services via Electric cables).
2006
60
When are we going to get IT as the 5th
utility?(water, electricity, gas, telephone, IT)
eScience eBusiness eGovernment eHealth Multilingua
l eEducation
61
Summary and Conclusion

• Grids exploit synergies that result from
  cooperation of autonomous entities
• Resource sharing, dynamic provisioning, and
  aggregation at global level ?Great Science and
  Great Business!
• Grids have emerged as enabler for
  Cyberinfrastructure that powers e-Science and
  e-Business applications.
• SOA Market-based Grid Management Utility
  Grids
• Grids allow users to dynamically lease Grid
  services at runtime based on their quality, cost,
  availability, and users QoS requirements.
• Delivering ICT services as computing utilities.
• Grids offer enormous opportunities for realizing
  e-Science and e-Business at global level.

62
Thanks for your attention!
We Welcome Cooperation in Research and
Development! http/www.gridbus.org
63
Gridbus MiddlewareBuilding Autonomic and
Market-Oriented Global Grids for Delivering IT
Services as the 5th Utility

• Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS)
LaboratoryDept. of Computer Science and Software
EngineeringThe University of Melbourne,
Australiawww.gridbus.org
Gridbus Sponsors
64
Backup Slides
65
Some Characteristics of Grids
Numerousresources
Owned by multiple organizations individuals
Connected by heterogeneous, multi-level networks
Different security requirements policies
Different resource management policies
Geographically distributed
Unreliable resources and environments
Resources are heterogeneous
Slide by Hiro
66
What are Grid benefits?

• Resource sharing across multiple administrative
  boundaries
• Effective utilisation of the (existing) resources
• Dynamic provisioning
• Application Acceleration
• Scalability
• Reliability
• Virtualisation
• applications, services, resources,

67
Time Minimization in Data Grids
68
Results Cost Minimization in Data Grids
69
Utility GridsAutonomic and Market-Oriented
Global Grids for Powering e-Science and
e-Business Applications

• Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS)
LaboratoryDept. of Computer Science and Software
EngineeringThe University of Melbourne,
Australiawww.gridbus.org
Gridbus Sponsors