Economy Grid: A New e-Paradigm for Grid/Internet Computing

1
Economy Grid A New e-Paradigm for Grid/Internet
Computing
Rajkumar Buyya (buyya.com) School of Computer
Science and Software Engineering Monash
University, Melbourne, Australia
Special Thanks David Abramson Jack
Dongarra Wolfgang Gentzsch Jonathan
Giddy Domenico Laforenza
http//www.gridcomputing.com
2
Presentation Online!(Updated slides)

• http//www.buyya.com/ecogrid
• http//www.gridcomputing.com

3
Agenda

• Computing Platforms Breaking Barriers
• Towards Global (Grid) Computing
• How the Grid is Different ?
• Is it Internet/Web ? Next Gen. Internet ?
• Grid Applications ?
• Grid Resource Management Issues
• Major Grid projects and Globus
• Grid Architecture for Computational Economy
  (GRACE)
• Economic Models for Resource Trading
• Nimrod/G Grid Resource Broker
• Analogy to Electric Power Grid
• Conclusions

4
Computing Power (HPC) Drivers

• Solving grand challenge applications using
  computer modeling, simulation and analysis

Life Sciences
Aerospace
Digital Biology
CAD/CAM
Military Applications
Military Applications
Military Applications
5
Computing Platforms Evolution Breaking
Administrative Barriers
?
PERFORMANCE
Administrative Barriers
Individual Group Department Campus State National
Globe Inter Planet Universe
Desktop (Single Processor?)
SMPs or SuperComputers
Local Cluster
Enterprise Cluster/Grid
Global Cluster/Grid
Inter Planet Cluster/Grid ??
6
(No Transcript)
7
(No Transcript)
8
Killer Cluster Cluster Applications

• Numerous Scientific Engineering Apps.
• Parametric Simulations
• Business Applications
• E-commerce Applications (Amazon.com, eBay.com .)
• Database Applications (Oracle on cluster)
• Decision Support Systems
• Internet Applications
• Web serving
• Infowares (yahoo.com, AOL.com)
• ASPs (application service providers)
• eChat, ePhone, eBook, eCommerce, eBank, eSociety,
  eAnything!
• Computing Portals
• Mission Critical Applications
• command control systems, banks, nuclear reactor
  control, star-war, and handling life threatening
  situations.

9
Science Portals
Pentiums Myrinet NetBSD/Linuux PM Score-D MPC
RWCP Japan http//www.rwcp.or.jp/papia/
PAPIA PC Cluster
10
Adoption of the Approach
11
Clusters of Clusters (HyperClusters)
12
Towards Grid Computing.
For illustration, placed resources arbitrarily on
the GUSTO test-bed!!
13
Global Computational Grids
(unification of geographically distributed
computational and instruments)
14
What is Grid ?

• An infrastructure that couples
• Computers (PCs, workstations, clusters,
  traditional supercomputers, and even laptops,
  notebooks, mobile computers, PDA, and so on)
• Software ? (e.g., ASPs renting expensive special
  purpose applications on demand)
• Catalogued Data/Databases (e.g., transparent
  access to human genome database)
• Special Instruments (e.g., radio
  telescope--SETI_at_Home Searching for Life in
  galaxy, Austrophysics_at_Swinburne for pulsars)
• People/collaborators (even animals who knows ?)
• and offers a simple, consistent, dependable,
  pervasive access across (local/wide-area)
  networks to present them as an unified integrated
  resource.

15
Grid at a glance
16
(No Transcript)
17
GRID APPLICATIONS(SKIP if TIME is LIMITED)
18
Grid Applications-Drivers

• Distributed HPC (Supercomputing)
• computational science.
• high-throughput computing
• large scale simulation/chip design parameter
  studies
• Remote software access / Renting Software
• application service provides (ASPs)
• Data-intensive computing
• data mining, particle physics (CERN)
• On-demand computing
• medical instrumentation network-enabled solvers
• Collaborative
• collaborative design, data exploration, education

19
Distributed Supercomputing (SF-Express/MPICH-G,
Caltech)
NCSA Origin

• SF-Express distributed interactive simulation
• 100K vehicles (2002 goal) using 13 computers,
  1386 nodes, 9 sites
• Globus mechanisms for
• Resource allocation
• Distributed startup
• I/O and configuration
• Security

Caltech Exemplar
CEWES SP
Maui SP
P. Messina et al., Caltech http//www.globus.org/a
pplications/
20
Ad Hoc Mobile Network Simulation
Ad Hoc Mobile Network Simulation (C. Koop,
Monash) Network performance under different
microware frequencies and different Weather
conditions -- Used Nimrod
21
Image-Renderinghttp//www.swin.edu.au/astronomy/p
bourke/povray/parallel/
22
Challenging Issues in Grid Technology Development
23

• Building computational grids requires

• New programming tools

• Software that can translate the requirements of
  an application into requirements for computers,
  networks, and storage

• Security mechanisms permitting resources to be
  accessed only by authorized users

• Computers and operating systems that are more
  tightly integrated with high-speed networks

• And strong Standardization-Harmonization
  EFFORTS

24
Grid Resource Management Challenging Issues

• Authentication (once)
• Specify simulation (code,
  resources, etc.)
• Discover resources
• Negotiate authorization, acceptable use, Cost,
  etc.
• Acquire resources
• Schedule Jobs
• Initiate computation
• Steer computation
• Access remote data-sets
• Collaborate on results
• Account for usage

Domain 1
Domain 2
Ack. globus..
25
(No Transcript)
26
Grid Components
Grid Apps.
Grid Tools
Grid Middleware
Grid Fabric
27
Major GRID Projects and Initiatives
28
mix-and-match
Object-oriented
Internet-WWW
Grid Computing Approaches
Web-based Technologies
Problem Solving Approach
Market / Computational Economy
Nimrod-G/CPM
29
Many GRID Projects and Initiatives

• PUBLIC FORUMS
• Computing Portals
• Grid Forum
• European Grid Forum
• IEEE TFCC!
• GRID CCGRID and more.
• Australia
• Nimrod/G
• EcoGrid and GRACE
• DISCWorld
• Europe
• UNICORE
• MOL
• METODIS
• Globe
• Poznan Metacomputing
• CERN Data Grid
• MetaMPI
• DAS

• Public Grid Initiatives
• Distributed.net
• SETI_at_Home
• Compute Power Market Grid
• USA
• Globus
• Legion
• Javelin
• AppLeS
• NASA IPG
• Condor
• Harness
• NetSolve
• NCSA Workbench
• AccessGrid
• GrADS
• and many more...
• Japan
• Ninf

http//www.gridcomputing.com/
30
Many GRID Testbeds...
GUSTO
Distributed ASCI Supercomputer
NASA IPG
31
(No Transcript)
32
Globus Architecture and (3rd party) Tools
Grid Apps.
Applications
High-level Services and Tools
GlobusView
Testbed Status
Grid Tools
Nimrod/G
DUROC
globusrun
MPI
MPI-IO
CC
Grid Middleware
Core Services
Heartbeat Monitor
GRACE
GRAM
Nexus
Globus Security Interface
GARA
Gloperf
GASS
MDS
Grid Fabric
Local Services
eCash
Condor
MPI
TCP
UDP
QBank
LSF
NQE
Easy
Solaris
Irix
AIX
Source Globus
33
(No Transcript)
34
(No Transcript)
35
Resource Management Architecture
Resource Brokers
(RSL Specialization)
RSL
Application
36
Building ofa brokerage system..
Who pays for that ???
Foundation for the Grid Economy
37
Who pays for this ??
38
Who pays for all this ?Any Incentive for GRID
resource owners ?
GUSTO
Distributed ASCI Supercomputer
NASA IPG
39
Economy Grid GRACE

• Grid Architecture for Computational Economy

• GRACE aims help Nimrod/G overcome the current
  limitations.
• GRACE middleware offer generic interfaces (APIs)
  that other developers of grid tools can use along
  with Globus services.

40
Why Computational Economy in Resource Management ?

• Observe Grid characteristics and current
  resource management policies
• Grid resources are not owned by user or single
  organisation.
• They have their own administrative policy
• Mismatch in resource demand and supply
• overall resource demand may exceed supply.
• Markets are an effective institution in
  coordinating the activities of several entities.
• Traditional System-centric (performance matrix
  approaches does not suit in grid environment.
• System-Centric --gt User Centric
• Like in real life, economic-based approach is one
  of the best ways to regulate selection and
  scheduling on the grid as it captures user-intent.

41
Advantages of Economic-based RM

• System Centric --gt User Centric Policy in RM
• Helps in regulating demand and supply
• resource access cost can fluctuate (based on
  demand and supply and system can adapt)
• Scalable Solution
• No need of central coordinator (during
  negotiation)
• Resources(sellers) and also Users(buyers) can
  make their own decisions and try to maximize
  utility and profit.
• Uniform Treatment of all Resources
• Everything can can be traded including CPU, Mem,
  Net, Storage/Disk, other devices/instruments
• Efficient allocation of resources

42
Computational Market Model for Grid Resource
Management
Grid Information Server(s)
Info ?
Health Monitor

Grid Explorer

Application
Job Control Agent
Grid Node1
Schedule Advisor
Trading
Trade Server
Charging Alg.
Trade Manager
Accounting
Resource Reservation
Other services

Deployment Agent
Jobs
Resource Allocation
Grid User
Grid Resource Broker

R1
R2
Rm
Grid Resource/Control Domains
Grid Middleware
43
Grid Open Trading Protocols
Trade Manager
Trade Server
Get Connected
Pricing Rules
Reply to Bid (DT)
Negotiate Deal(DT)
.
API
Confirm Deal(DT, Y/N)
DT - Deal Template - resource requirements (BM)
- resource profile (BS) - price (any one can
set) - status - change the above values
- negotiation can continue -
accept/decline - validity period
Cancel Deal(DT)
Change Deal(DT)
Get Disconnected
44
Open Trading Finite State Machine
DT
lt TM, Request for Resource gt
lt TM, Ask Price gt ltlt TS, Update gtgt
ltlt TM, Update gtgt
DT
ltTS, Bid gt
lt TM, Final Offer gt
lt TS, Final Offer gt
Offer TS
Offer TM
ltTM, Rej.gt
lt TS, Reject gt
lt TM, Accept gt
DT - Deal Template TM - Trade Manager TM -
Trade Server DA - Deal Accepted DN - Deal Not
accepted
DN
DA
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
Nimrod/G Resource Broker

• Nimrod/G Approach to Resource Management and
  Scheduling

58
What is Nimrod/G ?

• A global scheduler for managing and steering task
  farming (parametric simulation) applications on
  computational grid based on deadline and
  computational economy.
• Key Features
• A single window to manage control experiment
• Resource Discovery
• Trade for Resources
• Scheduling
• Steering data management
• It allows to study the behaviour of some of the
  output variables against a range of different
  input scenarios.

59
Nimrod/G Grid Resource Broker Architecture
Nimrod/G Client
Nimrod/G Client
Nimrod/G Client
Nimrod/G Engine
Schedule Advisor
Trading Manager
Grid Store
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS
Globus,Legion, Condor-g,, Ninf,etc.
GE GIS
Grid Information Server(s)
RM TS
RM TS
RM TS
L
N
G
G
Ninf enabled node.
Globus enabled node.
L
C
RM Local Resource Manager, TS Trade Server
Condor enabled node.
60
A Nimrod/G Client
Deadline
Legion hosts
Globus Hosts
Bezek is in both Globus and Legion Domains
61
Change deadline/budget Monitor activities
62
Active Sheets - MS Excel on the Grid!
NimCache
Nimrod/G
63
Nimrod/G Interactions
Resource location
Grid Info servers
Scheduler
Trade Server
Resource allocation (local)
Prmtc.. Engine
Queuing System
Job Wrapper
Dispatcher
Process server
Root node
Gatekeeper node
Computational node
64
Adaptive Scheduling algorithms ...
65
Resource Usage (for various deadlines)
66
(No Transcript)
67
(No Transcript)
68
Conclude with a comparison with the Electrical
Grid..

• Where we are ????

69
Alessandro Volta in Paris in 1801 inside France
National Institute shows the battery at the
presence of Napoleon I

• Fresco by N. Cianfanelli (1841)
• (Zoological Section "La Specula" of National
  History Museum of Florence University)

70
Oh, mon Dieu !
.and in the future, I imagine a worldwide Power
(Electrical) Grid ...
What ?!?! This is a mad man
71
2000 - 1801 199 Years
72
Grid Computing A New Wave ?
What will be the dominant grid approach in the
next future ??
73
Trends
It is very difficult to predict the future and
this is particular true in a field such as
Information Technology
I think there is a world market for about five
computers. Thomas J. Watson Sr., IBM Founder,
1943
74
Trends
GRID
The time is exciting but the way is hard and
long.
75
Conclusions

• The Emergence of Internet as a Powerful
  connectivity media is bridging the gap between a
  number of technologies leading to what is known
  as Everything on IP.
• Cluster-based systems have become a platform of
  choice for mainstream computing.
• A number of GRID project world-wide have been
  presented to explore computing trend!
• Economic based approach to resource management is
  the way to go in the grid environment.
• Both sequential and parallel applications run
  seamless on desktops, SMPs, Clusters, and the
  Grid without any change.
• Grid A Next Generation Internet ?

76
Further Information

• Cluster Computing Infoware
• http//www.buyya.com/cluster/
• Grid Computing Infoware
• http//www.gridcomputing.com
• IEEE DS Online - Grid Computing area
• http//computer.org/channels/ds/gc
• Millennium Compute Power Grid/Market Project
• http//www.ComputePower.com
• Books
• High Performance Cluster Computing, V1, V2,
  R.Buyya (Ed), Prentice Hall, 1999.
• The GRID, I. Foster and C. Kesselman (Eds),
  Morgan-Kaufmann, 1999.
• IEEE Task Force on Cluster Computing
• http//www.ieeetfcc.org
• GRID Forums
• http//www.gridforum.org http//www.egrid.org
• CCGRID 2001, www.ccgrid.org
• GRID Meeting - http//www.gridcomputing.org

77
(No Transcript)