PRAGMA Grid Research Projects in Universiti Sains Malaysia

1
PRAGMA Grid Research Projects in Universiti
Sains Malaysia

• Compute Power Market / P2P
• e-Science Grid

Presented by Chan Huah Yong Gian Chand Sodhy
Grid Computing Research Lab Universiti Sains
Malaysia
2
PRAGMA Grid Research Projects in Universiti
Sains Malaysia
Compute Power Market / P2P
3
Compute Power Market/Peer to Peer

• Compute Power Market (CPM)
• computational market in peer-to-peer computing
  style,
• introduces resource trading on idle computational
  power across the Internet.

4
CPM/P2P Framework
Layer 4 Applications
Portals
Applications
Scheduler
Layer 3 CPM
Utilities
CPM Agent
Market Server
Trader
Dispatcher
Accounting
Job management
Layer 2 P2P Middleware
JXTA Protocols (Peer Groups, Peer Pipes,
Monitoring etc)
JXTA Security
Layer 1 P2P Fabric
Internet
USM
Melbourne
5
CPM Design Architecture

• Three basic entities
• resource provider
• resource consumer
• market.
• These entities are communicating with each other
  through a set of utilities and services that will
  cater for resource searching, pricing
  negotiation, resource allocation, job deployment,
  transaction accounting and so on.

6
CPM Design Architecture

• Two key players driving the Compute Power Market
• Compute Power Market Agent
• Market Server
• Like the conventional marketplace, CPM Agent
  works on behalf of both resource owner and the
  resource consumer, while the market server
  provides the necessary infrastructure including
  communication, information, security, transport
  access to remote resources, market repository and
  information services that able to bring these 2
  parties together for resources trading.

7
Entities Interaction
Market
Market Server
1) Publish Resource Information
2) Discover Resource Information
CPM Agent
CPM Agent
3) Contract Negotiation 4) Job execution 5)
Accounting
Consumer
Provider
8
CPM Market Server

• Consists of two major components
• Membership Control
• maintains the access authentication of the market
  by applying membership among the provider and
  consumer.
• Resource Discovery component
• manipulates the resource publishing and discovery
  process.
• The resource publishing service allows the
  resource provider to advertise their resource and
  the resource discovery service provides consumer
  the facility to discover resource available in
  the market.

9
CPM Agent

• Application that comprises the fundamental
  services of CPM, includes
• Scheduler schedule and dispatch job
• Trader - handles the trading and negotiation
  between the provider and consumer.
• Job management - handles the job deployment and
  execution on the provider computer device and
  returns the result back to the consumer.
• Accounting - handles the fees and pays for the
  trading using a resource meter   to keep track of
  the resources that have been consumed.

10
CPM Agent (cont)

• Users interact with their CPM Agent for
• searching (search ideal resource provider base on
  consumer preferences),
• managing (negotiating with CPM Agent from the
  provider for a particular resources trading) and
• running (executing and scheduling job) resources
  in the CPM.
• CPM Agent will represent resource owners and do
  the arrangement and management of the resources
  and play an important role in trading
  negotiation. When the job finished, CPM Agent
  completes the accounting process and bill the
  consumer for the charge of resource usage.

11
Work Completed

• CPM/P2P Version 1.0
• Trader
• Job Management
• Accounting
• Generic Components
• Basic components
• Already uploaded to JXTA CVS on September 2002.

12
Complete System Flow for CPM Agent on Provider
Trader
provider
Generic Components using JXTA
Accounting
Job management
13
Complete System Flow for CPM Agent on Consumer
Trader
consumer
Generic Components using JXTA
Reject
Accept
Accounting
Job management
14
Work In Progress

• Scheduler
• Market Repository
• Provide advertisement manipulation on Market
  repository
• Multi Rendezvous Model

15
Scheduler
Accounting
Submit job
Receive result
User
Trader
Job Management
Master
Worker
16
Scheduler

• The scheduler has a Hierarchical Master-Worker
  architecture.
• Master (metascheduler) schedules job based on
  global scheduling policies.
• Worker (local scheduler) schedules and dispatches
  job based on local system policies.
• Job scheduling is based on dateline and budget
  constraint.

17
Market Repository
Market Server

• Discovery
• - Membership

Market Repository
User

• Market repository.
• Searching algorithm (advertisement)
• Index manipulation (add new, update, delete)
• Market-to-market Interaction

Resources
18
Updating Market Repository

• Once peer leave the network
• Update Repository indicate peer has left (delete
  the service offered by that peer)
• If there are new services to be shared by
  existing peer
• Update the old information (services) in
  Repository

19
Future Work

• Trading Policy
• The resource usage policy (such as resource
  reservation, resource availability)
• Security
• Handle the security issues in CPM/P2P, include
  secure communications, intrusion detection.
• Multi Market Model
• Adopt various market model (auction, bargain)
  into CPM/P2P trader. Currently, only Commodity
  (Flat Price) market model is being adopted.

20
CPM Project Schedule
21
PRAGMA Grid Research Projects in Universiti
Sains Malaysia
e-Science Grid
22
What is an e-Science grid Portal?

• A meeting point for
• scientific communities to run
• scientific applications through
• distributed global collaborations
• enabled by the Internet and
• grid engine,
• using distributed resources and
• high performance platform.

23
e-Science Grid The Complete Framework
Layer 4 Portal Application
Iterative Solver
Parallel Molecular Modeling
Layer 3 Application Toolkit
Data Management
Security
Layer 2 Core Grid Engine
Globus
Layer 1 Infrastructure
Short-Area- Network based PC Cluster
Existing Server/ Super Computer
24
Conceptual Model of e-Science grid

• Service-oriented grid
• Provider organization provides services to the
  grid
• Economic-based grid
• Based on the compute power market
• Contract agreement on services usage between
  provider and consumer
• Marketplace intermediate/ communication medium
  for both parties (organizations)

25
Difference between P2P and grid
E-Science GRID
CPM/p2p
O
Organization, etc USM, UKM, UTM
G
Group (group by Individuals in an organization)
Individual user, etc grid007.cs.usm.my
26
Visit us grid.cs.usm.my/portal/index.jsp
27
Portal Framework
GSI
HTTPS
HTTPS, Globus Services
LDAP
Tomcat, Java, COG kit, GPDK
28
E-Science Grid Architecture
User
Mobile Agent
Resources
Account Manager
Resource Usage Tracking Agent
29
GRID RESOURCE ALLOCATION Overall Framework
RESOURCE ALLOCATION
Resource Broker
Available resources?
Grid Scheduler
Cost? Bill?
Resource Usage?
Trader
Metering
Replica Management
Data? (by agent)
Select resources Sends agents
RESOURCES ON GRID
30
Framework (newly proposed)
grid.cs.usm.my - Resource Broker - GRAM -
DUROC
Job Submission request via http
GRAM Gatekeeper Job manager Condor
GRAM Gatekeeper Job manager Condor
Condor pool
PBS pool
Standalone PCs
GRAM Gatekeeper Job manager fork
31
Resource Broker Framework
Resource broker
User request to RSL mapper
Job Description in RSL
Resource request
Resource Monitoring
Matchmaker
Available resources
Job resource description RSL
Resource Reservation
Dispatch job
Dispatch job
Dispatch job
Condor pool
Individual resources
PBS pool
32
Metering Design
Budget/Deadline
Cost?
Charging
Trading
Accept?
No
Yes
Credit gt Budget?
No
Yes
Job Scheduling
Message top-up
Done?
No
Usage Tracking
Message failure
Yes
Accounting
Billing
33
REPLICA MANAGEMENT Overall Framework
Information Services
(4) List of available resources
(5) Conf. of each resources
Replica Catalog
(2) List of proteins
Replica Selection
(1) List of proteins to dock
(3) List of available resources
to a ligand
Not replicated protein files
Selected replicas
Compute Resources
Selected replicas
Set of replicated PDB
Grid Environments
34
Resource Monitoring
User
Access
Visualization
Metering (Project 2)
e-Sciences Grid Portal (Dynamic Information
Services)
Grid Scheduler (Project 2)
Other Applications (Project 5)
Processed Data
Authorized
Platform Type, Operating System, CPU, Memory,
Network, File System, Job Status
Prediction
Resource Monitor
Event Publication Information
Dispatch Agent
Directory Service
Mobile Agents Facility
Invitation/Correction
Event Publication Information
Resources
35
Focused Area Resource Monitoring
Status Monitoring
Event Archive
Resource Monitor
Job Monitoring
Mobile Agents Facility
Reverse Registering
Directory Service
Resource Discovery
Fault Correction
Resources
36
Prediction Module
Network Weather Service
Network Monitoring
Name Server
network measurement
Sensor
Persistent State
Forecaster
e-Science Grid
forecasts
forecasts
E-Science Portal
Client
37
Milestones

• Phase I (June - August 2002) Final Model and
  Design
• Phase II (Sep 2002 - May 2003) Major
  Implementation Integration

38
Manpower

• 4 lecturers
• 9 research officers
• 5 MSc. students
• 5 BSc. students

39
Resources (Machines)

• 16-node clusters x 2
• Pentium-4 dual-processor servers x 2
• Pentium-4 high-end workstations x 8
• Pentium-4 PCs x 29
• Pentium-3 PCs x 3

40
Visit us at http//grid.cs.usm.my
41
Thank You