Grid Computing Tutorial

1
Grid Computing Tutorial

• Ed Green
• Lecturer IST
• The Pennsylvania State University
• The Abington College
• 215-881-7332
• exg13_at_psu.edu
• www.personal.psu.edu/exg13

2
Grid Technology Abstract

• Emerging new field
• Beyond distributed computing
• Focus
• Large-scale resource sharing applications
• High-performance orientation
• Requires
• Flexible, secure, coordinated resource sharing
• Involves dynamic collection of
• Individuals
• Institutions
• Resources
• Characterized by unique authentication,
  authorization, resource access, and resource
  discovery

The virtual organization
3
Introduction and Background

• Term grid originated mid 1990s
• Proposed infrastructure for science and
  engineering
• Expanded to include broadest technology spectrum
• From advanced networking
• To artificial intelligence
• And everything in between
• Addresses real and specific problem space
• Distinct and separate from popular technology
  trends
• Internet
• Enterprise computing
• Distributed computing
• Peer-to-peer computing
• Symbiotic opportunities when popular technologies
  grow into the grid problem space

4
Grid Problem Space

• Coordinated resource sharing and problem solving
  in dynamic, multi-institutional, virtual
  organizations
• Essential needs
• Highly flexible sharing relationships ranging
• From client-server
• To peer-to-peer
• Sophisticated and precise levels of control over
  use of shared resources
• Fine-grained and multi-stakeholder
• Access control
• Delegation
• Application of local and global policies
• Sharing of resources
• From programs, files, and data
• To computers, sensors, and networks
• Diverse usage modes
• From single-user to multi-user
• From performance sensitive to cost-sensitive
• Quality of service
• Scheduling
• Co-allocation

Not addressed by current generation of
distributed computing technologies
5
What Grid Technology Offers

• Security solutions that support management
  credentials and policies across multiple
  enterprises
• Resource management services and protocols to
  support
• Secure remote access to computing and data
  resources
• Co-allocation of multiple resources
• Information query protocols and services that
  provide configuration and status information
  about
• Resources
• Organizations
• Services
• Data management services that locate and
  transport datasets between storage systems and
  applications

Compliment existing distributed computing
technologiesrather than competing with them!
6
Grid Technology Placement a Perspective

• Virtual organizations
• Set of collaborating enterprises
• Viewed as a single logical entity
• Leverage collaborator
• Processes
• Policies
• Systems
• Resources
• Single-enterprise viewpoint (s)
• Collaboration among diverse business units
• Merger/acquisition/divestiture ramifications
• Cooperative processing among less-than-compatible
  systems
• Multi-enterprise viewpoint (m)
• Collaboration among diverse enterprises
• msn where n is the number of enterprises
• Global considerations

7
Virtual Organizations

• Collaboration to achieve a common goal
• An enterprise can participate in multiple virtual
  organizations
• Domain-relevant
• Market-centric
• Industry-oriented
• Problem-centric
• Opportunity-centric
• Economics-driven
• Dynamic over time
• Resource sharing is managed
• Need to know accessibility
• Conditional availability who, what, when, how
• Discovery mechanism required to characterize the
  state of relationships at some particular point
  in time
• Peer-to-peer considerations
• Providers and consumers
• Subset relationships
• Single resource, multiple sharing opportunities

8
Concept of Grid Architecture

• Grid architectures require establishment of
  sharing relationships among potential
  participants
• Central issue ? interoperability ? protocols
• Grid architecture is a protocol architecture
• Mechanisms for users and resources to negotiate,
  establish, manage, and exploit sharing
  relationships
• Standards-based open architecture
• Facilitates extensibility, interoperability,
  portability, and code sharing
• Standard protocols enable definition of standard
  services that provide enhanced capabilities
• Application Programming Interfaces (API)
• Software Development Kits (SDK)

9
Importance of Interoperability

• Need to initiate sharing relationships among
  arbitrary partners
• Need to accommodate new partners dynamically
  across different computing environments
• Hardware
• Software
• Need to promote multilateral sharing arrangements
• Avoid bilateral resource sharing
• Ensure availability of sharing mechanisms in a
  dynamic partnership environment

10
Importance of Protocols

• Protocol definition specifies
• How distributed system elements interact with
  each other to achieve a specified behavior
• Structure of information during interaction
• Virtual organizations compliment existing
  enterprises/institutions
• Sharing mechanisms must avoid substantial changes
  to local policies
• Sharing must preserve individual (institution)
  control of (their) resources
• Protocols
• Govern the interaction between components
• Do not govern implementation of components
• Without standard protocols, interoperability
  requires
• Single implementation at the API level or
• Having every implementation know details of every
  other

11
Grid Architecture Description
Application
Application
Collective
Internet Protocol Architecture
Resource
Grid Protocol Architecture
Transport
Connectivity
Internet
Fabric
Link
A relationship exists between the Grid
Protocol Architecture and the Internet Protocol
Architecture.
12
Fabric Layer Local Control Interface

• Provides resources to mediate shared access to
  system facilities by Grid protocols
• Physical system facilities require external
  protocols computational components, storage
  systems, catalogs, network and/or sensors
• Logical system facilities require internal
  protocols distributed file system, computer
  cluster, and/or distributed computer cluster
• Implements local, resource-specific operations on
  specific (logical or physical) resources as the
  result of higher-level sharing operations
• Interdependence between fabric-layer functions
  and sharing operations
• Tightly coupled

13
Delivering Functionality
Application
Co-reservation Service API SDK
Co-reservation Protocol
Collective Layer
Co-reservation Service
Co-Allocation API SDK
Resource Mgmt API SDK
Resource Layer
Resource Management Protocol
Network Resource
Network Resource
Compute Resource
Compute Resource
Fabric Layer

• Can be combined in a variety of ways to deliver
  functionality to applications

14
Fabric Layer Local Control Interface

• Minimum implementation
• Enquiry mechanisms that permit discovery of
  resource structure, state, capabilities
• Resource management mechanism that provide
  control of delivered quality of service
• Capabilities
• Computational resources starting, monitoring,
  and controlling the execution of programs
• Storage resources getting and putting of files
• Network resources managing network transfers
• Code repositories managing versioned source and
  object code
• Catalogs implementing catalog query and update
  capabilities

15
Connectivity Communication and Authentication
Protocols

• Communication protocols enable exchange of data
  between Fabric Layer resources
• Communications includes
• Transport
• Routing
• Naming
• Authentication protocols
• Build on communications services
• Provide cryptographically secure mechanisms for
  identity verification
• Users
• Resources
• Security aspects standards based

16
Authentication Solutions for VO Environments

• Single sign on one-time authentication provides
  access to allowed Grid resources
• Delegation ability to endow a program to
  execute on the named users behalf
• Interoperate with local security solutions
• User-based trust relationships

17
Resources Layer Sharing Single Resources

• Defines protocols, APIs, and SDKs for
• Secure negotiation
• Initiation
• Monitoring
• Control
• Accounting
• Payment processing
• Call Fabric Layer functions to access and control
  local resources
• Concerned entirely with individual resources
• Primary protocols
• Information protocols obtain information about
  state and/or structure of resources
• Management protocols negotiate access to a
  shared resource

Operations
18
Collective Layer Coordinating Multiple Resources

• Protocols, APIs, and SDKs
• Not associated with any one particular resource
• Global in nature
• Capture interactions across collections of
  resources

19
Collective Layer Services

• Directory services discover existence and/or
  properties of VO resources
• Co-allocation, scheduling, and brokering services
  
• Allow VO participants to request allocation of
  one or more resources
• Allow VO participants to schedule tasks on
  appropriate resources
• Monitoring and diagnostics services monitor VO
  resources for failure, adversarial attack, or
  overload
• Data replication services support placement of
  data to maximize data access performance with
  respect to metrics such as response time,
  reliability, and cost
• Grid-enabled programming services allow use of
  familiar programming models to be used in Grid
  environments to address resource discover,
  security, and resource allocation

20
Collective Layer Services

• Workload management systems and collaboration
  frameworks
• aka problem solving environments
• Provide for description, use, and management of
  multi-step, asynchronous, multi-component
  workflows
• Software discovery services discover and select
  most appropriate implementation and execution
  platform based on parameters of problem being
  solved
• Community authorization services enforce
  community policies governing resource access, to
  generate access capabilities to community
  resources
• Community accounting and payment services
  gather resource usage information for accounting,
  payment, and/or resource usage management
• Collaboratory services supports information
  exchange among users

21
Applications

• Utilize services defined at any of the other
  layers
• Construction
• Utilization
• Implemented using SDKs
• Exchange protocol messages with appropriate
  services to perform desired actions
• Utilize
• Frameworks
• Libraries

F (well-defined protocols)
22
Applications
Key
Collective APIs SDKs
Collective Service Protocols
API/SDK
Collective Services
Resource APIs SDKs
Service
Resource Service Protocols
Resource Services
Connectivity APIs
Connectivity Protocols
Fabric
23
Bilateral Relationships
24
Multilateral Relationships
Grid
25
Grid Architecture Services
26
Problem Scenario

• Purchasing CollaborativeAs the result of a
  professional society survey initiative, a number
  of enterprises in some particular industry
  determine that they purchase similar items from a
  common set of suppliers. Upon reviewing the
  suppliers terms and conditions, a cost saving
  benefit is recognized if the several industry
  enterprises can engage in a common procurement
  activity. From the enterprise perspective, there
  would be common pricing based on shared catalogs
  price with advantages for larger orders. From
  the supplier perspective, there would be fewer
  purchase orders to handle, thereby reducing
  labor-intensive activities and associated
  operating costs. Initial discussions with the
  supplier community are encouraging the problem
  is that every participant has an individual
  information processing environment that offers
  minimal commonality.
• Could this problem be solved using traditional
  computing methods?
• What solution possibilities are offered by grid
  computing?
• Develop a schematic that illustrates a solution
  proposal.