Overview of Grid Technology

1
Overview of Grid Technology

• Tony Hey and Geoffrey Fox

Originally given at CERN March 30-April 4
2003 Lectures and Streaming Video version
available at http//academia.web.cern.ch/academia/
lectures/grid/ Given July 2003 by Geoffrey Fox
2
Part 2

• Overview of Web service and Grid Technology

3
What is a Grid I?

• Collaborative Environment (Ch2.2,18)
• Combining powerful resources, federated computing
  and a security structure (Ch38.2)
• Coordinated resource sharing and problem solving
  in dynamic multi-institutional virtual
  organizations (Ch6)
• Data Grids as Managed Distributed Systems for
  Global Virtual Organizations (Ch39)
• Distributed Computing or distributed systems
  (Ch2.2,10)
• Enabling Scalable Virtual Organizations (Ch6)
• Enabling use of enterprise-wide systems, and
  someday nationwide systems, that consist of
  workstations, vector supercomputers, and parallel
  supercomputers connected by local and wide area
  networks. Users will be presented the illusion of
  a single, very powerful computer, rather than a
  collection of disparate machines. The system will
  schedule application components on processors,
  manage data transfer, and provide communication
  and synchronization in such a manner as to
  dramatically improve application performance.
  Further, boundaries between computers will be
  invisible, as will the location of data and the
  failure of processors. (Ch10)

4
What is a Grid II?

• Supporting e-Science representing increasing
  global collaborations of people and of shared
  resources that will be needed to solve the new
  problems of Science and Engineering (Ch36)
• As infrastructure that will provide us with the
  ability to dynamically link together resources as
  an ensemble to support the execution of
  large-scale, resource-intensive, and distributed
  applications. (Ch1)
• Makes high-performance computers superfluous
  (Ch6)
• Metasystems or metacomputing systems (Ch10,37)
• Middleware as the services needed to support a
  common set of applications in a distributed
  network environment (Ch6)
• Next Generation Internet (Ch6)
• Peer-to-peer Network (Ch10, 18)
• Realizing thirty year dream of science fiction
  writers that have spun yarns featuring worldwide
  networks of interconnected computers that behave
  as a single entity. (Ch10)

5
What is Grid Technology?

• Grids support distributed collaboratories or
  virtual organizations integrating concepts from
• The Web
• Distributed Objects (CORBA Java/Jini COM)
• Globus Legion Condor NetSolve Ninf and other High
  Performance Computing activities
• Peer-to-peer Networks
• With perhaps the Web being the most important for
  Information Grids and Globus for Compute
  Grids
• Use Information Grids and not usual Data Grids as
  distributed file systems (holding lots of
  data!) are handled in Compute Grids

6
Grid Politics

• There is a Global Grid Forum meeting 3 times per
  year with about 700 attendees per meeting
• Exchange information and define standards for
  everything not done in W3C and OASIS
• e.g. Grid Service, Security, What is a Job,
  Database, Computer, How to build portals .
• There is a large project called Globus developing
  software largely for compute/file Grids
• There are some 50 Grid projects (mainly in Europe
  and USA) developing software and applications as
  well as installing infrastructure
• Some are deployment EDG NMI VDT ..
• There are related initiatives called
  CyberInfrastructure (NSF USA) and e-Science (UK)
• There is a proposed OMII (Open Middleware
  Infrastructure Institute) an international
  Alliance of separately funded projects with
  common coordination

7
SERVOGrid Caricature
Solid Earth Research Virtual Observatory
8
SERVOGrid Grid Requirements

• Seamless Access to Data repositories and large
  scale computers
• Integration of multiple data sources including
  sensors, databases, file systems with analysis
  system
• Including filtered OGSA-DAI
• Rich meta-data generation and access with
  SERVOGrid specific Schema extending industry
  standards
• Portals with component model for user interfaces
  and web control of all capabilities
• Collaboration to support world-wide work
• Basic Grid tools workflow and notification

9
OGSA/OGSI Top Level View
http//www.gridforum.org/Meetings/ggf7/docs/defaul
t.htm http//www.globusworld.org/globusworld_web/
jw2_program_tut.htm

• OGSA is the set of core Grid services
• Stuff you cant live without
• If you built a Grid you would need to invent
  these things

10
Taxonomy of Grid Functionalities
Note Term Data Grid not used consistently in
community so avoided
11
Taxonomy of Grid Operational Style
12
Underlying Grid Principles

• We will start from the Web view and assert that
  basic paradigm is
• Meta-data rich Web Services communicating via
  messages
• These have some basic support from some runtime
  such as .NET, Jini (pure Java), Apache
  TomcatAxis (Web Service toolkit), Enterprise
  JavaBeans, WebSphere (IBM) or GT3 (Globus Toolkit
  3)
• These are the distributed equivalent of operating
  system functions as in UNIX Shell
• Called Hosting Environment or platform

13
Some Basic Observations

• Grids manage and share asynchronous resources in
  a rather centralized fashion
• Peer-to-peer networks are just like Grids with
  different implementations of services like
  registration and look-up
• Web Services interact with messages
• Everything (including applications like
  PowerPoint will be a WS?) see later short
  discussion
• Computers are fast and getting faster. One can
  afford many strategies that used to be
  unrealistic
• All messages can be publish/subscribe
• Software message routing
• XML will be used for most interesting data and
  meta-data
• One will store/consider data and meta-data
  separately but often use same technology to
  manage both of them.
• Need Synchronous and Asynchronous Resource
  Sharing
• Integrate Grid and Collaboration technology

14
Classic Grid Architecture
Resources
Content Access
Composition
Middle TierBrokers Service Providers
Netsolve
Security
Collaboration
Computing
Middle Tier becomes Web Services
Clients
Users and Devices
15
What is a Web Service I

• A web service is a computer program running on
  either the local or remote machine with a set of
  well defined interfaces (ports) specified in XML
  (WSDL)
• In principle, computer program can be in any
  language (Fortran .. Java .. Perl .. Python) and
  the interfaces can be implemented in any way what
  so ever
• Interfaces can be method calls, Java RMI
  Messages, CGI Web invocations, totally compiled
  away (inlining) but
• The simplest implementations involve XML messages
  (SOAP) and programs written in net friendly
  languages like Java and Python
• Web Services separate the meaning of a port
  (message) interface from its implementation
• Enhances/Enables Re-usable component model of ANY
  electronic resource

16
Raw Data
RawResources
Raw Data
(Virtual) XML Data Interface
WS
WS
etc.
XML WS to WS Interfaces
(Virtual) XML Knowledge (User) Interface
Render to XML Display Format
(Virtual) XML Rendering Interface
Clients
17
What is a Web Service II

• Web Services have important implication that ALL
  interfaces are XML messages based. In contrast
• Most Windows programs have interfaces defined as
  interrupts due to user inputs
• Most software have interfaces defined as methods
  which might be implemented as a message but this
  is often NOT explicit

18
What is a Web Service III

• Everything electronic is a resource
• Computers Programs People
• Data (from sensors to this presentation to email
  to databases)
• Everything electronic is a distributed object
• All resources have interfaces which are defined
  in XML for both properties (data-structure) and
  methods (service, function, subroutine)
  (Resources are Services)
• We can assume that a data-structure property has
  getproperty() and setproperty(value) methods to
  act as interface
• All resources are linked by messages with
  structure, which must be specifiable in XML
• All resources have a URI such as unique//a/b/c
  .

19
WSDL Abstractions

• WSDL abstracts a program as an entity that does
  something given one or more inputs with its
  results defined by streams on one or more
  outputs.
• Functions are defined by method name and
  parametersmethodname(parm1,parm2, parmN)
• Where parameters are Input Output or both
• In WSDL, we will have a Web Service which like a
  (Java or CORBA Program) can be thought of as a
  (distributed) object with many methods
• Instead of a function call, the calling routine
  sends an XML message to the Web Service
  specifying methodname and values of the
  parameters
• Note name of function is just another parameter

20
Details of WSDL Protocol Stack

• UDDI finds where programs are
• remote( (distributed) programs are just Web
  Services
• (not a great success)
• WSFL links programs together(under revision as
  BPEL4WS)
• WSDL defines interface (methods, parameters, data
  formats)
• SOAP defines structure of message including
  serialization of information
• HTTP is negotiation/transport protocol
• TCP/IP is layers 3-4 of OSI
• Physical Network is layer 1 of OSI

21
Education as a Web Service

• Can link to Science as a Web Service and
  substitute educational modules
• Learning Object XML standards already exist
  from IMS/ADL http//www.adlnet.org need to
  update architecture
• Web Services for virtual university include
• Registration
• Performance (grading)
• Authoring of Curriculum
• Online laboratories for real and virtual
  instruments
• Homework submission
• Quizzes of various types (multiple choice, random
  parameters)
• Assessment data access and analysis
• Synchronous Delivery of Curricula
• Scheduling of courses and mentoring sessions
• Asynchronous access, data-mining and knowledge
  discovery
• Learning Plan agents to guide students and
  teachers

22
What are System and Application Services?

• There are generic Grid system services security,
  collaboration, persistent storage, notification,
  workflow
• OGSA (Open Grid Service Architecture) is
  implementing these as extended Web Services
• An Application Web Service is a capability used
  either by another service or by a user
• It has input and output ports data is from
  sensors or other services
• Consider Satellite-based Sensor Operations as a
  Web Service
• Satellite management (with a web front end)
• Each tracking station is a service
• Image Processing is a pipeline of filters which
  can be grouped into different services
• Data storage is an important system service
• Big services built hierarchically from basic
  services
• Portals are the user (web browser) interfaces to
  Web services

23
Application Web Services

• Note Service model integrates sensors, sensor
  analysis, simulations and people
• An Application Web Service is a capability used
  either by another service or by a user
• It has input and output ports data is from
  users, sensors or other services
• Big services built hierarchically from basic
  services

24
The Application Service Model

• As bandwidth of communication (between) services
  increases one can support smaller services
• A service is a component and is a replacement
  for a library in case where performance allows
• Services (components) are a sustainable model of
  software development each service has
  documented capability with standards compliant
  interfaces
• XML defines interfaces at several levels
• WSDL at Service interface level and XSIL or
  equivalent for scientific data format
• A service can be written as Perl, Python, Java
  Servlet, Enterprise JavaBean, CORBA (C or
  Fortran) Object
• Communication protocol can be RMI (Java), IIOP
  (CORBA) or SOAP (HTTP, XML)

25
Application with W3C DOM Structure as a Web
Service
Data
Resource Facing Ports
Application as a Web service Application Model
Remaining W3C DOM Semantic Events
MVCM Model
Control
User FacingPorts
View
CControl
Events as Messages
Rendering as Messages
Application Viewand SelectedControl
V View
26
7 Primitives in WSDL

• types which provides data type definitions used
  to describe the messages exchanged.
• message which represents an abstract definition
  of the data being transmitted. A message consists
  of logical parts, each of which is associated
  with a definition within some type system.
• operation an abstract description of an action
  supported by the service.
• portType which is a set of abstract operations.
  Each operation refers to an input message and
  output messages.
• binding which specifies concrete protocol and
  data format specifications for the operations and
  messages defined by a particular portType.
• port which specifies an address for a binding,
  thus defining a single communication endpoint.
• service which is used to aggregate a set of
  related ports

27
lt?xml version"1.0" encoding"UTF-8"?gt ltwsdldefin
itionsgt ltwsdlmessage name"execLocalCommandRes
ponse"gt ltwsdlmessage name"execLocalCommandReques
t"gt ltwsdlportType name"SJwsImp"gt ltwsdloperation
name"execLocalCommand" parameterOrder"in0"gt
ltwsdlinput message"implexecLocalCommandReque
st" name"execLocalCommandRequest"/gt
ltwsdloutput message"implexecLocalCommandRespons
e" name"execLocalCommandResponse"/gt
lt/wsdloperationgt lt/wsdlportTypegt ltwsdlbinding
name"SubmitjobSoapBinding" type"implSJwsImp"gt
ltwsdlsoapbinding style"rpc"
transport"http//schemas.xmlsoap.org/soap/http"/gt
ltwsdloperation name"execLocalCommand"gt
ltwsdlsoapoperation soapAction""/gt
ltwsdlinput name"execLocalCommandRequest"gt
ltwsdloutput name"execLocalCommandResponse"gt lt/ws
dloperationgt lt/wsdlbindinggt ltwsdlservice
name"SJwsImpService"gt ltwsdlport
binding"implSubmitjobSoapBinding"
name"Submitjob"gt lt/wsdlservicegt lt/wsdldefinit
ionsgt
28
Discussion of 7 WSDL Primitives

• types specify data-structures which are
  equivalent to arguments of methods
• message specifies collections of types and is
  equivalent to set of arguments in a method call.
  Note that it is an abstract method in Java
  terminology
• operation is a a collection of input output and
  fault messages there are 4 types of operation
  one-way(service just receives a message),
  request-response(RPC), solicit-response,
  notification (services pushes out a message)
• portType represents a single channel that can
  support multiple operations. It is abstract as
  specified as a set of operations. It is
  equivalent to a interface or abstract class in
  Java
• binding tells you transport and message format
  for a porttype (which can have multiple bindings
  to reflect say performance-portability trades)
• port combines a binding and an endpoint network
  address (URL) and is like a class instance
• service consists of multiple ports and is
  equivalent to a program in Java