A New Era in Computing Moving Enterprise Services onto Grid

1
A New Era in Computing Moving Enterprise
Services onto Grid

• Ian Foster
• Argonne National Laboratory
• University of Chicago
• Univa Corporation

2
Abstract

• The Grid seems to be everywhere, with
  announcements appearing almost every day of Grid
  products, sales, and deployments from major
  vendors. However, in spite of the popularity of
  the term, there is often confusion as to what the
  Grid is and what problems it solves. Is there any
  "there there" or is it all just marketing hype?
• In this talk, I will address these questions,
  describing what the Grid is, what problems it
  solves, and what technology has been developed to
  build Grid infrastructure and create Grid
  applications. I will review the current status of
  Grid infrastructure and deployment and give
  examples of where Grid technology is being used
  not only to perform current tasks better, but to
  provide fundamentally new capabilities that are
  not possible otherwise.

3
The Future of Enterprise IT?
Virtualization
Service-oriented Architecture
Autonomic Computing
Grid
Adaptable Computing
Cluster Computing
On Demand Computing
Agile IT
Utility Computing
.Net
Software as Service
4
(No Transcript)
5
The Enterprise IT Environment
(1) The Good Old Days

• gt95 utilization with mixed application
  workload
• High reliability
• Uniform environment

6
The Enterprise IT Environment(2) Attack of the
Killer Micros
And the Internet and high-speed networks
7
The Enterprise IT Environment
(3) The Emergence of Silos

• Overprovisioning x10 or more
• High management costs
• Inflexible environment

8
The Enterprise IT Environment
(4) IT as Service
Define compose services ? Service-oriented
applications Allocate resources to services ?
Service-oriented infrastructure
9
The Enterprise IT Environment
(4) IT as Service
Users

• Service-oriented applications
• Wrap applications as services
• Compose applicationsinto workflows

Composition
Workflows
Invocation
ApplnService
ApplnService

• Service-oriented Gridinfrastructure
• Provision physicalresources to support
  application workloads

10
SAP Demonstration _at_ TechEdVariable Demand on
Applications

• Three applications
• IPC Internet Pricing Configurator
• CRM Workforce Management
• SCM Advanced Planner Optimizer

Scenario I Too few IPC serversResult Bad
user experience
Scenario II Too many IPC serversResult
Unnecessarily high TCO
11
SAP Demonstration _at_ TechEd Grid-enabling Apps
using Globus
Provision New Worker Process
Grid-enabled scenarioIPC server applications
started and decommissioned by a re-factored IPC
dispatcher Globus
Result Always the right number of IPC server
applications
12
Realizing IT as ServiceThe Crucial Role of Grid
Infrastructure
Application
IT Facility
IT Facility

• IT Resources
• Computing
• Storage
• Communications

13
The Future of Enterprise IT?
Virtualization
Service-oriented Architecture
Autonomic Computing
Grid
Adaptable Computing
Cluster Computing
On Demand Computing
Agile IT
Utility Computing
.Net
Software as Service
14
IT as Service
Grid Infrastructure
15
Grid InfrastructureProblems Opportunities

• Overprovisioning
• ? Reduce IT capital expenses 5-10x
• Diverse, independently managed silos
• ? Reduce IT operational expenses
• Application platform limitations
• ? Optimize performance with QoS
• Inflexible IT infrastructure
• ? Improved responsiveness to business needs
• Vendor lock-in
• ? Open source, standards, commodity IT

16
Value Proposition Analysis
Industry Characteristic Core Value Drivers, Needs
Financial Services High percentage of home-grown applications Value Time-to-value for new applications Need Rapid rollout of new applications to support new business capabilities
Manufacturing, Automotive, RD Core business driven by product design and engineering Value Effective collaboration among globally distributed design teams Need Efficient access to large data sets and HPC capabilities
Energy, Pharmaceutical Exploration/research is the highest cost component Value Time-to-market for new products Need Seamless integration of custom devices delivering large volumes of data into analytical applications
17
Grid Infrastructure Specifics

• Provide
• Resource modeling
• Monitoring and notification
• Reservation
• Provisioning, lifecycle, decommissioning
• Accounting auditing
• For
• Computers, storage, networks, software,

18
Open Standards Open Source
Internet TCP/IP BSD Unix
Web HTTP Apache
Grid OGSA Globus
19
Grid InfrastructureOpen Standards
Applications of the framework(Compute, network,
storage provisioning,job reservation
submission, data management,application service
QoS, )
WS-Agreement(Agreement negotiation)
WS Distributed Management(Lifecycle, monitoring,
)
WS-Resource Framework WS-Notification(Resource
identity, lifetime, inspection, subscription, )
Web services(WSDL, SOAP, WS-Security,
WS-ReliableMessaging, )
20
Web Servicesand Stateful Resources

• State appears in almost all applications
• Data in a purchase order
• Current usage agreement for resources
• Metrics associated with work load on a server
• Web services can model, access and manage state
  in many different ways
• Ad-hoc, per-application approaches
• WS-Resource Framework (WSRF) WS-Notification
  propose a standard approach

Modeling Stateful Resources with Web Services,
Foster, Frey, Graham, et al. www.globus.org/wsrf,
2004
21
WSRF WS-Notification

• Naming and bindings (basis for virtualization)
• Every resource can be uniquely referenced, and
  has one or more associated services for
  interacting with it
• Lifecycle (basis for fault resilient state mgmt)
• Resources created by services following factory
  pattern
• Resources destroyed immediately or scheduled
• Information model (basis for monitoring,
  discovery)
• Resource properties associated with resources
• Operations for querying and setting this info
• Asynchronous notification of changes to
  properties
• Service groups (basis for registries, collective
  svcs)
• Group membership rules membership management
• Base Fault type

22
(No Transcript)
23
Core
Grid Infrastructure Open Source
Contrib/Preview
Grid Telecontrol Protocol
Globus Toolkit version 4 (GT4)
Depre-cated
Community Scheduling Framework
Delegation
Data Replication
Python WS Core
WebMDS
Data Access Integration
CommunityAuthorization
Trigger
C WS Core
Workspace Management
Web ServicesComponents
Authentication Authorization
Reliable File Transfer
Grid Resource Allocation Management
Index
Java WS Core
Pre-WS Authentication Authorization
GridFTP
Pre-WS Grid Resource Alloc. Mgmt
Pre-WSMonitoring Discovery
C Common Libraries
Non-WS Components
Replica Location
eXtensible IO (XIO)
Credential Mgmt
www.globus.org
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
24
GT4 Web Services
Custom Services
Custom WSRF Services
GT4WSRF Web Services
Registry Admin
GT4 Container(e.g., Apache Axis)
WS-A, WSRF, WS-Notification
WSDL, SOAP, WS-Security
25
Open Source With SupportUniva Globus
Application
Univa nPlus
IT Facility
IT Facility

• IT Resources
• Computing
• Storage
• Communications

26
Managing Computers Computation

• GRAM (Grid Resource Allocation Management)
  service
• Negotiate access
• Stage code
• Monitor service
• Manage service
• Collect accounting data
• Can negotiate access to clusters,creation of
  virtual machines,establishment of virtual
  networks,

Client
GRAM
27
Dynamic Provisioning ofComputational Services
Provision New Worker Process
Grid-enabled scenarioIPC server applications
started and decommissioned by a re-factored IPC
dispatcher Globus
28
Open Science Grid use over 6 months
Dynamic Provisioning ofComputational Services
Usage CPUs
29
Managing Storage Data

• Service interfaces for managing storage data
  movement
• Storage management (SRM, NeST)
• Data movement (GridFTP, RFT)
• Replica management (RLS, DRS)
• Service interfaces for accessing data in diverse
  formats
• OGSA Data Access Integration
• GridFTP data access movement

30
Reliable Wide Area Data Replication
LIGO Gravitational Wave Observatory
Birmingham
Replicating gt1 Terabyte/day to 8 sites gt30
million replicas so far MTBF 1 month
31
Managing Networks

• The network is a vital part of a virtualized
  infrastructure
• Must be able to negotiate E2E paths, e.g.
• Firewall traversal
• Security properties
• Bandwidth and related QoS parameters
• Optical paths
• Also establishment of overlay networks with
  negotiated properties
• Flat IP name space (IPv6) IPsec make all of
  this easier!

32
Managing Networks (Nortel with Globus)
A Grid Domain Virtual Organization
A
Core Network
1
Access
Access
3
RSL2
2
Network Service Overlay
RSL2
2
MDS Index
Broker/Metascheduler
Grid Overlay

1. Application requests broker/metascheduler for job
   services and resources
2. Broker/metascheduler generates RSL2 for resource
   allocation requests after consulting MDS/Index
3. xSPs co-ordinate to allocate requested resources

33
Managing SecurityDynamic Policy Overlays
Grid security must define mechanisms for bridging
identity, policy and trust across autonomous
domains with minimal a priori arrangements
34
Authentication Authorization
Users
35
Managing InstrumentationNetwork for Earthquake
Eng. Simulation
Links instruments, data, computers, people
36
IT as Service
37
The Integrating Role ofGrid Infrastructure
38
Grid Infrastructure and Enterprise IT

• Coordinate management of previously distinct
  application silos
• Reduce over-provisioning
• Improve application performance
• Improve IT responsiveness to business needs
• Enable new applications
• Avoid vendor lock-in
• Provide a basis for utility computing, software
  as service, etc.

39
For More Information

• Globus Alliance
• www.globus.org
• Global Grid Forum
• www.ggf.org
• Globus Consortium
• www.globusconsortium.org
• Univa Corporation
• www.univa.com
• Background information
• www.mcs.anl.gov/foster

2nd Edition www.mkp.com/grid2