EP102 24 x 7 x 365 x 2000 = Continuous Availability

1
EP10224 x 7 x 365 x 2000 Continuous
Availability
Dmitry Chernizer Enterprise Systems
Architect Dcherniz_at_sybase.com Ian Smart Sr.
Manager Europe Ians_at_sybase.com
2
Authors

• Dmitry Chernizer
• Ian Smart

3
Agenda
In the e-Generation.. What is Continuous
Availability Why CA is Important Goals,
Vendors, Technologies Architecture
Components Sample Scenarios Real World
Examples Sybase Technologies Summary
4
The e-Generation A new revolution?
Did the Internet REALLY Change Everything? Are
there REALLY solutions for a Small Planet? If
you know where you want to go today, WHY does it
always take at least 2 days to get there? You
should know better
5
The e-Generation Meet the new boss.. Same as
the old boss.
Transaction Speed is Still Important Data
Backup and Recovery more important then
ever Data Volumes are Growing Exponentially Ente
rprise System design grows more complex Batch
Process windows are shrinking Business focus on
Continuous, Non-stop Services
6
What is Continuous Availability?Two Steps to
Success...

• First Step Fleshing out High Availability
• A coupled server system of two or more machines
  or
• processors
• If a member of the cluster (node) experiences
  an outage
• other nodes pickup the work load.
• Sybase servers managing independent workloads
  can take
• over each other's workload in the event of a
  failure.
• Clients in transit are migrated (Failed Over)
  to another node
• A clustered or coupled system is said to be
  Highly Available

7
What is Continuous Availability?Two Steps to
Success...

• Second Step Defining Continuous Availability
• A coupled or clustered system which supports
  100 Up Time
• Highly Distributed System
• Designed with multiple levels of redundancy
• No Single Point of Failure
• No PLANNED Down Time Either

8
What is Continuous Availability?Metrics, Lies
and Statistics
Continuous Availability Class of Service
Maximum down time to be considered an HA System
DBMS Vendors typically endorse HA at this level
Early stage of Continuous Availability
Clinton stops to take notice..
Source Jim Gray and Andreas Rueter in
Transaction Processing.
9
Sybase HA/CA Quality of Service
Transparent Application Fail-over Transaction
State Management Application Data Partitioning
100 Up Time
Continuous Availability
99.999
Shared Disk/Shared Nothing Single Image
System Client Load Partitioning
High Availability
99.99 Up Time
High Availability
99.9 Up Time
Fault Tolerance
90 Up Time
Disaster Recovery
Delivering Continuous Availability to the
Enterprise
10
What Drives Demand for Continuous Availability?

• The shift to Internet Computing
• A need for Redundant/Reliable Data Services
• Fear of an E-Nightmare
• If your site is down.. Your business is down
• (ask eBay!)
• Re-emergence of Federated Data Systems
• Digital Commerce Hubs
• Electronic Commerce Networks
• Enterprise Portal Initiatives
• Corporate Standards, Federal Requirements, etc..
• Shrinking Nightly Batch Cycle windows
• Need for high system redundancy (scalability)
• The Need for an uninterrupted service (ASP)

11
Estimated cost of downtime to businessesBillions
lost every year!

• Sample Per Hour Cost
• Airline reservation system outage 89,000.
• Credit-card sales operations outage 2.65
  million.
• Financial-services company outage 6.45
  million.
• Sample Per Minute Cost
• Messaging 200
• ATP/POSFT 5,000
• Customer Service 5,000
• Personal Services 7,000
• Internet Banking 8,000
• E-Commerce 9,500
• Supply Chain 12,000
• ERP 13,000
• Call Center 26,000

12
What is the HA/CA architecture goal?Reduce
Unplanned Down Time to .001Reduce Planned Down
time to 0

• Improve Performance throughput, response time,
  query turn around time - how ever it is measured
  
• Maintain Availability with multiple nodes,
  when failure occurs in one node, other nodes can
  assume the processing tasks of the failed node
• Price/Performance commodity components which
  individually demonstrate superior
  price/performance form the building blocks of a
  multi-node system
• Allow for Incremental Growth the ability to
  add additional, usable processing nodes without
  limit is a crucial element in the differentiation
  between clusters and symmetric multiprocessors.

13
HA/CA Hardware Players
Examples include Sun - ClusterHA, Veritas,
Qualix HP - MC/Service Guard Microsoft MS
Cluster Server IBM - HA/CMP (SP2
systems) Compaq TruCluster, Veritas EMC -
Symmetrix SRDF and TimeFinder
Current Sybase specific solution available
14
What do HA Hardware Vendors Provide Today?Not
Much

• Customers often left on their own to implement
  HA/CA
• Coordination Modules
• Hardware Level restart scripts
• Heart-beat monitor scripts
• Failure Alert generators
• Hardware level Data Mirroring and Replication
• Bit-wise state change replication
• Transaction- neutral change management (no
  recovery)
• Mirrored Device Parallel Reads
• Data Farm Monitoring Utilities and Staff
• Fault monitoring software
• On-site support

15
Recent and Emerging TechnologiesWatch this
space!

• Veritas
• Cluster Server (Software Level Clustering)
• VxFS (Journaled file Systems)
• www.veritas.com
• HP Fail Safe
• State and Application migration
• www.hp.com
• EMC
• Storage Area Network (SANS) Data Replication
• Federated System Storage Arrays (Software Level
  Clustering)
• www.emc.com

16
Dividing the HA/CA Architecture Tiers What are
all the game pieces?
The Client Tier 1 An end-user graphic
rendering environment May reside anywhere
desktop, hand held May be an up-stream system (no
graphics)
Thin Client, HTML Applets, Servlets Open API
Browser or Other Client
The Presentation Layer Tier 2 Graphical
Content Generation Stand-alone or Embedded
Functionality May serve as multiple system entry
points
Stand-alone Web Servers (ie. Apache, Netscape,
IIS) Web-Application Servers (ie. iPlanet,
Silverstream) Middleware Application
Servers (ie. Sybase EAS, IBM WebSphere) Graphics
Server (ie. X-Windows Server, Citrix)
Web Server or Presentation Layer
The Application Layer Tier 3 Application
Content Management and Logic Business Rules and
Transaction Processing May serve as multiple
system entry points
Application Server
Database System (ie. Sybase, Oracle, IBM
etc..) Hybrid Data Store (ie. OLAP, ROLAP, VSAM
etc..)
The Data Storage Layer Tier 4 Structured /
Unstructured Data Management Data- Centric
Transaction Coordination Data- Centric Access
Language and Rules
A DBMS or Data Repository
To Down Stream 3rd Party Systems
17
Dividing the HA/CA Architecture Tiers How do all
the pieces plug together?
Web Servers
Objects and Components
Tools
Browsers
Databases
Application Servers
18
Sample Architectures Shared Repository Peer
Cluster

• Configuration Benefits
• Single Source Object Persistence
• Somewhat Easy to Scale
• Easy to Maintain
• Automatic Node Synchronization
• Automatic Component fail-over
• Configuration Drawbacks
• Repository a single point of failure
• Repository Contention Issues
• Limited Application Partitioning

Component Sync User Credential Sync Load Balancing
Application Server Engine
Application Server Engine
Application Server Engine
Shared Persistent Store Failover
19
Sample Architectures Slave-Master (Cascading
Directory) Shared Nothing Cluster

• Configuration Benefits
• Full Application Partitioning
• Very Easy to Scale
• Easy to Clone Data Islands
• Fully Partitioned Recovery Model
• Topology- based Load Balancing
• Configuration Drawbacks
• Requires Naming Topology Design
• Requires External Repository Sync
• No Automated Component Failover
• Less easy to maintain

Cluster Parent Name Server
Cluster Slave Name Server
Cluster Slave Name Server
May be symmetric or asymmetric across
hardware (NT, UNIX)
20
Sample Architectures Redundant (Dual Node)
Shared Repository Peer Cluster

• Configuration Benefits
• Redundancy at Every Layer
• Easier to Maintain
• Automatic Node Synchronization
• Automatic Component Fail-over
• Automatic Repository Fail-over
• Data Store less prone to failure
• Configuration Drawbacks
• Planning for Multi-level Fail-back
• Scalability limited by cluster design
• Cluster Interconnect Issues

Component Sync User Credential Sync Load Balancing
Application Server Engine
Application Server Engine
Application Server Engine
Redundant Persistent Store Failover
Database Repository
Database Repository
21
Sample Architectures Shared Nothing Peer Cluster

• Configuration Benefits
• Limited Application Partitioning
• Very Easy to Scale
• Easy to Clone Data Islands
• Partitioned Recovery Model
• Data store not point of failure
• Configuration Drawbacks
• Requires more planning
• Less easy to maintain
• No Common Object Persistence
• Requires External Repository Sync

Component Sync User Credential Sync Load Balancing
22
Sample Architectures Virtual Database and Single
Images Systems

• Configuration Benefits
• Common Access Point
• Common Access Method
• Easy to plug-in Data Islands
• Single Sign-On for Data Stores
• Data store not point of failure
• Auto-sync Logins/Permissions
• Configuration Drawbacks
• Requires much more planning
• Less easy to maintain
• Network Overhead Issues
• Distributed Optimizer Issues

Component Sync User Credential Sync Load Balancing
Application Server Engine
Database Repository
CIS Proxy Services
23
Okay, Real World Examples
Pause Exhale Yes, this stuff actually works.
24
Sample Architectures An N-Tier Architecture with
Repository Synchronization (Extranet Server Cloud)
In Theory..
Server Cloud Network
Application Server Cluster
DBMS Cluster
25
Sample Architectures An N-Tier Architecture with
Repository Synchronization (Extranet Server Cloud)
In Practice..
DRE
CMS (passive)
Spider
DRE Database
EAServer
DQH ()
OpenSwitch
OpenSwitch
EAServer
DQH ()
DRE Database
CMS
Distributed IP Mechanism
DRE
Spider
DMZ Ring
CMS (passive)
DQH ()
EAServer
OpenSwitch
ASE - Main ConnectionASE - HA Failover
ConnectionASE - OpenSwitch ConnectionAutonomy
DHQ - DRE Connection
OpenSwitch
DRE
DRE Database
Spider
EAServer
DQH ()
() Note the DQH is not part of the Sybase EP
product
CMS
26
Sample Architectures Using Messaging and
Workflow with Persistence (Intranet Server Cloud)
In Theory
A Queue Listener attempts to apply the
Message to one or more targets
End- User Transaction Logged Locally
Inbound Messages are received and logged
to a Stable Device
Operation results are funneled into a
Response Queue regardless of status.
Server Cloud Network
Remove Local Message Instance
Send an Acknowledgement
27
Sample Architectures Using Messaging and
Workflow with Persistence (Intranet Server Cloud)
In Practice..
Middleware Server
Event Broker
Message TransactionDatabase
Message Repository
Middleware Server
Event Broker
Common Message Bus
Middleware Server
Client Side Message Persistence
Event Broker
Message TransactionDatabase
Message Repository
Middleware Server
Event Broker
3rd Party Message ExchangeASE Client
ConnectionSybase Messaging Exchange
28
Sybase Continuous Availability Technologies

• High Availability and Load Balancing for Popular
  Object Models
• Java/EJB, C/C, CORBA, ActiveX, Power Builder
  NVO, Java Servlet
• IOR and Digital Certificate Caching
• State-full and State-less Component Transaction
  Fail-over
• Object Transactional Services (OTS, X/Open-XA,
  MS-DTC)
• Integrated Transarc Encina XA Libraries
• Symmetric and Asymmetric Hardware Clustering
• UNIX and NT Bridging
• Cluster Aware Naming Services
• Servers use JNDI and COS Naming Services
• IOR Browse and Lookup
• Support for Peer and Slave/Master
  Configurations

29
Sybase Continuous Availability Technologies

• Adaptive Server 11.5
• Page Level Fault Isolation
• Point-In-Time Recovery
• Adjustable Recovery Interval
• Function Shipping and Proxy Table and Procedure
  Support
• Adaptive Server 11.9
• Exportable Object Statistics
• Background-Line Garbage Collection (deleted
  space management)
• Adaptive Server 12.0
• HA Type 2 Cluster Support
• Active / Active Server Configuration
  (multi-entry point)
• Active / Passive Server Configuration
• Automatic Client Fail-over
• Single Image System
• On-Line Index Reorganization

30
Sybase Continuous Availability Technologies

• Single Image System Architecture
• Administrator DDL Propagated to Companion Node
• Automated Schema Scavenging (proxy definitions)
  
• Proxy Data Base (Active/Active multi- entry
  point system)
• Hybrid Shared Nothing Implementation
• Distributed Transaction Manager (XA and DTC
  Functionality
• De-coupled Transaction State Management
• Forget/Complete transaction
• Attach/Detach transaction
• Open Client Changes (HA_FAILOVER)
• Set Up for Transparent Application Fail-Over
• Hardware Vendor Specific Features
• HA Coordination Modules
• Sun, HP, NT, AIX, DEC
• Quiesce DB
• EMC Symmetrix, Platinum

31
Sybase Continuous Availability Technologies

• Data Transfer Rates
• Full Database Dump tested at 520GB Per Hour
• Full Database Load tested at 540GB Per Hour
• Enhancements to Backup Server
• Server API enhancements
• Buffer and I/O Zones
• Private Shared Segment
• Parallel Data Transfer Operations (sybmultbuff)
• Variable Device Block Size
• No Limits Project - Increased Dump Stripe
  Partition Number (512)
• No Limits Project - Increased User File System
  Partitions (32GB)
• Early Technology Adopters
• Bear Stearns, AOL, Morgan Stanley others..

32
Sybase Continuous Availability Technologies

• Why Use Software Level Warm Standby?
• Hardware Solutions Do NOT Guarantee Recovery
  (transaction unaware)
• Hardware Replication WILL Replicate Corruption
• Hardware Mirroring Impossible over WAN
• Over 70 of the outages are due to data
  corruption?
• Sybase Replication Functionality
• Message Based, Asynchronous Data Delivery
• No Two-Phase Commit, No Snap Shot
• Minimal (less then 6) overhead
• Transaction Level Recovery even when data
  corruption occurs
• Proven technology, pioneered by Sybase
• Allows Many-to-Many (NN) Data Movement
• Schemas Need Not Match
• Heterogeneous DBMS Source and Targets
• Object Data Type Support for Sybase ORDBMS
  Model
• Changes in Java Object are replicated

?Source of numbers is Gartner Group and GIGA
1997,1998 Reports
33
Sybase Continuous Availability Technologies

• Replication Complements HA Fail-Over in cluster
  configurations
• Companion server takes over primary servers
• workload upon fail-over
• Allows automatic fail-over of RS connection to
• ASE Companion Server
• Replication Server 12 Supports ASE/ASA Java
  Objects
• May be used as an Object Backup and
  Distribution Mechanism
• May be used for Web Site content Backup and
  Distribution
• Enhanced Heterogeneous Replication Capabilities
• Dynamic Date-Time Conversions
• Built in conversion routines for DB2, Oracle,
  Informix, others..
• Interoperability with Commercial Message
  Substrates (Event Broker)

34
Sybase Continuous Availability Technologies

• Is an Open Client (TDS) Concentrator
• Manages pools of users
• Allows Transparent Application Fail-Over
• Transaction State Recovery
• Performs Transaction and SQL buffering
• Implements timeout/retry callback
• No application code changes required
• Sybase Supplies Custom Coordination Modules
• Maintain fail-over Rules
• Generate Network Alerts
• Can be used in conjunction w/ other HA tools

35
Summary

• Much like the telephone and the electrical
  power grid before, resilient e-Systems are highly
  distributed and highly redundant by design.
• The more things change
• HA ! CA
• Why do it? Consider the impact of your service
  outage
• Hardware alone will NOT solve HA/CA issues
• Software alone will NOT solve HA/CA issues
• Shared- nothing Systems are prevailing as result
• Designing Fault Tolerant Systems is a full time
  job

36
Sample Architectures Shared Repository Peer
Cluster (internals)
Hardware HA and Load Balancing (Level 4
Switch) CISCO Local Director, GloBal HydraWeb,
Foundry Networks Server Iron, RadNetworks WSD
PRO, F5 BigIP
HTTP HTTPS
SSL Encryption
Firewall
OSPF, EIGRP, RIP, BGRP
DMZ Ring
Web Server Presentation Layer
IIOP, IIOPS Open Client Java RMI
HA Aware Application Hosting Nodes
Ping
Ping
Ping
Open Client IBM DRDA SQLNet ODBC/JDBC
Shared Data Repository
Intranet
Back
37
Sample Architectures Slave-Master (Cascading
Directory) Shared Nothing Cluster (internals)
Hardware HA and Load Balancing (Level 4
Switch) CISCO Local Director, GloBal HydraWeb,
Foundry Networks Server Iron, RadNetworks WSD
PRO, F5 BigIP
HTTP HTTPS
Firewall
SSL Encryption
DMZ Ring
OSPF, EIGRP, RIP, BGRP
Web Server Presentation Layer
Cascading Naming Directory Application Hosting
Nodes (ie. JNDI, LDAP, CORBA)
HTTP HTTPS SSL Encryption IIOP, IIOPS Open
Client Java RMI
Lead Name Server
Lookup
Lookup
Lookup
Open Client IBM DRDA SQLNet ODBC/JDBC MS
Replication
Data Repositories
Data Synchronization (ie. Replication Server)
Intranet
Back
38
Sample Architectures Redundant (Dual Node)
Shared Repository Peer Cluster (internals)
Software HA and Load Balancing (Multiple Entry
Point System) Static URL Redirect, Mirrored Web
Sites, Dynamic URL Redirects
HTTP HTTPS
SSL Encryption
Firewall
OSPF, EIGRP, RIP, BGRP
DMZ Ring
Web Server Presentation Layer
IIOP, IIOPS Open Client Java RMI
HA Aware Application Hosting Nodes
Ping
Ping
Ping
HA Aware Data Repository Nodes
Open Client IBM DRDA SQLNet ODBC/JDBC
Ping
Back
39
Sample Architectures Shared Nothing Peer Cluster
(internals)
Software HA and Load Balancing (Multiple Entry
Point System) Static URL Redirect, Mirrored Web
Sites, Dynamic URL Redirects
HTTP HTTPS
SSL Encryption
Firewall
Firewall
OSPF, EIGRP, RIP, BGRP
DMZ Ring
DMZ Ring
Web Server Presentation Layer
IIOP, IIOPS Open Client Java RMI
HA Aware Application Hosting Nodes
Ping
Ping
Ping
Open Client IBM DRDA SQLNet ODBC/JDBC
Stand-Alone Data Repositories
Data Synchronization (ie. Replication Server)
Intranet
Back
40
Sample Architectures Virtual Database and Single
Images Systems (internals)
Hardware HA and Load Balancing (Level 4
Switch) CISCO Local Director, GloBal HydraWeb,
Foundry Networks Server Iron, RadNetworks WSD
PRO, F5 BigIP
HTTP HTTPS
SSL Encryption
Firewall
OSPF, EIGRP, RIP, BGRP
DMZ Ring
HA Aware Application Hosting Nodes
Web Server Presentation Layer
IIOP, IIOPS Open Client Java RMI
HA Aware Data Repositories
Ping
Ping
Ping
Single Image Repository
HA Un-Aware 3rd Party Repositories
Open Client IBM DRDA SQLNet ODBC/JDBC
CIS Proxy Services
Back
41
Mirrored Device Parallel Reads (internals)
User A
Client Write I/O
High Speed Interconnect
Avoids Some Disk Head Collision by taking
advantage of Mirror (ie. Sun Photon, EMC)
A
B
Busy
Device Mirror
Device Master
Client Reader I/O
User B
Back
42
Typical hardware HA configuration
High Speed Switch (Token Ring or SAN/WAN) Local
FDDI Loop
High Speed Interconnect
HA Sub-System
HA Sub-System
Heart Beat Monitor
Run Level 0,1,2,5,6 scripts Coordination
Modules DBMS Vendor Software
Machine A
Machine B
Cluster Monitor Server (optional)
Cluster Server Monitor Software
Back