Title: Arial 28pt'

1
REP705 A Simple DR/HA Implementation for
Replication Servers Based on NAS Filer
Tung Chen Senior DBA Tung.Chen_at_Barclaysglobal.com
August 15-19, 2004
2
Agenda

• What is the Problem We Are Solving
• What is a Filer (Network Based Device)
• System Architecture Overview
• How to Build the DR System for Replication Server
• Cross-site (DR) Failover Procedure
• Local Failover Procedure
• Performance Benchmark
• QA

3
What is the Problem We Are Solving?

• Everybody knows how to use Warm Standby
  Replication to replicate ASE Databases from main
  site to DR site. But, what happens to the Rep
  Servers after the DR failover?
• Here is the big picture for most of the DR
  implementation

4
What is the Problem We Are Solving?

• Before the DR Switch-over

San Francisco D.C.
INV Usr
Sacramento D.C.
Rep Svr
W.S.
Subs Rep
RSSD ASE
WAN
80 miles
P2DS.PRC_DB
W.S.
PRC Usr
5
What is the Problem We Are Solving?

• After the DR Switch-over

San Francisco D.C.
Sacramento D.C.
Rep Svr
PDS.INV_DB
W.S.
Subs Rep
RSSD ASE
WAN (high spd)
80 miles
P2DS.PRC_DB
W.S.
6
What is the Problem We Are Solving?

• A Logical View Before

W.S.
Logical Entity.INV_DB
t_a
Logical Entity.PRC_DB
P2DS.PRC_DB..t_a ..t_c
W.S.
7
What is the Problem We Are Solving?

• A Logical View After

Logical Entity.INV_DB
t_a(??)
Logical Entity.PRC_DB
8
What is the Problem We Are Solving?

• Once switched to DR site (DR failover), there is
  no Replication Server on the DR site. Only ASE
  databases are brought up on DR site.
• Subscription replication (table t_a) from INV_DB
  to PRC_DB is lost Going forward, there is no
• subs. replication

9
Idea

• If we mirror Rep Servers to the DR site, we can
  bring up Rep Servers on the mirror image at DR
  site

10
What Constitute A Rep Server ?

• The following components are mirrored across the
  WAN for Rep Server DR Implementation
• Rep Server software directory (file system)
• Rep Server stable devices (file system)
• ASE Server software directory (file system) For
  RSSD
• ASE Server Device files (file system)

11
Network Appliance Filer Cluster

• Our Solution Network Appliances filer, or FAS
  (fabric-attached storage) series with SnapMirror
  software
• FAS 900 and FAS 200 series support NFS and
  block-storage access through network
• FAS960 cluster the storage system is fully
  redundant, with RAID protection
• FAS series is certified by Sybase as database
  devices
• According to NetApp, FAS data availability is
  greater than 99.998

12
Network Appliance Filer Cluster

• NetApp FAS 960 cluster on both data centers
• Four heads in each cluster each head has 9
  trays each trays provides 600 GB usable space
  after formatting
• The filers are backed up using NDMP Netbackup

13
What is SnapMirror?

• NetApp SnapMirror is a software which mirrors
  data from one filer to one or more network
  filers.
• It continually updates the mirror data to keep it
  current and available for DR.
• After the initial sync-up, it only updates the
  new and changed blocks incrementally over the
  network reduced the network bandwidth
  requirement.
• SnapMirror is based on IP
• It has 3 levels of synchronicity sync, semi-sync
  and async.

14
What is SnapMirror?

• We use async SnapMirror to replicate Primary-site
  filer cluster to DR site filer-cluster
• SnapMirror is scheduled every 10 minutes for both
  directions (not on the same volume)

15
SnapMirror
San Francisco D.C.
Sacramento D.C.
WAN (high spd)
80 miles
SnapMirroring
Filer Cluster
Filer Cluster
16
How Are Disk Volumes Organized/Mirrored?

• Data on a NetApp filer is organized in Volumes
• Each Volume has at least one RAID group.
• We use 8-disk RAID group.
• Each volume has multiple Qtrees
• Each Qtree hosts a user file system
• In our implementation, we have 4 file systems on
  4 Qtrees sharing one volume

17
How Are Disk Volumes Organized/Mirrored?

• Each Qtree inherits has security config, quota,
  etc
• Qtree is our work unit for SnapMirroring and
  server failover, i.e. disk image for a Rep Server
  is contained in a Qtree
• Each Qtree is snapMirrored to a mirror Qtree

18
Failover Package

• Rep Server needs an RSSD on ASE (Consider ERSSD
  12.6)
• We define the Failover Package as a Rep Server
  and RSSD ASE
• There are 4 file systems on the primary filer
  cluster, snapMirrored to DR-site filer cluster
• File system for Rep Sever executables, libs,
  config files, etc REP-12_5, OCS-12_5, locales,
  etc
• File system for stable devices
• File system for ASE executables, libs, config
  files, etc
• File system for ASE data devices

19
Failover Package/Filer Redundancy
San Francisco D.C.
Sacramento D.C.
WAN (high spd)
80 miles
SnapMirroring
Filer Cluster
Filer Cluster
20
HW Redundancy

• With redundant HW standing by, failover can occur
  across the WAN for DR, or in the same data center
  for a local HA implementation
• Stand-by HW does not have to be idle before
  failover
• Need to ensure at the time of failover, the
  target host will have appropriate
• CPU resources
• memory resources,
• mount points for NFS mounts

21
DR/HA System
Host A
Rep Svr
Sacramento D.C.
San Francisco D.C.
RSSD ASE
WAN (high spd)
Local Failover Target
DR Failover Target
80 miles
Host B
Host C
SnapMirroring
Filer Cluster B
Filer Cluster A
22
DR Failover
Host A
Sacramento D.C.
San Francisco D.C.
Rep Svr
WAN (high spd)
Local Failover Target
80 miles
RSSD ASE
Host B
Host C
SnapMirroring
Filer Cluster B
Filer Cluster A
23
Local Failover
Host A
Sacramento D.C.
San Francisco D.C.
Rep Svr
WAN (high spd)
DR Failover Target
80 miles
RSSD ASE
Host C
Host B
SnapMirroring
Filer Cluster
Filer Cluster B
Filer Cluster A
24
How to Build the DR System?
Host A
. 4 FSs . HOME
Sacramento D.C.
San Francisco D.C.
WAN (high spd)
Local Failover Target
DR Failover Target
80 miles
Host B
Host C
SnapMirroring
Filer Cluster B
Filer Cluster A
25
How to Build the DR System?
1. (Unix SA) Create volume and Qtrees on the
primary filer cluster A create volume
nas_rep Qtree create nas_rep Qtree security
/vol/nas_rep Unix 2. (Unix SA) Export the FS on
primary filer Edit the exports file with the
following lines /vol/nas_rep/hostA
-accessadmin,roothostA exportfs -a
26
How to Build the DR System?
3. (Unix SA) Mount FS on host A Edit the
/etc/vfstab file on hostA to include the
following lines filerA/vol/nas_rep/hostA -
/nfsapps/hostA/sybase/apps/syb-RDC_NAS nfs - yes
rw,bg,hard,intr,rsize32768,wsize32768,prototcp,
forcedirectio,noac,vers3
27
How to Build the DR System?
4. (Unix SA) Create volume on host C (cluster
filer B) and schedule Qtrees for snapMirroring
from filer A to filer B All following
commands are issued in DR site filer/host
create volume vol2 Change volumes to restrict
vol restrict vol2 Edit snapmirror.conf file on
target filer B with lines Source
filer/volume/qtree Destination
filer/volume/qtree rate-of-transfer filerA/vol/n
as_rep/hostA filerB/vol/vol2/hostC kbs3072,
restartalways 0-59/10 snapmirror
initialize /vol/vol2/hostC Note As long as
Qtrees are replicated, we cannot write into the
target volume/Qtrees
28
How to Build the DR System?

• 5. (Unix SA) Configure 4 file systems on filer
  cluster A and mounts them on primary host A (2
  GB each)
• /apps/RDC_NAS ? /nfsapps/hostA/sybase/apps
  /syb-RDC_NAS
• /apps/RDC_NAS_RS ? /nfsapps/hostA/sybase/apps/rep
  -RDC_NAS_RS
• /data/RDC_NAS ? /nfsapps/hostA/sybase/dat
  a/syb-RDC_NAS
• /data/RDC_NAS_RS ? /nfsapps/hostA/sybase/data/re
  p-RDC_NAS_RS

29
How to Build the DR System?
Host A
. 4 FSs . HOME
Sacramento D.C.
San Francisco D.C.
WAN (high spd)
Local Failover Target
DR Failover Target
80 miles
Host B
Host C
SnapMirroring
Filer Cluster B
Filer Cluster A
30
How to Build the DR System?

• 6. (DBA) Build RSSD ASE Server RDC_NAS on host A
  using the file systems /apps/RDC_NAS and
  /data/RDC_NAS
• 7. (DBA) Build Rep Server RDC_NAS_RS on host A
  using the file systems /apps/RDC_NAS_RS and
  /data/RDC_NAS_RS

31
How to Build the DR System?
San Francisco D.C.
INV Usr
Sacramento D.C.
Host A
Rep Svr
RDS.INV_DB
RSSD ASE
WAN (high spd)
80 miles
Local Failover Target
Host B
Host C
R2DS.PRC_DB
P2DS.PRC_DB
SnapMirroring
PRC Usr
Filer Cluster A
Filer Cluster B
32
How to Build the DR System?

• 8. (DBA) Use rs_init to build WS replication
  for PDS.INV_DB and P2DS.PRC_DB. Rep Defs are
  based on logical connection
• 9. (DBA) Build subscription replication based
  on logical connections

33
DR/HA System
34
DR/HA System

• Note It is important that all subscription
  replications are based on logical connections!

35
DR/HA System

• Each W.S. Replication swtich-over can occur
  independently

W.S.
Logical Entity.INV_DB
t_a
Logical Entity.PRC_DB
P2DS.PRC_DB..t_a ..t_c
W.S.
36
DR Failover Procedure

• Failover from Sacramento D.C. to S.F. D.C.
• System is running nominally,.. when the disaster
  strikes
• Sacramento D.C. is gone hosts A B are gone

37
DR Failover Procedure
San Francisco D.C.
Sacramento D.C.
INV Usr
Host A
Rep Svr
W.S.
Subs Rep
RSSD ASE
WAN (high spd)
80 miles
Local Failover Target
Host B
Host C
P2DS.PRC_DB
W.S.
SnapMirroring
PRC Usr
Filer Cluster A
Filer Cluster B
38
DR Failover Procedure
Failover from Sacramento D.C. to S.F. D.C. 1.
(Unix SA) Break the snapMirroring from filer
cluster A to filer cluster B. Assuming data
updates and replication are continuous, filer B
may be 10 minutes behind filer A snapmirror
quiesce /vol/nas_rep/hostA snapmirror break
/vol/nas_rep/hostA Note Once this is done, all
target volume and Qtrees become writable 2.
(Unix SA) Mount the 4 file systems on host C
(filer cluster B)
39
DR Failover Procedure
San Francisco D.C.
W.S.
WAN (high spd)
80 miles
Host C
W.S.
SnapMirroring
Filer Cluster A
Filer Cluster B
40
DR Failover Procedure
San Francisco D.C.
W.S.
WAN (high spd)
80 miles
Host C
W.S.
SnapMirroring
Filer Cluster A
Filer Cluster B
41
DR Failover Procedure
3. (DBA) Modify interfaces file on host C
Update the IP for Rep Server and RSSD ASE Server.
Push out new interfaces file to S.F. server
hosts only
42
DR Failover Procedure
Before Server RDC_NAS hostA
(49.38.100.22) port 16117 RDC_NAS 9 9
query tli tcp /dev/tcp \x000237257777777d000000000
0000000 master tli tcp /dev/tcp
\x000237257777777d0000000000000000 Server
RDC_NAS_BS hostA (49.38.100.22) port
16127 RDC_NAS_BS 9 9 query tli tcp
/dev/tcp \x0002372f7777777d0000000000000000
master tli tcp /dev/tcp \x0002372f7777777d000000
0000000000 Server RDC_NAS_RS hostA
(49.38.100.22) port 16157 RDC_NAS_RS 9 9
query tli tcp /dev/tcp \x0003333d7777777d000000000
0000000 master tli tcp /dev/tcp
\x0003333d7777777d0000000000000000

43
DR Failover Procedure
After Server RDC_NAS hostC
(48.37.102.21) port 16117 RDC_NAS 9 9
query tli tcp /dev/tcp \x000237255555555d000000000
0000000 master tli tcp /dev/tcp
\x000237255555555d0000000000000000 Server
RDC_NAS_BS hostC (48.37.102.21) port
16127 RDC_NAS_BS 9 9 query tli tcp
/dev/tcp \x0002372f5555555d0000000000000000
master tli tcp /dev/tcp \x0002372f5555555d000000
0000000000 Server RDC_NAS_RS hostC
(48.37.102.21) port 16157 RDC_NAS_RS 9 9
query tli tcp /dev/tcp \x0003333d5555555d000000000
0000000 master tli tcp /dev/tcp
\x0003333d5555555d0000000000000000

44
DR Failover Procedure
4. (DBA) Bring up RSSD ASE Server RDC_NAS on
host C on mirror image 5. (DBA) Bring Rep
Server RDC_NAS on host C on mirror image
45
DR Failover Procedure
San Francisco D.C.
W.S.
WAN (high spd)
80 miles
Host C
W.S.
SnapMirroring
Filer Cluster A
Filer Cluster B
46
DR Failover Procedure
6. (DBA) Perform WS switch-over for databases
INV_DB and PRC_DB. New primaries are RDS.INV_DB
and R2DS.INV_DB 7. (DBA) Both DBs in dbo use
only mode Reconcile lost data on PRC_DB and
INV_DB 8. (DBA) Open up S.F. DBs Push out new
interfaces file to all client machines
47
DR Failover Procedure
San Francisco D.C.
W.S.
WAN (high spd)
80 miles
W.S.
SnapMirroring
Filer Cluster B
48
DR Failover Procedure
San Francisco D.C.
W.S.
WAN (high spd)
80 miles
W.S.
SnapMirroring
Filer Cluster B
49
DR Failover Procedure

• Notes
• Depending on the time of the crash/disaster, the
  mirror image may be 10 minutes behind the primary
  image for Rep Server stable queue. Data loss
  is/should be expected due to snapMirror delay
• Crash test confirmed data loss in some cases, but
  we are always able to bring up Rep Server and
  RSSD ASE Server on the mirror image on DR site
• Test on Filer Cluster shows a reliable and
  transparent failover in the filer cluster

50
Local Failover

• Local Failover on filer is very useful and much
  simpler
• Stay on the same filer and same data center
• -- No need to break SnapMirror or do anything
  on filers
• -- No need to perform WS switch-over on Rep
  Server
• Only need to mount the FSs on new target host B
• There will be no data loss on Rep server stable
  queues ( because primary filer cluster remains up
  )

51
Local Failover
Host A
Sacramento D.C.
San Francisco D.C.
RSSD ASE
WAN (high spd)
DR Failover Target
Local Failover Target
80 miles
Host C
Host B
SnapMirroring
Filer Cluster
Filer Cluster B
Filer Cluster A
52
Local Failover
Host A
Sacramento D.C.
San Francisco D.C.
Rep Svr
WAN (high spd)
DR Failover Target
80 miles
RSSD ASE
Host C
Host B
SnapMirroring
Filer Cluster
Filer Cluster B
Filer Cluster A
53
Local Failover Procedure

• Local Failover from Host A to Host B
• System running nominally, when Host A crashed
• (Unix SA) Mount the 4 file systems on target host
  B
• (Unix SA) Issue the command sm_mon l hostA to
  release internal lock on master device file
• (DBA) Push out new interfaces file globally to
  server hosts and client hosts
• (DBA) Bring up RSSD ASE Server and Rep Server on
  Host B

54
SAN-FAS Sybase Benchmark
55
Summary

• With proper NAS filers and snapMirror
  implementation, Rep Servers can be practically
  brought up on any host in any data center for
  failover
• FAS960 filer performance is good
• Data loss should be expected due to snapMirror
  delay. A data reconciliation mechanism should be
  implemented and tested

56
QA