Server Appliance Proposal

1
High Availability in the ERHow McKesson
Leverages Oracle Real Application Clusters and
ASM in Critical Hospital Environments
Jay Caviness Sr. Systems Engineer McKesson
Provider Technologies

2
Topics

• McKesson / Oracle History
• Why use RAC?
• Why ASM?
• RAC Management
• Issues Questions

3
McKesson Provider Technologies

• Largest Healthcare IS Company
• McKesson Clinicals Applications
• 17 Applications Suites
• Meds Manager
• ER
• Billing/Revenue Management
• Nursing
• NICU/PICU

4
About Me

• Oracle DBA for 19 years
• Senior Engineering DBA for McKesson Provider
  Technologies
• Administered Oracle from v5.1 11g
• OPS on VAX/VMS to RAC on Oracle 10g
• Oracle Certified Professional
• Known as the Grumpy DBA

5
McKesson Horizon Clinicals and RAC

• Over 800 hospital deploy McKesson applications
• Over 400 hospitals use Horizon Clinicals suite of
  applications
• RAC deployed starting in late 2005 (10gR1)?
• As of today more than 75 hospitals run Oracle RAC
• - over 250 clusters over 800 nodes
• - 94 on Linux- remainder on HP/UX and AIX
  (largest supports 60 hospitals)?
• - 98 of new installations are on RAC- 65 of
  existing legacy hardware customers intend to
  migrate to linux and RAC in the next three
  years.

6
McKesson Horizon Clinicals Client

• 1-3 clusters
• HCI Horizon Care Infrastructure
• HCR Horizon Care Record data mart
• HMM Horizon Medication Manager
• 1-6 nodes each cluster
• 3-5 databases per cluster
• live/qa/training configuration databases
• Total Disk space per cluster 400GB
• depends on how often data is sent to data mart
  (HCR)?
• Growth 2030 year over year
• Retention requirements vary by HIPAA rules,
  typically 7-10 years

7
RAC Overview

• Multiple instances running on separate servers
  (nodes)?
• Single database on shared storage accessible to
  all nodes
• Instances exchange information over an
  interconnect network

8

• Why use RAC?

9
Why RAC?

• Cost
• Commodity hardware x86 - 64bit
• Linux in most cases
• Highly Available
• Deployable Automated Scripts for
• Installation
• Add/Remove Nodes
• Scalable
• Load Balancing
• Client/Server Side
• Services

10
Deployment

• Standard silent install process
• Standard install difficult over limited bandwidth
• Same script for all platforms
• Pre-defined nodes, interconnect, vip, network
  storage from McKesson OS install process
• Script determines RAC/non-RAC and prompts user
  for
• RAC - full install (CRS, ASM DB),
  infrastructure (CRS ASM) or database
• Non-RAC full install, ASM Only, Database Only
• Installation includes software, patchsets and
  configuration

11
Typical Oracle RAC Installation

• Red Hat Linux 4.6
• Oracle 10.2.0.4
• Dell 2950 or HP DL380 w/dual quad core processors
• 16G RAM
• Powerpath on EMC Clariion
• 4 Nodes
• UDP Interconnect via Private VLAN
• No crossover cables for two node clusters
• NIC bonding is supported
• 40 Minute Installation Time

12
Failover

• New and legacy applications have different
  connection types
• JDBC
• OCI
• SQLNet
• Application Failover via TAF for select
  statements
• Applications designed to automatically reconnect

13
Load Balancing

• Two of three types of load balancing handled by
  RAC
• Server side load balancing via the cluster
  listeners
• Client side load balancing via tnsnames.ora
  entries and jdbc connection strings
• Application load balancing with hardware and
  virtual load balancers
• Services are used to offload reporting and other
  processes to specific node(s) of a cluster

14
Application Changes

• Few, but important changes were made to the
  applications to fully support RAC
• Cached/No order sequences
• JDBC connection strings
• SQLNet connection strings
• Reconnection process

15
Rolling Upgrades

• With RAC different upgrade types can be done
  without downtime
• Hardware upgrades
• Operating system patches/upgrades
• Oracle clusterware/ASM/database patches and some
  upgrades
• Application upgrades

16

• Automated Storage Management

17
Why ASM

• ASM is required by McKesson for RAC and strongly
  suggested for non-RAC databases
• We chose ASM for several reasons
• Setup and Management
• Simplicity
• Scalability Performance
• Robust and Stable

18
ASM - Setup

• Installed in its own Oracle Home
• Typically external redundancy
• Some clients have two fail groups on physically
  separated SANs
• 400G/per cluster to start
• Easily expandable by adding 50G luns to ASM
• Datafiles set to autoextend-
• caveat 32g limit for 8k blocks unless using
  bigfile tablespaces

19
ASM - Simplicity

• ASM allows DBAs to be less concerned about file
  management
• db_create_file_dest
• ex create tablespace foo datafile
• db_create_online_log_dest_n
• ex alter database add logfile thread 2 size
  200m
• Oracle Managed Files

20
ASM Scalability Performance

• ASM allows management without affecting running
  databases
• Adding/removing ASM luns/disks
• Add the lun (owned by oracle)
• Mark the disk with the oracleasm service (linux
  only)
• alter tablespace dgroup1 add disk vol009
• Performance by striping
• Datafiles striped in 1m chunks across all ASM
  disks
• Creates a plaid on previously striped luns

21
ASM - Stability

• Even at the first generation (10gR1) there where
  no show stopper bugs with ASM
• All versions are very stable
• Minor issues due to lun management, mainly at the
  OS level
• Luns lose ownership on reboot HP/UX
• Special files to denote ownership can be lost and
  must be recreated if the luns are re-imported
  AIX

22

• RAC Management

23
RAC Management

• Grid Control 10gR4
• 10gR4 agents
• Automated install process
• One button install
• Deploy agents
• Standard alerts
• Easy for DBAs and support personnel

24
Issues

• The largest issues faced are generally due to the
  varying use of the applications.
• Client use varies
• Modifiable applications
• Client queries
• Client modifications
• Reporting
• Inconsistent setup
• Hostnames
• Network
• Storage
• Number/type of applications

25
Future

• The future of Oracle/RAC is happening now!
• OEL 5.2 retiring other operating systems for
  new clients
• McKesson System Appliance
• Consistency
• Uniformity
• Cost savings for clients in hardware, energy,
  support/services
• Virtual Machines

26

• Questions?

27
More information

• Contact jay.caviness_at_mckesson.com
• Corporate www.McKesson.com/clinicals
• Personal www.grumpy-dba.com