PNFS and DB Server Configuration

1
PNFS and DB Server Configuration

• Hardware and software configuration for pnfs and
  database servers

2
Why the change is needed

• Increasing volume of the database
• Increasing load on the pnfs server
• Backups take longer (D0 - 1.1h, size 8.7G, CDF
  - 1.7h, size 17G, STKen - 2.3h size 23G)
• In case of DB failure recovery will take a very
  long time
• Metadata entered between backups will be
  difficult to recover in case of the DB failure
  (backups are taken every 4 hours)
• To increase robustness and to substantially
  reduce a downtime in case of failure

3
Pnfs backup time
CDF
D0
General
4
Average CPU Loads
D0 - Pentuim III Dual not threaded 1.4GHz General
Xeon. Dual Threaded 2.4GHz CDF Xeon. Dual
Threaded 2.8GHz
5
Current pnfs configuration

• One separate pnfs server for each system d0,
  CDF, general. CMS has recently installed its own
  pnfs server.
• DB is kept on external HW-RAID disk (RAID-5)
• Gdbm was converted to PostgreSQL allowing for the
  bigger (practically unlimited DB size, maximal
  table size 16TB).
• PostgreSQL DB server runs on the same machine
  with pnfs server.
• Backups are made on the separate machine.

6
Few words about enstore File DB

• Used as internal enstore file DB
• Runs on separate servers
• Current sizes D0 3 GB, CDF 1.4GB, General
  8.2 GB.
• We are planning to combine all File DBs into one.
  

7
Proposed Modification

• Live replication of pnfs DB (PostgresQL 8 and /or
  Slon-I, etc). In case of DB failure the system
  can be quickly reconfigured for backup system to
  become a main one.
• Enstore file database can be replicated using the
  same approach.
• Need an auxiliary server to make a replica and
  be ready to run as a replacement pnfs server.

8
Main auxiliary pnfs and DB server

• Each system will require 2 machine
  configuration. One will run a PostgreSQL master
  and pnfs server. The other will be used for the
  replication performing a live DB backup.
• There still will be periodic backups running on
  ether main or auxiliary machine.
• In case of failure of the main server the
  machines can be switched either by cloning main
  server name and IP or using network aliasing.
• Aliasing requires reloading of DNS server. All
  clients have to remount NFS
• File DB will be replicated the same way and does
  not require modification of the network
  presentation of the file and volume servers.
• Additional pnfs databases (EAG, LQCD) can also be
  replicated to an auxiliary machine supporting the
  General enstore system, but we do not have
  machines to replace these pnfs servers in case of
  failure.
• In addition to this configuration, one spare
  machine is required

9
Main auxiliary host configuration
Main configuration
Pnfs server DB server
Pnfs server DB server
Pnfs server DB server
DB live backup
DB live backup
DB live backup
Spare Pnfs server DB server
Emergency configuration
Pnfs server DB server
Pnfs server DB server
Pnfs server DB server
Pnfs server DB server
DB live backup
DB live backup
10
Verification of the integrity

• In current configuration to synchronize DB the
  backup needs to run when pnfs server is down and
  may require a special downtime for pnfs server.
• In proposed configuration only a very short
  downtime of pnfs server is required and can be
  done during a regular enstore system downtime.

11
System backup

• After the configuration changes backup will still
  be done from the main machine to the current
  backup server for a while
• After we are sure that the new configuration
  works well and databases on the main and
  auxiliary machines are continuously identical
  well switch backup to the auxiliary machine.

12
Proposed hardware solution

• We assume that pnfs servers will be upgraded
  using new machines and existing pnfs servers will
  become backup machines. The new machines should
  have
• 4GB ram
• Fast processors.
• Fast disk. Raid-10 would give the best
  throughput, and should be faster for I/O
  dominated by short writes, but at a cost in
  capacity. Raid-5 may be adequate, but we'd need
  to do some performance testing. The capacities
  below assume Raid-10.
• Both systems below have a common 1 SuperMicro 3U
  SC833T-R760 chassis, with X6DH8-XG2 motherboard 2
  Intel Xeon 3.0GHZ processors with 800MHZ
  Front-Side Bus 4 GB RAM 2 Mirrored 10K RPM system
  disks in hot-swap carriers

13
Detailed configuration

• SCSI 9,300 3U box
• SCSI system disk mirror with Adaptec controller.
• SCSI data array of 7 146 GB 10K RPM, hot spare
  (438 GB)
• Adaptec 2120S SCSI RAID controller
• SATA - 6,500 3U box
• SATA system disk mirror with 3-Ware controller.
• SATA data array of 8 300GB 7200 RPM, hot spare
  (900 GB).
• 3-Ware RAID controller (like what many Dcache
  nodes have)