BACKUP/GETTING STARTED: Backup School

1
BACKUP/GETTING STARTEDBackup School

• Presented by
• W. Curtis Preston
• VP, Services Development
• GlassHouse Technologies

2
Making good on your investment

• Many SANs are built in order to simplify backup
  yet often fail for lack of good design, processes
  and procedures
• There are several common mistakes that people
  make when building a backup system
• Avoiding these mistakes and taking proper action
  can create a backup system that is reliable and
  restorable

3
What will we cover?

• Common backup configuration mistakes
• How to avoid them
• Sizing your backup system
• Configuration examples for NetBackup
• Configuration examples for NetWorker

4
Common backup configuration mistakes
5
Where do these lessons come from?

• Audits of real backup and recovery systems
• Lessons learned from real horror stories
• Many, many sleepless nights

6
Too little power

• Not enough tape drives
• Tape drives that arent fast enough
• Not enough slots in the tape library
• Not enough bandwidth to the server

7
Too much power

• Streaming tape drives must be streamed
• If you dont, you will wear out your tape drives
  and decrease aggregate performance
• Must match the speed of the pipe to the speed of
  the tape
• You can actually increase your throughput by
  using fewer tape drives

8
Not using multiplexing

• Defined Sending multiple backup jobs to the same
  drive simultaneously
• Again, drives must be streamed
• Multiplexing will impact restore performance but
  not as much as you might think
• Multiplexing can actually help your restore just
  as it can help your backups
• Using multiplexing can greatly increase the
  utilization of your backup hardware

9
Not using multistreaming

• Defined Sending multiple simultaneous backup
  jobs from a single client
• Large systems cannot be backed up serially
• Multistreaming creates a different job for each
  file system

10
Using include lists

• Most major backup software supports file system
  discovery
• Still, many administrators use manually created
  include lists
• Any perceived value is significantly outweighed
  by the risk it creates

11
Too many full backups

• If you are using a commercial backup and recovery
  product with automated media management and
  multiple levels, weekly full backups are a waste
  of tape, time and money
• Monthly full backups, weekly cumulative
  incrementals (1), and daily incrementals (9) work
  just as well and use ¼ as much tape
• Depending on the level of incremental activity,
  quarterly backups can work just as well

12
Not standardizing

• Creating custom configurations for each client is
  easier, but much riskier
• Creating a standard backup client configuration
  can significantly decrease risk
• Create a standard exclude list, etc. and push it
  out to each client

13
Not even noticing!

• Backups go ignored so often. Its like theyre
  the bill collector nobody wants to talk to.
• Backup reporting products can really help
  automate easy reporting
• Dont ignore backups. They will bite you.

14
Its just backups, right?

• Im an experienced, seasoned systems
  administrator. This is just backups. How hard can
  they be?
• The data being backed up has become very complex,
  and the complexity of backup systems has matched
  that complexity with equally complex
  functionality.

15
Not thinking about disk

• Tape is not as cheap as you thought
• Lets examine a 4 TB library
• 20 slots, 2 drives 17K
• 20 tapes, 70 apiece 14K
• Robotic license 10K
• Total 41K
• (does not include labor costs)
• Thats about 10/GB

16
Disk is cheaper than you thought

• ATA-based storage arrays as low as 5/GB(disk
  only, needs file system)
• Special function arrays
• Quantum DX-30 looks and behaves like a Quantum
  P1000. Can be used as target for tape-based
  backups (3 usable TB, 55K list, or 18/GB)
• NetApp R100 looks like other NetApp filer.
  Target for SnapVault and disk-based backups,
  source for SnapMirror (9 usable TB, 175K list,
  or 18/GB)
• ATA disks not suited for heavy, random access,
  but perfect for large block I/O (e.g. backups!)

17
You can do neat things with disk

• Incremental backups are one of the greatest
  backup performance challenges
• Use as a target for all incremental backups.
  (Full, too, if you can afford it.)
• For offsite storage, duplicate all disk-based
  backups to tape
• Leave disk-based backups on disk

18
Now that I know

• Building a reliable
• and restorable backup system

19
Sizing the backup system
20
Server size/Power

• I/O performance more important than CPU power
• CPU, memory, I/O expandability paramount
• Avoid overbuying by testing prospective server
  under load
• If you use Suns, youve got snoop and truss

21
Catalog/Database size

• Determine number of files (n)
• Determine number of days in cycle (d)
• (A cycle is a full backup and its associated
  incremental backups.)
• Determine daily incremental size (i n .02)
• Determine number of cycles online (c)
• 150-250 bytes per file, per backup
• Use a 1.5 multiplier for growth and error
• Index Size (n (id)) c 250 1.5

22
Library size Drives

• Network backup
• Buy twice as many backup drives as your network
  will support
• Use only as many drives as the network will
  support. (You will get more with less.)
• Use the other half of the drives for duplicating

23
Library size Drives (2)

• Local backup
• Most large servers have enough I/O bandwidth to
  back themselves up within a reasonable time if
  youre using NetBackup
• Usually a simple matter of mathematics
• 8 hr window, 8 TBs 1 TB/hr 277 MB/s
• 30 10 Mb/s drives, 15 20 MB/s drives
• Must have sufficient bandwidth to tape drives
• Filesystem vs. raw recoveries
• Allow drives and time for duplicating

24
Library size Slots (all-tape environment)

• Should hold all onsite tapes
• Onsite tapes automatically expire and get reused
• Only offsite tapes require physical mgmt.
• Should monitor library via a script to ensure
  that each pool has enough free tapes before you
  go home
• Watch for those downed drive messages

25
Library size Slots (disk/tape environment)

• Do incremental backups to disk
• Library needs only to hold onsite full tapes and
  the latest set of copies
• On-site tapes and disk-based backups
  automatically expire and get reused
• Only offsite tapes require physical mgmt.
• Should monitor library and disk via a script to
  ensure that each pool has enough free tapes
  before you go home
• Watch for those downed drive messages

26
Local or remote backup?

• Throughput (in 8hrs), if you own the wire
• 10 Mb 20 GB, 100 Mb 200 GB
• GbE 500 GB 1 TB (Also must own the box.)
• Greater than 500 GB should be local
• LAN-free backups allow you to share a large tape
  library by performing local backups to a
  remote, shared device
• More than one 500 GBserver, buy a SAN!
• Only one 500 GB server, plan for a SAN!
• (NetBackup SSO, NetWorkerDDS)

27
BreakBeverage Service
200 215 PM

• Located in the Grand Ballroom foyer, floor
  2Sponsored by

28
BACKUP/GETTING STARTEDBackup School, Part 2

• W. Curtis PrestonVP, Services DevelopmentGlassHo
  use Technologies

29
Multistreaming NetBackup

• Defined Starting multiple simultaneous backup
  jobs from a single client
• Maximum jobs per client gt 1
• Check Allow multiple data streams.
• ALL_LOCAL_DRIVES, or multiple entries in file
  list
• Maximum jobs per policy gt 1 or unchecked
• Need storage unit with more than one drive, or
  one drive with multiplexing enabled
• Can change max jobs per client using the Server
  Properties -gt Clients tab (4.5)
• By default, will not exceed one job per
  filesystem, but can bypass this if you make your
  own file list

30
Multistreaming (Parallelism) NetWorker

• Use All saveset or multiple entries in the
  saveset list
• Set the parallelism setting for server and, if
  necessary, the storage node
• Set client parallelism value in client attributes
• Must have multiple drives available, or one drive
  with target sessions set higher than one
• Will not exceed number of disks or logical
  volumes on the client (see maximum-sessions in
  manual)

31
Multiplexing NetWorker

• Set target sessions per device, allocating how
  many sessions may be sent to that device
• Global setting for all backups that go to that
  device

32
Multiplexing NetBackup

• Max multiplexing per drive in storage unit
  configuration gt 1
• Media multiplexing in schedule gt 1
• Use higher multiplexing for incremental backups
  if going to tape (6-8)
• Use lower multiplexing for local backups (2)
• No need to multiplex disk storage units
• Multiple policies can multiplex to the same
  drive, but multiple media servers cannot

33
Using Include lists -- not

• NetBackup ALL_LOCAL_DRIVES in file list
• NetWorker All in saveset field
• Automatically excludes NFS/CIFS drives
• Does not include dynamically mounted drives not
  in /etc/fstab

34
What about database clients?

• Use scripts that parse lists of databases
• /var/opt/oracle/oratab for Oracle
• MS-SQL list in registry
• Master database in Sybase
• Some backup products support All for databases
• Remember to write standardize script with
  parameters to backup databases

35
Incremental backups NetBackup

• Create staggered monthly full backups using
  calendar-based scheduling
• Create staggered weekly cumulative incrementals
  using CBS
• Create daily incremental backups using
  frequency-based backups
• (Check Allow after run day)
• Delete window from previous day for CBS

36
Incremental backups NetWorker

• Do not use the Default schedule!
• Create 28 schedules with a monthly full, weekly
  level 1, and daily incremental name them after
  the full day
• Do not specify a schedule for the Group
• Assign the 28 schedules evenly across all clients
  based on size

37
Standardization NetWorker

• Use All saveset entry
• To exclude files, use standard directives for all
  clients

38
Standardization NetBackup

• Use ALL_LOCAL_DRIVES
• Non-Windows clients Use standard exclude list
  and push out from master using bpgp.
• Windows clientsUse standard exclude list and
  push out from master using bpgetconfig M and
  bpsetconfig h.

39
Backup reporting NetBackup

• Watch activity and device monitors
• bperror
• bpdbjobs -report
• bpdbjobs report all_columns
• /usr/openv/netbackup/logs
• /usr/openv/logs
• /usr/openv/volmgr/logs

40
Backup reporting NetWorker

• Watch nwadmin screens
• mminfo
• nsrinfo
• mmlocate
• nsrmm
• /nsr/logs

41
Disk-to-disk backup NetWorker

• If using regular disk, use file type device
• Disk backup extra cost with options
• If using virtual tape library, treat it like a
  tape library
• Use cloning to duplicate disk-based backups to
  tape and send them offsite

42
Disk-to-disk backup NetBackup

• If using regular disk, use disk-based storage
  unit
• (No extra cost for disk storage units!)
• If using virtual tape library, treat it like a
  tape library
• Use vault to duplicate disk-based backups to tape
  and send them offsite

43
What about my SAN and NAS?
44
SAN LAN-free, Client-free, and Server-free
backupNAS NDMP filer to self, filer to filer,
filer to server, server to filer
45
LAN-free backups

• How does this work?
• SCSI Reserve/release
• Third-party queuing system

• Levels of drive sharing
• Restores

46
How client-free backups work
Backup transaction logs to disk
Establish backup mirror
Split backup mirror and back it up
47
How client-free recoveries work
Restore backup mirror from tape
Restore primary mirror from backup mirror
Replay transaction logs from disk
48
Server-free backups

• Server directs client to take a copy-on-write
  snapshot
• Client and server record block and file
  associations
• Server sends XCOPY request to SAN

49
Server-less Restores

• Changing block locations
• Image levelrestores
• File levelrestores

50
NDMP Configurations

• Filer to self
• Filer to filer
• Filer to server
• Server to filer

51
Using NDMP

• Level of functionality depends on the DMA you
  choose
• Robotic Support
• Filer to Library Support
• Filer to Server Support
• Direct access restore support

52
Resources
53
Resources

• Directories of products to help you make a better
  backup system
• http//www.storagemountain.com
• Send questions to
• cpreston_at_glasshouse.com.com

54
ASK THE EXPERTin the Northeast Exhibit Hall

• MONDAY
• 5-6 PM
• TUESDAY
• 5-6 PM

55
Please rate the content of W. Curtis Prestons
talk, Backup School.

1. Poor
2. Fair
3. Good
4. Very Good
5. Excellent

Cross-Tab Label
56
Please rate W. Curtis Prestons presentation as a
speaker.

1. Poor
2. Fair
3. Good
4. Very Good
5. Excellent

Cross-Tab Label