1. Making Reliable and Restorable Backups
Presented by:
W. Curtis Preston
President
The Storage Group, Inc.

2. Making good on your investment
Many SANs are built in order to simplify backup, yet often fail for lack of good design, processes and procedures.
There are several common mistakes that people make when building a backup system
Avoiding these mistakes and taking proper action, can create a backup system that is reliable and restorable

3. What will we cover? Common Backup Configuration Mistakes
How to Avoid Them
Sizing your backup system
Configuration examples for NetBackup
Configuration examples for NetWorker

4. Common Backup Configuration Mistakes

5. Where do these lessons come from?
Audits of real backup and recovery systems
Lessons learned from real horror stories
Many, many sleepless nights

6. Too little power Not enough tape drives
Tape drives that aren’t fast enough
Not enough slots in the tape library
Not enough bandwidth to the server

7. Too much power Streaming tape drives must be streamed
If you don’t, you will wear out your tape drives and decrease aggregate performance
Must match the speed of the pipe to the speed of the tape
You can actually increase your throughput by using fewer tape drives

8. Not using multiplexing
Defined: Sending multiple backup jobs to the same drive simultaneously
Again, drives must be streamed
Multiplexing will impact restore performance, but not as much as you might think
Multiplexing can actually help your restore just as it can help your backups
Using multiplexing can greatly increase the utilization of your backup hardware

9. Not using multistreaming
Defined: Sending multiple simultaneous backup jobs from a single client
Large systems cannot be backed up serially
Multistreaming creates a different job for each filesystem

10. Using include lists
Most major backup software supports file system discovery
Still, many administrators use manually created include lists
Any perceived value is significantly outweighed by the risk it creates

11. Too many full backups
If you are using a commercial backup and recovery product with automated media management and multiple levels, weekly full backups are a waste of tape, time, and money
Monthly full backups, weekly cumulative incrementals (1), and daily incrementals (9) work just as well and use ¼ as much tape
Depending on the level of incremental activity, quarterly backups can work just as well.

12. Not standardizing
Creating custom configurations for each client is easier, but much riskier
Creating a standard backup client configuration can significantly decrease risk
Create a standard exclude list, etc. and push it out to each client

13. Not even noticing!
Backups go ignored so often. It’s like they’re the bill collector nobody wants to talk to
Backup reporting products can really help automate easy reporting
Don’t ignore backups. They will bite you.

14. It’s just backups, right?
“I’m an experienced, seasoned systems administrator. This is just backups. How hard can they be?”
The data being backed up has become very complex, and the complexity of backup systems have matched that complexity with functionality – that also happens to be complex

15. Not thinking about disk
Tape is not as cheap as you thought
Let’s examine a 4 TB library
20 slots, 2 drives $17K
20 tapes, $70 apiece $14K
Robotic license $10K
Total $41K
(does not include labor costs)
That’s about $10/GB

16. Disk is cheaper than you thought
ATA-based storage arrays as low as $5/GB (disk only, needs filesystem)
Special function arrays
Quantum DX-30 looks and behaves like a Quantum P Can be used as target for “tape-based” backups (3 usable TB, $55K list, or $18/GB)
NetApp R100 looks like other NetApp filer. Target for SnapVault and disk-based backups, source for SnapMirror (9+ usable TB, $175K list, or $18/GB)
ATA disks not suited for heavy, random access, but perfect for large block I/O (e.g. backups!)

17. You can do neat things with disk
Incremental backups are one of the greatest backup performance challenges
Use as a target for all incremental backups. (Full, too, if you can afford it)
For off-site storage, duplicate all disk-based backups to tape
Leave disk-based backups on disk

18. Building a reliable and restorable backup system
Now that I know…
Building a reliable
and restorable backup system

19. Sizing the backup system

20. Server Size/Power I/O performance more important than CPU power
CPU, memory, I/O expandability paramount
Avoid overbuying by testing prospective server under load
If you use Suns, you’ve got snoop and truss

21. Catalog/database Size
Determine number of files (n)
Determine number of days in cycle (d)
(A cycle is a full backup and its associated incremental backups.)
Determine daily incremental size (i = n * .02)
Determine number of cycles on-line (c)
bytes per file, per backup
Use a 1.5 multiplier for growth and error
Index Size = (n + (i*d)) * c * 250 * 1.5

22. Library Size - drives Network Backup
Buy twice as many backup drives as your network will support
Use only as many drives as the network will support (You will get more with less.)
Use the other half of the drives for duplicating

23. Library Size - drives Local Backup
Most large servers have enough I/O bandwidth to back themselves up within a reasonable time if you’re using NetBackup
Usually a simple matter of mathematics:
8 hr window, 8 TBs = 1 TB/hr = 277 MB/s
30 10 Mb/s drives, MB/s drives
Must have sufficient bandwidth to tape drives
Filesystem vs. raw recoveries
Allow drives and time for duplicating!

24. Library Size - slots (all tape environment)
Should hold all onsite tapes
On-site tapes automatically expire and get reused
Only offsite tapes require phys. mgmt.
Should monitor library via a script to ensure that each pool has enough free tapes before you go home
Watch for those downed drive messages

25. Library Size - slots (disk/tape environment)
Do incremental backups to disk
Library only needs to hold on-site full tapes and the latest set of copies.
On-site tapes and disk-based backups automatically expire and get reused
Only offsite tapes require phys. mgmt.
Should monitor library and disk via a script to ensure that each pool has enough free tapes before you go home
Watch for those downed drive messages

26. Local or Remote Backup? (NetBackup= SSO, NetWorker=DDS)
Throughput (in 8hrs), if you “own the wire:”
10 Mb = 20 GB, 100 Mb = 200 GB
GbE = 500 GB – 1 TB (Also must “own the box.”)
Greater than 500 GB should be “local”
Lan-free backups allow you to share a large tape library by performing “local” backups to a “remote, shared” device
More than one 500+ GBserver, buy a SAN!
Only one 500+ GB server, plan for a SAN!
(NetBackup= SSO, NetWorker=DDS)

27. Multistreaming - NetBackup
Defined: Starting multiple simultaneous backup jobs from a single client
Maximum jobs per client > 1
Check “Allow multiple data streams”
ALL_LOCAL_DRIVES, or multiple entries in file list
Maximum jobs per policy > 1 or unchecked
Need storage unit with more than one drive, or one drive with multiplexing enabled
Can change max jobs per client using the Server Properties -> Clients tab (4.5)
By default, will not exceed one job per filesystem, but can bypass this if you make your own file list

28. Multistreaming (Parallelism) - NetWorker
Use “All” saveset or multiple entries in the saveset list
Set the parallelism setting for server and, if necessary, the storage node
Set client parallelism value in client attributes
Must have multiple drives available, or one drive with target sessions set higher than one
Will not exceed number of disks or logical volumes on the client (see maximum-sessions in manual)

29. Multiplexing – NetWorker
Set target sessions per device, allocating how many sessions may be sent to that device.
Global setting for all backups that go to that device

30. Multiplexing - NetBackup
Max multiplexing per drive in storage unit configuration > 1
Media multiplexing in schedule > 1
Use higher multiplexing for incremental backups if going to tape (6-8)
Use lower multiplexing for local backups (2)
No need to multiplex disk storage units
Multiple policies can multiplex to the same drive,
but multiple media servers cannot

31. Using Include lists -- not
NetBackup – ALL_LOCAL_DRIVES in file list
NetWorker – All in saveset field
Automatically excludes NFS/CIFS drives
Does not include dynamically mounted drives not in /etc/*fstab

32. What about database clients?
Use scripts that parse lists of databases:
/var/opt/oracle/oratab for Oracle
MS-SQL list in registry
Master database in Sybase
Some backup products support “All” for databases
Remember to write standardize script with parameters to backup databases.

33. Incremental backups - NetBackup
Create staggered monthly full backups using calendar-based scheduling
Create staggered weekly cumulative incrementals using CBS
Create daily incremental backups using frequency based backups
(Check Allow after run day.)
Delete window from previous day for CBS

34. Incremental backups - NetWorker
Do not use the Default schedule!
Create 28 schedules with a monthly full, weekly level 1, and daily incremental, name them after the full day
Do not specify a schedule for the Group
Assign the 28 schedules evenly across all clients based on size

35. Standardization – NetWorker
Use All saveset entry
To exclude files, use standard directives for all clients

36. Standardization - NetBackup
Use ALL_LOCAL_DRIVES
Non-Windows clients - Use standard exclude list and push out from master using bpgp
Windows clients – Use standard exclude list and push out from master using bpgetconfig –M and bpsetconfig –h

37. Backup Reporting - NetBackup
Watch activity and device monitors
bperror
bpdbjobs -report
bpdbjobs –report –all_columns
/usr/openv/netbackup/logs
/usr/openv/logs
/usr/openv/volmgr/logs

38. Backup Reporting – NetWorker
Watch nwadmin screens
mminfo
nsrinfo
mmlocate
nsrmm
/nsr/logs

39. Disk-to-disk Backup - NetWorker
If using regular disk, use file type device
Disk backup extra cost with options
If using virtual tape library, treat it like a tape library
Use cloning to duplicate disk-based backups to tape and send them off-site

40. Disk-to-disk Backup - NetBackup
If using regular disk, use disk-based storage unit
(No extra cost for disk storage units!)
If using virtual tape library, treat it like a tape library
Use vault to duplicate disk-based backups to tape and send them off-site

41. What about my SAN and NAS?

42. SAN: LAN-free, Client-free, and Server-free backup NAS: NDMP filer to self, filer to filer, filer to server, & server to filer

43. LAN-free backups How does this work? Levels of drive sharing Restores
SCSI Reserve/Release
Third-party queuing system
Levels of drive sharing
Restores

44. How client-free backups work
Backup transaction logs to disk
Establish backup mirror
Split backup mirror and back it up

45. How client-free recoveries work
Restore backup mirror from tape
Restore primary mirror from backup mirror
Replay transaction logs from disk

46. Server-free backups
Server directs client to take a copy-on-write snapshot
Client and server record block and file associations
Server sends XCOPY request to SAN

47. Server-less Restores Changing block locations Image level restores
File level restores

48. NDMP Configurations Filer to self Filer to filer Filer to server
Server to filer

49. Using NDMP Level of functionality depends on the DMA you choose
Robotic Support
Filer to Library Support
Filer to Server Support
Direct access restore support

50. Resources

51. Resources
Directories of products to help you make a better backup system
Send questions to: