Backups Chapter 10

Introduction At most sites, the information stored on computers is worth more than the computers themselves. It is also much harder to replace. Protecting this information is one of the system administrator’s most important tasks and, unfortunately, most tedious Chapter 10 - Backups

Introduction If executed correctly, backups allow administrators to restore filesystems (or any part of a filesystem) to the condition it was in at the time of the last backup. Backups must be done carefully and on a strict schedule The backup system and backup media must be tested regularly to verify that they are working correctly. Chapter 10 - Backups

Introduction Chapter Overview We begin this chapter with some general backup philosophy, Followed by a discussion of the most commonly used backup devices and media Their strengths, weaknesses, and costs Next, we discuss the standard UNIX backup and archiving commands and give some suggestions as to which commands are best for which situations Chapter 10 - Backups

Introduction We then talk about how to design a backup scheme and review the mechanics of the UNIX commands dump and restore. Finally we take a look at Amanda, a free network backup package and offer some comments about its commercial alternatives Chapter 10 - Backups

1. Motherhood and apple pie Before we get into the meat and potatoes of backups, we want to pass on some general hints that we have learned over time (usually the hard way) None of these suggestions is an absolute rule, but you will find that the more of them you follow, the smoother your dump process will be. Chapter 10 - Backups

1. Motherhood and apple pie Perform all dumps from one machine rdump allows you to perform dumps over the network. Although there is some performance penalty for doing this, the ease of administration makes it worthwhile. We have found the best method is to run a script from a central location that executes rdump (via rsh or ssh) on each machine that needs to be dumped Or use a software package that automates this process. Chapter 10 - Backups

1. Motherhood and apple pie All dumps should go to the same backup device If your network is too large to be backed up by a single tape drive, you should try to keep your backup system as centralized as possible. Centralization makes administration easier and allows you to verify that all machines were dumped correctly. Chapter 10 - Backups

1. Motherhood and apple pie Label your tapes It is essential that you label each dump tape clearly and completely. An unlabeled tape is a scratch tape. Detailed information such as lists of filesystems and dump dates can be written on the cases. Chapter 10 - Backups

1. Motherhood and apple pie You must be able to restore root and /usr filesystems without looking at dump scripts label the dump tapes with their format, the exact syntax of the dump command used to create them, and any other information you would need to restore from them without referring to on-line documentation Free and commercial labeling programs abound. Save yourself a major headache and invest in one. Chapter 10 - Backups

1. Motherhood and apple pie Pick a reasonable backup interval The more often backups are done, the smaller amount of data that can be lost in a crash. However, backups use a systems resources and an operator’s time. On busy systems it is generally appropriate to back up filesystems with home directories every workday. The question is: How much data are your users willing to lose? Chapter 10 - Backups

1. Motherhood and apple pie Choose filesystems carefully Filesystems that are rarely modified do not need to be backed up as frequently as user’s home directories. If only a few files change on an otherwise static filesystem (such as /etc/passwd in the root filesystem), these files can be copied every day to another partition that is backed up regularly. Chapter 10 - Backups

1. Motherhood and apple pie Make daily dumps fit on one tape In a perfect world, you could do daily dumps of all your filesystems onto a single tape. You can mount a tape every day before you leave work and run the dump later at night from cron. This way, dumps occur at a time when files are not likely to be changing, and the dumps have minimal impact on users. Chapter 10 - Backups

1. Motherhood and apple pie If you can’t fit your daily backups on tape, you have several options: Buy a higher capacity tape device. Buy a stacker or library and feed multiple pieces of media to one device Change your dump sequence Write a smarter script Use multiple backup devices. Chapter 10 - Backups

1. Motherhood and apple pie Make filesystems smaller than your dump device dump is perfectly capable of dumping filesystems to multiple tapes. But if a dump spans multiple tapes, an operator must be present to change tapes. Unless you have a good reason to create a really large filesystem, don’t do it. Chapter 10 - Backups

1. Motherhood and apple pie Keep tapes off-site Most organizations keep backups off-site so that a disaster such as fire cannot destroy both the original data and the backups. The speed with which tapes are moved off-site should depend on how often you need to restore files and how much latency you can accept. Chapter 10 - Backups

1. Motherhood and apple pie Protect your backups “What does a backup do? It reliably violates file permissions at a distance.” Not only should you keep your tapes off-site, but you should also keep them under lock and key Some companies feel so strongly about the importance of backups that they make duplicates. Chapter 10 - Backups

1. Motherhood and apple pie Limit activity during dumps Filesystem activity should be limited during dumps because changes can cause dump to make mistakes. You can limit activity by doing the dumps when few active users are around (at night or weekend) or by making the filesystem accessible only to dump. Most users will not like this option. They want 24/7 access. Chapter 10 - Backups

1. Motherhood and apple pie Check your tapes There are many horror stories about system administrators that did not discover problems with their dump regime until after a serious system failure. Checks: Have your dump software attempt to reread the tapes immediately after it has finished dumping. It is often useful to run restore t to generate a table of contents for each filesystem and store the results to disk. Chapter 10 - Backups

1. Motherhood and apple pie Develop a tape life cycle Tapes have a finite life cycle. It is great to recycle your media. Most manufacturers quantify their recommended life in terms of the number of passes that a tape can stand. A backup, a restore, and an mt fsf (file skip forward) each represent one pass. Chapter 10 - Backups

1. Motherhood and apple pie Prepare for the worst After you have established a backup procedure, explore the worst case scenario: your site is completely destroyed. Determine how much data would be lost and how long it would take to get your system back to life. Include the time it would take to get new hardware. Then determine if you can live with your answers. Chapter 10 - Backups

2. Backup devices and media Since many types of failure can damage several pieces of hardware at once, backups should be written to some sort of removable media Mirroring is a great form of backup, but if the controller dies, then what…. Many kinds of media store data using magnetic particles. These are subject to damage by electrical and magnetic fields. Chapter 10 - Backups

2. Backup devices and media Here are some specific hazards to avoid: Audio speakers contain large electromagnets Transformers and power supplies including UPS boxes Monitors use transformers and high voltages. Many monitors retain an electrical charge even after being turned off. Color monitors are the worst. Long Shelf life All tapes will become unreadable over a period of years. Most media will keep for three years. If you need longer than that, use optical media or re-record the data. Chapter 10 - Backups

2. Backup devices and media The following sub-sections describe some of the media that can be used for backups. Although cost and media capacity are both important considerations, it is important to consider throughput as well. Fast media are more pleasant to deal with, and they allow more flexibility in the scheduling of dumps. Chapter 10 - Backups

2. Backup devices and media Floppy disks The most inconvenient way to store backups. They are slow and do not hold much data. Although the individual disks are cheap, they hold so little data that they are in fact the most expensive backup medium overall. Chapter 10 - Backups

2. Backup devices and media Super floppies Iomega Zip drives 100MB to 250MB SCSI, USB, Parallel, and Serial interfaces Imation Drives can read and write normal floppies and 120MB media Although these products are useful for exchanging data, their high media costs make them a poor choice for backups. Chapter 10 - Backups

2. Backup devices and media CD-R and CD-RW Price changes make these a far more attractive medium than they were a few years ago. CD-R’s are not quite as durable as stamped CDs. They are good for archiving, but not a good choice for backup media. Recordable DVD technology, with capacity around 10GB, will change how this field is viewed. Chapter 10 - Backups

2. Backup devices and media Removable hard disks The removable drive market is becoming very competitive, and prices vary daily. Lifetime is 5-10 years The main advantage of these products is speed (close to normal drives) They are attractive as backup devices for small systems and home machines, although the disks themselves are somewhat pricey Chapter 10 - Backups

2. Backup devices and media 8mm cartridge tapes Small format videotapes - also called Exabyte drives The original format held 2GB, and newer formats hold up to 7GB. Hardware compression build into the drives will push this even higher. The size of the tapes makes off-site storage convenient. Susceptible to heat damage. Chapter 10 - Backups

2. Backup devices and media DAT (4mm) cartridge tapes Digital Audi Tape drives - or Digital Data Storage (DDS) Drives Original format was 2GB, DDS-4 holds up to 20GB DDS-4 drives seek rapidly and transfer data at up to about 2.5MB/s, making the drives relatively fast. They also do not have a history of alignment problems. Chapter 10 - Backups

2. Backup devices and media Travan tapes The next generation of QIC tape technology Travan drives use a linear recording technology and support media from 2.5GB to 10GB Tapes cost about $3/GB (slightly more than others) Marketing hype claims 1MB/s transfer rate. UNIX drive support is sketchy. Chapter 10 - Backups

2. Backup devices and media OnStream ADR A relative newcomer to backups. Based upon Linear recording technology Supports media sizes of 15GB and 25GB Drives are cheap and fast Chapter 10 - Backups

2. Backup devices and media DLT Popular backup device reliable, affordable, and hold large amounts of data. Tapes hold up to 40GB Transfer rates at 6 MB/s Media is expensive ($65 per tape) Manufacturers boast that tapes will last 20 to 30 years (will the drives still be running then?) Chapter 10 - Backups

2. Backup devices and media AIT Sony’s 8mm product on steroids AIT-2 claims 6 MB/s native transfer rate and media capacity of 50GB Tapes Long life cycle Contain an EEPROM to give the media some smarts. Chapter 10 - Backups

2. Backup devices and media Mammoth Produced by Exabyte - after falling out with Sony Mammoth-2 drives offer blindingly fast 12MB/s native transfer rate far faster than other tape drives in this price range. Chapter 10 - Backups

2. Backup devices and media Jukeboxes, stackers, and tape libraries With the low cost of disks these days, most sites have so much disk space that a full backup requires multiple tapes, even at 20GB per tape. One solution for these sites is a stacker, jukebox, or tape library. Chapter 10 - Backups

2. Backup devices and media Hard disks We would be remiss if we did not mention the decreasing cost of hard drives as a reason to consider disk to disk backups. Although we suggest that you not duplicate one disk to another within the same physical machine, hard disks can be a good low-cost solution for storage over a network. Chapter 10 - Backups

2. Backup devices and media Summary of media types Chapter 10 - Backups

2. Backup devices and media What to buy DAT and Exabyte drives are excellent solutions for small workgroups DLT, AIT, and Mammoth-2 are all roughly comparable and target University and Corporate environments. Chapter 10 - Backups

3. Setting up an incremental backup regime The dump and restore commands are the most common way to create and restore from backups. These programs have been part of UNIX for a very long time, and their behavior is well known. At most sites, dump and restore are the underlying commands used by automated backup software. Chapter 10 - Backups

3. Setting up an incremental backup regime Dumping filesystems The dump command builds a list of files that have been modified since a previous dump, then packs those files into a single large file to archive on an external device. dump has several advantages over other utilities described later in this chapter Backups can span multiple tapes Files of any types (even devices) can be backed up and restored Permissions, ownerships, and modification times are preserved. Files containing holes are handled correctly Backups can be performed incrementally (with only recently modified files being written to tape) Chapter 10 - Backups

3. Setting up an incremental backup regime dump understands the layout of raw filesystems, and it reads the filesystem’s inode tables directly to decide which files must be backed up This allows dump to be very efficient It also imposes a few limitations: Every filesystem must be dumped individually Only filesystems on the local machine can be dumped you can overcome this with rdump Chapter 10 - Backups

3. Setting up an incremental backup regime The most important feature of dump is its support for the concept of an “incremental” backup. Incremental dumps make it possible to back up only filesystems that have changed since the last backup. When you do a dump, you assign it a backup level (0-9). A level 0 backup places the entire filesystem on the tape. A level N dump backs up all files that have changed since the last dump of less than N Chapter 10 - Backups

3. Setting up an incremental backup regime The first argument to dump must be the incremental dump level. dump uses the /etc/dumpdates file to determine how far back to go. The u flag causes dump to automatically update /etc/dumpdates when the dump completes. Why should it be after it completes? Chapter 10 - Backups

3. Setting up an incremental backup regime Dump sequences Because dump levels are arbitrary, dumps can be performed on various schedules. The schedule that is right for you depends on: The activity of your filesystem The capacity of your dump device The amount of redundancy you want. The number of tapes you want to buy. Chapter 10 - Backups

3. Setting up an incremental backup regime A simple schedule If your total disk space is smaller than the capacity of your tape device, you can use a trivial dump schedule Do level zero dumps of every filesystem each day. Reuse a group of tapes, but every N days, keep the tape forever Don’t reuse the exact same tape for every night’s dump. It is better to rotate among a set of tapes so that even if one night’s dump is blown, you can still fall back to the previous night. Chapter 10 - Backups

3. Setting up an incremental backup regime A moderate schedule Assign a tape to each day of the week, and each month of the year Every day do a level 9 dump to the daily tape. Every week do a level 5 dump to the weekly tape. And every month do a level 3 dump to the monthly tape. Do a level 0 at least once per year. Chapter 10 - Backups

4. Restoring from dumps Most of the many variations of the program that extracts data from dump tapes are called restore. We will first discuss restoring individual files (or a small set of files), then explain how to restore entire filesystems. Chapter 10 - Backups

4. Restoring from dumps Restoring individual files The first step to take when you are notified of a lost file is to determine which tapes contain versions of the file. Users often want the most recent version of a file, but that is not always the case. It is helpful if you can browbeat the users into telling you not only what files are missing, but also when they were lost and when they were last modified Chapter 10 - Backups

4. Restoring from dumps If you do not keep on-line catalogs, you must mount tapes and repeatedly attempt to restore the missing files until you find the correct tape. If the user remembers when the files were last changed, you may be able to make an educated guess about which tape the files might be on. Chapter 10 - Backups

4. Restoring from dumps After determining which tapes you want to extract from, create and cd into a temporary directory (/var/restore) where a large hierarchy can be created Most versions of restore must create all of the directories leading to a particular file before it can be restored. Don’t use /tmp Your work could be wiped out if the machine crashes and reboots before the restored data has been moved to its original location. Chapter 10 - Backups

4. Restoring from dumps restore has many command options. i – for interactive restores of individual files and directories r – for a complete restore of a filesystem x – for non interactive restores of specified files In general, restore i is the easiest way to restore a few files or directories from a tape. However, it will not work if the tape does device cannot be moved backward a record at a time (a problem with some 8mm tape drives) Chapter 10 - Backups

4. Restoring from dumps Restoring entire filesystems With luck you will never have to restore an entire filesystem after a system failure. Before you begin a full restore, you must create and mount the target filesystem (see Chapter 8) cd to the mount point, Put the first tape of the most recent level 0 dump in the drive Type restore r Chapter 10 - Backups

5. Dumping and restoring for upgrades When you perform an OS upgrade, you must back up all filesystems with a level 0 dump and, possibly, restore them. The restore is needed only if the new OS uses a different filesystem format, or if you are changing the partitioning of your disk. A complete set of backups also gives you the option of reinstalling the old OS if the new version does not prove satisfactory. You should also do a level 0 after the install. System file time stamps may have the creation date, which will mess up dump’s incremental backups Chapter 10 - Backups

6. Using other archiving programs dump is not the only program you can use to archive files to tapes; however, it is usually the most efficient way to back up an entire system. tar, cpio, and dd can also move files from one medium to another Chapter 10 - Backups

6. Using other archiving programs tar: package files tar reads multiple files or directories and packages them into one file, often a tape file. tar is a useful way to back up any files whose near-term recovery you anticipate. tar is also useful for moving directory trees from place to place. Chapter 10 - Backups

6. Using other archiving programs cpio: Sys V-ish archiving cpio is similar to tar in functionality. It dates from the beginning of time and is rarely used today. When using cpio, read your man pages carefully; the options vary greatly among systems. Chapter 10 - Backups

6. Using other archiving programs dd: twiddle bits dd is a file copying and conversion program. Unless it is told to do some sort of conversion, dd just copies from its input to its output One historical use for dd was to create a copy of an entire filesystem. A better way is to use newfs, dump, and restore. Chapter 10 - Backups

6. Using other archiving programs volcopy: duplicate filesystems volcopy makes an exact copy of a filesystem on another device. It is available on Solaris, HP-UX, and Linux systems. You can use the volcopy command to back up a filesystem to a removable disk or to make a complete copy of a filesystem on tape. Chapter 10 - Backups

7. Using multiple files on a single tape In reality, magnetic tape contains one long string of data. However, it’s often useful to store more than one “thing” on a tape, so tape drives and their UNIX drivers conspire to provide you with a bit more structure. When dump or some other command writes a stream of bytes out to a tape device then closes the file, an “end of file” marker is automatically placed on the tape. Chapter 10 - Backups

7. Using multiple files on a single tape You can use the mt command to position a tape at a particular stream or “file set” as mt calls them. mt is especially useful if you put multiple files (for example multiple dumps) on a single tape. It also has some of the most interesting error messages of any UNIX utility. Consult your manual for an exact list of commands supported by mt. Chapter 10 - Backups

8. Amanda The Advanced Maryland Automatic Network Disk Archiver, Amanda, is a sophisticated network backup system that can replace the home-grown scripts used by many sites. It supports multiple flavors of UNIX and many different kinds of backup media. The latest information and source code are available from www.amanda.org Chapter 10 - Backups

9. Commercial backup products There are a lot of other commercial products available. ADSM/TSM Veritas Legato For a lot more information check out the web site www.backupcentral.com, or the book on the next page Chapter 10 - Backups

10. Recommended Reading Unix Backup and Recovery By Curtis W. Preston O’Reilly 1999 Chapter 10 - Backups

Chapter 10 - Backups