CIT 470: Advanced Network and System AdministrationSlide #1 CIT 470: Advanced Network and System Administration Disaster Recovery

CIT 470: Advanced Network and System AdministrationSlide #2 Topics 1.Planning 2.Disasters 3.Mitigation

CIT 470: Advanced Network and System AdministrationSlide #3 What is a Disaster Recovery Plan? 1.Considers potential disasters. 2.Describes how to migitate potential disasters. 3.Makes preparations to enable quick restoration of services. 4.Identifies key services and how quickly they need to be restored and in what order.

CIT 470: Advanced Network and System AdministrationSlide #4 Disaster Recovery Plans 1.Define (un)acceptable loss. How much could you lose in a disaster? 2.Back up everything. Backup data, metadata, and instructions on how to restore your system. 3.Organize everything. Can you find the backup tapes you need when disaster strikes?

CIT 470: Advanced Network and System AdministrationSlide #5 Disaster Recovery Plans 4.Protect against disasters. Natural disasters and many more. 5.Document what you have done. Plan must be detailed enough for people to follow in a disaster w/o additional info. 6.Test, test, test. A disaster recovery plan that has not been tested is not a plan; it's a proposal.

CIT 470: Advanced Network and System AdministrationSlide #6 Define loss Loss of service How much employee productivity lost? How much customer revenue lost? Loss of data Irreplaceable data Medical image records Stock purchases Re-creatable data. At what cost? Code for a software product Simulation results

CIT 470: Advanced Network and System AdministrationSlide #7 Backup Everything On a system Project directories Home directories System files (fstab, kernel, passwd, LVM config, MBR) Types of systems Laptops Plugin to network and backup to backup server on command. Desktops Store everything on network disks. Servers Permanent connection to backup system.

CIT 470: Advanced Network and System AdministrationSlide #8 Organize Everything What resources do you back up? On what schedule? Media organization Bar code labels on each tape. Stored securely at proper temp/humidity. Media database Maps servers/drives to tapes and their locations. Indicates whether tapes are on- or off-site. Must be backed up w/ humanly-readable label.

CIT 470: Advanced Network and System AdministrationSlide #9 Protect against Disasters On-site vaults. Off-site storage. Test your media regularly. Store documentation securely too.

CIT 470: Advanced Network and System AdministrationSlide #10 Document Store documentation in portable format. Ensure documentation accessible in disaster. Paper copies on and off-site.

CIT 470: Advanced Network and System AdministrationSlide #11 Test Can other people understand procedures? Sample test tapes on regular (weekly) basis. Attempt a full system recovery 2/year.

CIT 470: Advanced Network and System AdministrationSlide #12 What is a Disaster? A catastrophic event that causes loss of data and/or service. Human disasters Errors or intentional. Typo, backhoe, or hacker tools. Natural disasters Small scale: Hardware or power failure. Large scale: Hurricane, earthquake, fire.

CIT 470: Advanced Network and System AdministrationSlide #13 Types of Disasters User errors Accidental file deletion / overwrite. Very common. Snapshots can automate. Sysadmin errors Accidental mass file destruction. Regular backups will prevent loss. Drive failure Single disk failure: RAID can prevent loss. System failure Loss of an entire system. RAID won’t help. Need backups.

CIT 470: Advanced Network and System AdministrationSlide #14 Types of Disasters Power/Network Failure Need UPS/generator or redundant network connection. Software Failure Software corrupts its own or other apps data store. Need regular and perhaps historical backups. Security Breach An attacker / worm destroys/corrupts data. Need long-term historical backups. Natural Disaster Potential loss of entire data center, including backups. Need off-site backups to restore data. Need off-site (virtual) data center to restore service.

CIT 470: Advanced Network and System AdministrationSlide #15 Risk Analysis Evaluate risk cost of disaster Cost * Probability Determines budget for disaster mitigation. Ex: power failure 70% chance per year Average downtime: 4 hours Average web site revenue / hour: $1000 Budget = 4 hrs * (1000 $/hr) * 0.7/yr = $2800/yr

CIT 470: Advanced Network and System AdministrationSlide #16 Disaster Mitigation Power Failures UPS Generator System Failures Redundancy: CPU, ECC RAM, NICs, power Cluster of servers Network Failures Multiple internet connections f/ diff ISPs.

CIT 470: Advanced Network and System AdministrationSlide #17 Disaster Mitigation Drive Failures RAID Backups Accidental Deletion Snapshots Backups Security Incident Backups

CIT 470: Advanced Network and System AdministrationSlide #18 Redundant Site Redundant site at a different location –Location far enough away to be unaffected by whatever disaster took down primary site. –Automatic or manual switchover. DNS names with short experimation times. Cheaper solution: use existing second site –Duplicate critical services at both data centers. –Rebuild less critical servers at second site.

CIT 470: Advanced Network and System AdministrationSlide #19 References 1.Aeleen Frisch, Essential System Administration, 3 rd edition, O’Reilly, Evi Nemeth et al, UNIX System Administration Handbook, 3 rd edition, Prentice Hall, Thomas A. Limoncelli and Christine Hogan, The Practice of System and Network Administration, 2 nd edition, Addison-Wesley, W. Curtis Preston, UNIX Backup & Recovery, O’Reilly, 1999.