1. Disaster Recovery Technical Infrastructure at George Mason University
January 13, 2011
Ron Secrest
Mark Craft
Barry Freese

2. Copyright Ron Secrest, Barry Freese and Mark Craft, 2011.
This work is the intellectual property of the authors. Permission
is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

3. Historical DR Strategies
“Sister Site”
Lack of excess capacity
Configuration issues
Rapid Replacement
Hardware availability?
Unable to test the process
DR Service Center Concept
Costly
Configuration management issues

4. The Basics: Risk Mitigation Techniques
24 X 7 Vendor Support for Production Systems
RAID Disk Subsystems
System Backups (on-site & off-site)
Redundant Servers
UPS and Emergency Generator
Smoke Detection & Fire Suppression System

5. DR System Objectives Off-Site University DR Facility Phased deployment
Cost effective, scalable, ACTIVE-PASSIVE environment
Support mission critical systems, applications and services
Recovery Time Objective (RTO): 4 hours
Recovery Point Objective (RPO): 5 minutes

6. Features of the DR System
Dedicated gigabit network connectivity
Replication of mission critical databases, files, and systems
Redundancy
Remote monitoring
Remote systems management
Essentially a “lights-out” operation
On-site backups: Data Domain system
Active Services: DNS, DHCP, LDAP, AD, Kerberos

7. APC Rack Infrastructure

8. Replication Strategies
In-line SAN Replication
Pros
No vendor lock-in
Similar hardware not required; JBOD could be deployed remotely.
Cons
Not integrated with hardware (software solution on 3rd party hardware). Failover is problematic.
More technology to maintain
Additional vendor for support issues (e.g. problems, firmware/OS driver code levels).

9. In-line SAN Replication

10. Replication Strategies
Native SAN Replication
Pros
Integrated with storage hardware
Less technology to manage
Cons
Vendor lock-in
Often requires similar hardware local and remotely

11. Native SAN Replication

12. Current Data Replication Methodology
Dedicated, private, (2) 1 Gb/sec network links between the Fairfax campus data center and the DR facility at the Prince William campus
3PAR SAN Remote Copy
Asynchronous
Block Level
Data changes are replicated every 5 minutes
Oracle Database Replication as a backup (Oracle standby)
Backup replication: Data Domain system
rsync (Linux) (Samba file server, Web server content)
Active Directory & LDAP synchronization
Kerberos

13. DR Technologies VMware Site Recovery Manager (SRM)
SAN-to-SAN Asynchronous Replication
Native Replication Applications (e.g. LDAP, Active Directory, Kerberos, DNS)
DNS switchover

14. VMware Site Recovery Manager

15. VMware Site Recovery Manager
Pros
Automated failover
Non-interruptive testing
Summary report of testing results
Cons
Additional VMware licensing required
Falling back requires a new recovery plan to be created.
Possible IP/DNS changes needed for Virtual Machines on the recovery side.

16. Testing Methods Phase Testing Testing is cumulative
Fine tuning after technology changes
Active Services (e.g. Active Directory): Extended Testing
VMware SRM failover (anytime including during business hours)

17. DR Failover Process Shutdown local databases (if possible).
Disable data replication from the local Storage Area Network (SAN) to the remote DR SAN.
Initiate VMware Site Recovery via Site Recovery Manager (SRM).
After the SRM failover is finished initiate network routing changes.
Present DR SAN storage to the remote DR servers (SRM automatically provides this function for VMware servers).
Promote DR Active Directory domain controller.
Check status of DR LDAP, DR Kerberos and DR Active Directory.
Run scripts to switch DNS for servers or update DNS manually if necessary. This includes the emergency static web page. NOTE: DNS changes for all virtual hosts for the server will be necessary to provide a robust web environment if deemed necessary.

18. DR Site Documentation Kindle DR PDF Document Repository
Allows local access to documents at a remote site even if the servers are headless (USB port on Solaris server).
No additional infrastructure necessary to view the PDF documents
No maintenance necessary
Auto charges / auto mounts allowing remote access for updates without operational support.
Secure: only way to access documents is through the secure server
Efficient: first person on site can unplug and view procedures immediately.

19. Lessons Learned
DR funding should be included in the budget for new IT projects
Phased deployment is the right approach but be prepared for refreshing the technical infrastructure
Active-Passive is less complicated and more cost effective than an Active-Active architecture but fall-back time is significant

20. Summary of Benefits of Mason’s DR Program
Helps ensure business continuity
Highly scalable architecture
Allows for an incremental rollout
Proven technology
Ease of acquisition
Provides a framework for meeting state and internal audit requirements
Due Diligence: A “real” Disaster Recovery solution

21. Questions?
Ron Secrest: Basic system architecture, acquisition, DR program management and testing
Barry Freese: System and storage architecture/management, replication and LDAP
Mark Craft: VMware, Site Recovery Manager and Active Directory

22. Mason Contact Information
Ron Secrest
Director, Enterprise Servers & Messaging
Mark Craft
Senior Technology Advisor
Barry Freese
Senior Computer Systems Engineer/System Architect

23. Thanks for attending!