1. Disaster Recovery at UNC
Brad Sharp, Manager of Technology Operations
Department of Information Management & Technology

2. Agenda Infrastructure Overview UNC Disaster Recovery Planning Process
UNC E-For3t Presence
Next Steps / Challenges

3. UNC Infrastructure Overview
Microsoft Windows Server Environment
About 98% Virtualized with VMware
Currently Running on HP C7000 Blade Servers
HP EVA Storage Arrays
4 Data Centers
Presence at E-For3t Facility

4. UNC’s DR Planning process

5. UNC’s DR Planning Scope
Although DR planning is an institutional issue, the purpose of this group is to focus on Information Technology and the potential causes and remediation of disasters as they pertain to the technical services delivered internal and external to our campus.

6. Potential Types of Disasters
Fire
Flood
Tornado and High Winds
Earthquake
Computer Crime
Terroristic Actions and Sabotage

7. The Disaster Recovery Plan Project
The Disaster Recovery Plan Project took our organization through the complete Disaster Recovery Planning process:
Business Impact Analysis
System Documentation
Recovery Planning

8. Business Impact Analysis
Eliminates the risk of over spending for a DRP that doesn’t match the potential risk.
Gives management a tool to adequately determine an appropriate plan.

9. Documentation of Systems
Gather information required to recover on all hardware, software, networks, communications, service and resources.
Contact lists, service agreements and vendor support.
Specific instructions for rebuilding.

10. Recovery Planning Determine tolerance for downtime.
Prioritize recovery.
Build recovery teams.
Define hot / cold site requirements. State of Colorado E-Fort
Define responsibilities for each team.
Steps required to test and accept the plan.

11. Security and the DRP are Linked
Security vulnerabilities are potentially your highest risk for disaster. Our DRP focuses on mitigating the risk and enabling recovery. A comprehensive evaluation of our readiness is essential to our survival.

12. UNC’s DR PLAN

13. UNC’s Disaster Recovery Plan
What is the Disaster Recovery Plan?
A 55 page document that provides consolidated core information needed to drive the recovery of disrupted services maintained by the Information Management &Technology Department of the University of Northern Colorado. Disrupted services can range from a simple application failure to the destruction of facilities. In addition some cases the disruptions can result from external services such as local area power or communications services. The contents of this document, referenced documentation, and the external e-FOR3T recovery site should provide the university with the ability to recover and maintain support services to the university within a tolerable disruption.

14. Disaster Recovery History
November 2007 – Document Started
February 2008 – First Draft Completed
June 2008 – First Table Top Exercise
July 2008 – First Release
July 2009 – Annual Table Top Exercise / Review / Annual Security Risk Assessment
July 2010 – Annual Table Top Exercise / Review / Annual Security Risk Assessment
July 2011 – Annual Table Top Exercise / Review / Annual Security Risk Assessment
July 2012 – Annual Table Top Exercise / Review / Annual Security Risk Assessment

15. UNC’s DR Process
Event Identification / Notification – An event (on any scale) is observed and appropriate notifications are made.
Repair / Recover Efforts are Determined – Based upon the event impact and severity, priorities are established and recovery efforts are determined.
Recovery Teams Assembly – Based upon the recovery effort needed, Recovery Teams are notified. Assignments are made to recover failing components based on priorities.
Asset Acquisition – Where physical damage occurs to critical components, the appropriate recovery teams will acquire the needed assets.
Recovery is completed – As recovery of failing components is completed, access by end users is made available.
Reestablishment – When the relocation of services is required to e-For3t, services will be reestablished at the designated production location.
Review and Adjustment – After complete recovery, the process should be reviewed and analyzed. Adjustments in all procedures should be updated or modified as needed.

16. DR Teams & Assignments IM&T Staff Hardware & OS Recovery Team
Review and be familiar with the complete Disaster Recovery Plan
Verify personal “IT Contact” information
IM&T Staff
Determine replacement hardware, provide specifications, and coordinate with Finance and Management Teams to acquire hardware components needed for recovery
Hardware & OS Recovery Team
Determine “Point-In-Time” at which data can be recovered from backups. Perform data restore, transaction roll-forward as needed. Coordinate with Hardware and Network Recovery Teams
Data Recovery Team
Determine status of network structure including Network hardware, software and configurations. Coordinate with Hardware Recovery Team and External Services Recovery Team.
Network Recovery Team
Determine status of external services. Interface with Public Utilities and Internet Providers.
External Services Recovery Team

17. Teams & Assignments Cont.
Determine status of physical structure. Coordinate with Facilities and Operations, Public Utilities, Internet Providers.
Engineering Recovery Team
Determine needs of recovery team personnel and external replacement personnel. Coordinate all Personnel Issues.
Human Resources Team
Provide support for the acquisition of any and all services needed for recovery. Including physical structure, hardware, software, services
Finance Team
Coordinate communications within university and external to university.
Media Relations Team
Coordinate all aspects of recovery.
Management Team
Determine risk factors and manage security.
Security Team

18. Critical Services

19. DR Folder

20. DR Periodic Maintenance Tasks
As Events Occur
Update Software Inventory
Update Organizational Charts
Update Critical Services
Annually
Review Disaster Recovery Plan
Test Disaster Recovery Plan
Property Control Audit
Risk Assessment
Weekly
Verify / Update critical contacts:
Daily
Update Public Folder Contacts
Update Documentation as appropriate

21. External DR Site

22. Disaster Recovery Site
Partnered with the State at E-for³t (Enterprise Facility for Operational, Recovery, Readiness, Response & Transition)
Acquired Space in 2007
At first service was free
Now is costs us ~$10K Annually

23. E-for³t Equipment HP EVA 4400 HP C7000 Blade Chasis 3 HP BL 465 G1
4 TB Available Storage
HP EVA 4400
iSCSI converter to EVA
HP C7000 Blade Chasis
2 Quad Core 2.3 GHz
16 GB RAM
3 HP BL 465 G1

24. Services Available at E-for³t
Active Directory
AD Site for Both Faculty/ Staff and Student Domains
Internet Native Banner
Oracle Data Guard
Logs replicate every 15 minutes
Faculty / Staff
Empty mailboxes
Web Communications
Password Vault
Password Manager Pro Fully Redundant
IM&T Disaster Recovery Plan and Associated Documentation
Weekly Robocopy of all Information

25. Future Considerations/ Challenges
VMware Site Recovery Manager
Faculty / Staff Hosted Services
Additional Colocation
Lessons Learned
Managing Departmental Expectations
Keeping DR Updated with Limited Support Staff

26. Questions? Brad Sharp, Manager of Technology Operations
Department of Information Management & Technology