DATA CENTER OUTAGE BRIEFING JANUARY 10–11, 2014 INFORMATION AND EDUCATIONAL TECHNOLOGY
Review of Events Cause Analysis and Current Efforts Communications Vulnerabilities Mitigation Plans Lessons Learned Communication Improvements Agenda
INFORMATION AND EDUCATIONAL TECHNOLOGY Summary: 3 incidents Friday, Jan 10: Virtualization and uConnect firewall Saturday, Jan 11: Virtualization Virtualization outage affected most major systems on campus Some mitigation lessened impact on Saturday uConnect firewall outage Extended , authentication and DNS service outage for uConnect users (additional 4 hours) Review of Events:
INFORMATION AND EDUCATIONAL TECHNOLOGY 11 am 12 pm 1 pm 2 pm 3 pm 4 pm 5 pm 6 pm 7 pm 8 pm 12 am 1 am 2 am 3 am 4 am 5 am 6 am 7 am 8 am 9 am 10 am 11 am 12 pm 1 pm 2 pm Outage Timeline: 3 incidents Virtualization Outage Friday, January 10th Saturday, January 11th uConnect Firewall Outage CAS & Smartsite restored routing restored VM guests started to restore Services Most services restored except uConnect Virtualization Degradation (critical services stable) Virtualization Outage VM hosts rebooted VM guests started to restore Services Most services restoredAll services restored VM hosts rebooted Firewall fail over to secondary w/o success Hard power cycle restores firewall and uConnect Services 1 2 3
INFORMATION AND EDUCATIONAL TECHNOLOGY Services Impacted Admissions Banner Central Authentication Services (CAS)* Computing Accounts Electronic Death Registry System Data Center File Services DaFIS DavisMail Data Center Virtualization Final Grade Submission Geckomail Kuali Financial Services Identity and Access Management IET Web Sites MyInfoVault MyUCDavis ServiceNow and SSC Case Management Shibboleth Smartsite* Time Reporting System Web Content Management System uConnect Services UC Davis Directory Listings UC Davis Home Site * CAS was restored to physical hardware on Fri 1:40pm which restored dependent services such as departmental applications and Smartsite.
INFORMATION AND EDUCATIONAL TECHNOLOGY Regular outage communication channels were unavailable Websites (status page, Communications issued Automated notices on IT-Express phone system (updated 3 times) Twitter updates (8 on 01/10; 5 on 01/11) Progress updates on Status web page (status.ucdavis.edu) starting Friday mid-afternoon to 300+ campus technologists (01/11) Communications
INFORMATION AND EDUCATIONAL TECHNOLOGY Hardware is redundant, but many services are hosted in single location on a single SAN Critical uConnect directory services reside on a single network The system status page is dependent on the local infrastructure IET is not aware of all critical services that rely on our infrastructure Vulnerabilities
INFORMATION AND EDUCATIONAL TECHNOLOGY SAN Software Upgrade completed Implement diversification for critical services (Authentication, uConnect Directory Services, Status Page, WWW) Integrate cloud services to improve diversity Develop process to identify critical campus services dependent on IET infrastructure. Mitigation Plans
INFORMATION AND EDUCATIONAL TECHNOLOGY Move from disaster recovery to business continuity Normal communication channels were unavailable Communication and decision-making protocols when normal channels unavailable Not prepared for normal channels being unavailable Lessons Learned
INFORMATION AND EDUCATIONAL TECHNOLOGY Review service outage communication protocols, contacts, and venues Ensure multiple modes of communication (text, cell, , web, phone, social media) are available; leverage new WarnMe system extension for non-emergency notifications Closer collaboration with Emergency Manager and Strat Comm Ensure broad awareness of outage communication channels Launch cloud-based status page – Status Page I/O Leverage AggieFeed for broader communication Communication Improvements
INFORMATION AND EDUCATIONAL TECHNOLOGY Status Page I/O
INFORMATION AND EDUCATIONAL TECHNOLOGY Architecture