It’s all about the dpisd dr plan Weathering the storms It’s all about the dpisd dr plan
history Allison - June 5, 2001 Rita - September 23, 2005 Ike – September 13, 2008 Harvey – August 25-29, 2017 Post Harvey – September 15, 2017
Lessons learned
Allison 2001 Have On Hand An industrial Sink/sump pump and large drying fans Ensure that all conduit is “foamed in” if coming from underground Watch the weather reports
Rita 2005 Have a formal disaster recovery plan Consolidate all servers into 1 location Move to electronic payments for employees Designate a remote offsite working location Store at least 1 set of backups offsite Maintain a transfer LINe to rollover the main number that allows us to leave an update
Additional Changes Expanded the Technology building moving the data center upstairs Addition was built to Miami dade standards with fire doors Added a natural gas generator and Replaced diesel generator with natural gas generator Installed a fire suppression system and Liebert systems (air & battery backup) in the data system Purchase a district-wide communication system Began tape backups at a remote site
Ike 2008 We can survive! Our plan is good! Our team works well!
Changes over the last 9 years Added coating to windows to protect from bullets and flying objects Moved several systems to the cloud or customer hosted services including our website Establish d.r. agreements with our finance and student data providers Multiple backups for district– disc backups at 1 campus; Tape backups at another campus which are sent to safesite every 2 weeks for safekeeping; and real data backups at technology center
Harvey & post-Harvey 2017 We can not predict the weather Do not trust certain vendors who claim to test your equipment We need to pass the November bond so we can update our systems We still have a good plan We still have a great team!
Exit Procedures today Notify our hosted vendors of pending danger and enact steps toward d.r. Have generators and Liebert systems tested Make additional backups and have safesite pick them up Before leaving the district, pull all electronics up off the floors and bring down unnecessary systems
Procedures upon return Bring all systems up Return electronic systems to working positions
Disaster recovery plan
Steps to developing a plan Write with a team Define your scope and mission Assess your risks Define priorities and perform a business impact analysis Define recovery strategies and procedures Develop a communication plan Appendix with comprehensive inventory of all it resources test often
Scope and mission Explain why and how you’re going to create the disaster recovery plan. The mission clearly defines the document’s main purpose. The purpose of this Technology Disaster Recovery Plan is to serve as an operations protocol to protect and preserve the DPISD network infrastructures and related assets from any unplanned events that could disrupt or discontinue network operations.
Identify RISKS or potential disasters Hurricane General or plant explosion Tornado or high winds Smoke or fire Shelter in place Rising water or flood Air-conditioner failure Electrical failure Loss of critical components cybercrime
Priorities & impact on operations Develop lists of applications, servers, and appliances Develop Rank them according to priority for running district operations Group them according to priorities Set approximate date ranges as to when systems should be placed back on line Identify which warrant additional consideration for D.R. Agreements, duplication of equipment, off-site hosting, etc.
Outline steps to recovery For each identified risk, recommend the actions for recovery Outline the associated estimated costs with the recovery efforts Identify the persons or teams responsible for the recovery actions Leadership Damage assessment Network Desktop Telecommunications Information Systems Help Desk
Communication plan Most important aspect Who will communicate to the superintendent? Who will work with district communications department to find a means of communicating with the parents, community, and staff about the loss of technology?
Appendix Solid technical documentation to recover your systems Comprehensive inventory of all it resources Data backup policies Vendor lists Service contract lists Diagrams Other technical specifications
Consider the disaster recovery plan a living document
Test often
Dr. Kari murphy Kmurphy@dpisd.org @karimurphy