Principles of Incident Response and Disaster Recovery Chapter 8 Disaster Recovery: Operation and Maintenance

Principles of Incident Response and Disaster Recovery2 Objectives Understand the key challenges an organization faces when engaged in disaster recovery operations Know what actions organizations take to prepare for the activation of the DR plan Recognize what critical elements compose the response phase of the DR plan Know what occurs in the recovery phase of the DR plan

Principles of Incident Response and Disaster Recovery3 Objectives (continued) Understand how an organization uses the resumption phase of the DR plan Know how an organization resumes normal operations using the restoration phase of the DR plan

Principles of Incident Response and Disaster Recovery4 Introduction An organization should operate on the premise that it is only a matter of time until a disaster strikes Proper response to a disaster requires meticulous preparation and ongoing diligence In the event of a total loss, an organization must be prepared to promptly reestablish operations at a new permanent location

Principles of Incident Response and Disaster Recovery5 Facing Key Challenges Disasters are not confined to the IT department or limited to the assets of the organization Disasters may also affect the community and employees personally, as well as vendors and suppliers In a major or widespread disaster, there may be challenges associated with local emergency services, service providers, and other-non business issues

Principles of Incident Response and Disaster Recovery6 Facing Key Challenges (continued) Areas possibly affected in a major disaster: –Basic emergency and transportation services –Food and survival supplies –Water supplies and sanitation –Electrical power –Products and services delivered by vendors and suppliers –Telecommunications services (land and cellular) –Transportation services (freeways, highways, and local streets)

Principles of Incident Response and Disaster Recovery7 Facing Key Challenges (continued) Major disaster can result in: –Declaration of state of emergency –Imposition of martial law –Restrictions on movement or quarantines DR plan typically involves 5 phases: –Preparation –Response –Recovery –Resumption –Restoration

Principles of Incident Response and Disaster Recovery8 Preparation: Training the DR Team and the Users In DR planning, there is no prevention phase Take steps during preparation to minimize losses Preparation: making an organization ready for possible contingencies that escalate to disaster Preparation phase is continuous, but other phases are activated by triggers such as: –Management notification –Employee notification –Emergency management notification –Local emergency services –Media outlets

Principles of Incident Response and Disaster Recovery9 Disaster Recovery Planning as Preparation 3 primary objectives of the DR plan: –Eliminate or reduce potential for injuries or loss of life, damage to facilities, and loss of assets and records to minimize disruption and financial loss and reduce or limit liability exposure –Stabilize the effects of the disaster to allow recovery efforts to begin –Implement DR procedures

Principles of Incident Response and Disaster Recovery10 Disaster Recovery Planning as Preparation (continued) Recovery efforts must be prioritized as follows: –Employees –Customers –Facilities –Assets –Records CP team creates scenario development and impact analysis, and categorizes the level of threat for each potential disaster

Principles of Incident Response and Disaster Recovery11 Disaster Recovery Planning as Preparation (continued) Key features of the DR plan: –Clear delegation of roles and responsibilities –Execution of the alert roster and notification of key personnel –Use of employee check-in systems –Clear establishment and communication of business resumption priorities –Complete and timely documentation of the disaster –Preparations for alternative implementations

Principles of Incident Response and Disaster Recovery12 Disaster Recovery Planning as Preparation (continued) All employees should have 2 types of emergency information in possession at all times: –Personal emergency information (who to notify) –Instructions on what to do in the event of an emergency (snapshot of the DR plan) Emergency info should include contact number or hotline for the organization, emergency services numbers, evacuation and assembly locations, disaster recovery coordinator, etc. Crisis management: focused steps that deal with safety of people who are involved in the disaster

Principles of Incident Response and Disaster Recovery13 DR Training and Awareness DR training focuses on the roles each individual is expected to execute during an actual disaster For most employees, training is limited to awareness General job function training is key to being prepared for disaster recovery actions Cross-training should also be considered, both vertically and horizontally, to deal with personnel shortages Training should include operating in degraded mode

Principles of Incident Response and Disaster Recovery14 DR Training and Awareness (continued) Disaster management team (command and control group) training is primarily about communication Communications team training involves preparing information notices, news releases, and internal memorandums and directives Hardware recovery team training may include training to rebuild damaged systems by scavenging from other damaged systems

Principles of Incident Response and Disaster Recovery15 DR Training and Awareness (continued)

Principles of Incident Response and Disaster Recovery16 DR Training and Awareness (continued) Systems recovery team training is mostly the same as their normal operations training Network recovery team training may include wireless network installation as a quick recovery mechanism, walkie-talkie deployment, and other connectivity mechanisms Storage recovery team training may include rebuilding damaged storage systems and recovering data from offsite

Principles of Incident Response and Disaster Recovery17 DR Training and Awareness (continued)

Principles of Incident Response and Disaster Recovery18 DR Training and Awareness (continued) Applications recovery team training primarily consists of skills used in normal operations Data management team training focuses on rapid data restoration and recovery from backup Vendor contact team training focuses on methods of obtaining resources as quickly as possible Damage assessment and salvage team training primarily consists of hardware repair skills that enable team members to determine if items are repairable or not

Principles of Incident Response and Disaster Recovery19 DR Training and Awareness (continued)

Principles of Incident Response and Disaster Recovery20 DR Training and Awareness (continued) Business interface team training includes communication skills and mechanisms for assisting with routine needs Logistics team training includes training in purchasing and procurement and providing rest and comfort for other workers

Principles of Incident Response and Disaster Recovery21 DR Plan Testing and Rehearsal Testing of the plan and the training and rehearsal of the plan can overlap Testing can involve several levels of assessment: –Employee self-assessments –Peer evaluations –Formally appointed internal assessors –External certification or accreditation groups Classroom training should come first before actual rehearsals

Principles of Incident Response and Disaster Recovery22 DR Plan Testing and Rehearsal (continued) Testing strategies include: –DR plan desk check: individual review of plan –DR plan structured walk-through: group exercise –DR plan simulation: each individual works independently –DR plan parallel testing: act as if the disaster had occurred but do not interfere with normal operations –DR plan full interruption: act as if disaster had occurred, and perform all steps including data recovery –DR plan war gaming: few tools available for this in the private sector

Principles of Incident Response and Disaster Recovery23 Rehearsal and Testing of the Alert Roster Alert roster must be tested more often than other plan components due to employee turnover Quarterly testing is recommended Alert message contains just enough information to allow employees to determine which part of the DR plan to implement Auxiliary phone alert and reporting system: automated system for activating the alert roster You are never completely ready for a disaster Key skills to retain from rehearsals are flexibility, decisive decision making, and professionalism

Principles of Incident Response and Disaster Recovery24 Disaster Response Phase Response phase: the phase associated with implementing the reaction to a disaster Response phase focuses on controlling or stabilizing the situation for the purposes of: –Protecting human life and well-being –Limiting or containing damage to facilities and equipment –Managing communications with employees and other stakeholders

Principles of Incident Response and Disaster Recovery25 Recovery Phase Recovery phase: –Initiates the recovery of the most time-critical business functions –Focuses on getting up and running as quickly as possible, even in degraded mode; less critical operations must wait for the resumption phase Primary goals of the recovery phase: –Recover critical business functions –Coordinate recovery efforts –Acquire resources to replace damaged or destroyed equipment or materials –Evaluate whether to implement the business continuity plan

Principles of Incident Response and Disaster Recovery26 Resumption Phase Resumption phase: focuses on non-critical functions BIA should guide in the prioritization of critical and secondary functions Goals of the resumption phase: –Initiate implementation of secondary functions –Finalize implementation of primary functions –Identify additional needed resources –Continue planning for restoration

Principles of Incident Response and Disaster Recovery27 Restoration Phase Restoration phase: the final phase of disaster recovery Primary goals of restoration phase: –Repair all damage to primary site or select or build a replacement facility –Replace damaged or destroyed contents of primary site including supplies, equipment, and material –Coordinate relocation from temporary offices to primary site or suitable new replacement facility –Restore normal operations at primary site, beginning with critical functions, then secondary operations –Stand down the DR team and conduct the after- action review

Principles of Incident Response and Disaster Recovery28 Repair or Replacement Two possibilities in restoration phase: –Reestablish operations at primary site –Establish operations at a new permanent site Reestablish operations at primary site: –Must be able to rebuild damaged facilities –May need to relocate administrative functions to provide space to the operational functions while rebuilding is underway New permanent site options: –New location –Complete rebuild on site of destroyed facilities

Principles of Incident Response and Disaster Recovery29 Restoration of the Primary Site After physical facilities are rebuilt, the contents must be replaced, including: –Office furniture, PCs, photocopies, filing systems, office supplies, etc. Must assess what will be covered by insurance and service contracts

Principles of Incident Response and Disaster Recovery30 Relocation from Temporary Offices Transition back to the primary site must be carefully coordinated to minimize additional disruptions to business functions If data management functions must move, may want to use a movement coordinator to plan the relocation of personnel, equipment, materials, and data back to the primary site

Principles of Incident Response and Disaster Recovery31 Resumption at the Primary Site Must reestablish all normal operations, including tertiary operations that may have been suspended due to relocation, such as: –Managing employee benefit packages –Employee training and awareness programs –Organizational planning retreats and meetings –Routine progress meetings and reports –Long-term planning activities –Research and development activities

Principles of Incident Response and Disaster Recovery32 Standing Down and the After-Action Review Standing down: the deactivation of the disaster recovery team, releasing individuals back to normal duties After-action review provides a method for management to obtain input and feedback from each group represented in the team AAR log serves as legal and planning record and tool for future training Official report should include AAR and reports from individual teams

Principles of Incident Response and Disaster Recovery33 Summary An organization should operate on the premise that it is only a matter of time until a disaster strikes 5 phases in the DR plan: preparation, response, recovery, resumption, restoration Goals of DR and business resumption planning: eliminate or reduce potential for injuries or loss of life, stabilize the effects of the disaster, implement the DR plan based on type and impact of disaster Recovery phase attempts to recover the most critical business functions immediately

Principles of Incident Response and Disaster Recovery34 Summary (continued) Resumption focuses on the remaining unrestored functions Restoration seeks to: –Repair all damage to primary site or arrange a replacement facility –Replace all damaged or destroyed contents –Coordinate relocation from temporary back to primary site –Restore normal operations at primary site –Stand down the DR teams and conduct the AAR