General Membership Meeting April 15, 2004 St. Louis County Emergency Operations Center

PLANNING AND EXECUTING CONTINGENCY EXERCISES WORKAREA, SYSTEMS, AREA-WIDE, REGIONAL Anna M. Bathon, CBCP Bank of America 1

 Why test recovery plans?  Recovery Strategy Considerations  Types of Exercises  Establish A Testing Strategy  Exercise Phases  Planning the Exercise  Preparing for the Exercise  Executing/Conducting the Exercise  Follow-up / Issues Resolution  Closure / Next Exercise Date  Questions Agenda 2

Why Test Recovery Plans? 1.The confluence of five major trends are driving acceptance and adoption of more aggressive recovery solutions:  Businesses’ increased reliance on IT and data  Availability of solutions  Economics – impact of downtime and declining cost of solutions  IT data management challenge 2.Gartner Group comments: “… Enterprises that today tolerate two-day recovery time objectives will see that horizon diminish to one day or less.” 3.Key disaster-related statistics:  43% of companies having a disaster never reopen. An additional 29% close within two years.  68% of businesses that lose their computers for more than 7 days never reopen.  Within 2 weeks of the loss of computer support, 75% of those organizations affected reach critical or total loss of business functions.  Average hourly revenue lost from downtime is $78, Businesses’ availability requirements being measured in hours. 3

5.Demonstrates to Management ability of critical business processes to continue functionality within required timeframes following a disruption. 6.Recognizing a workable plan and making a plan work are two different things. 7.Regular testing and maintaining the plan accordingly will ensure optimum performance. 8.Exercising a plan is not a PASS or FAIL situation, but an opportunity to identify plan deficiencies and improve the recovery processes. 9.Testing is a dynamic process. 10.Provides an opportunity to stress test plans already reviewed as good; exercise strenuously to identify flaws. 11.Environments – workarea, systems – change and should be monitored continuously to assess the impact of changes to recovery strategies. 12.Major revisions to recovery plans require testing and appropriate documentation updated. Why Test Recovery Plans? 4

Recovery Strategy Considerations  Workarea – physical workspace of business units, including critical components, to ensure functionality can be resumed appropriately:  Equipment / hardware  Software  Telecom  Vital records  Compliance  Associate support / Intellectual Capital – What if most or all associates or lost in a disaster situation?  Support partners  Regional impacts  Applications – systems, infrastructure:  File-and-print servers  Application components / locations:  Simple configurations  Complex configurations  Infrastructure dependencies (firewalls, shared components)  External dependencies 5

Recovery Strategy Considerations  Third-Party Service Providers – Dependencies on vendors increasing, thus creating a greater impact when vendors encounter disruptions.  Who are the major strategic suppliers?  What is the product flow throughout your company?  Contingency plan options if vendor suffers a disruption?  Specialized equipment or processes?  Maximum potential for lost income if disruption encountered?  Does an interdependency chart exist?  Regional scenarios :  Natural  Weather (hurricane, earthquake, tornado, ice/snow)  Man-made  Fire  Terrorism  Disgruntled associate reactions  Accidental construction disruptions 6

Recovery Strategy Considerations  Crisis Management :  Call tree notification processes  Associate impacts  Decision-making process to diminish roadblocks in recovery process 7

Types of Exercises 1.Talk-Through / Table Top 2.Simulation / Connectivity 3.Integrated 4.Live 8

Types of Exercises Talk-Through / Table Top  Generally considered first test of a plan  Cost-effective method of exercising plans  Minimal disruption to business  Raise level of awareness of the actual state of readiness  Identify major weaknesses or steps requiring further documentation 9

Types of Exercises Simulation / Connectivity  Validates the facility, supplies, and equipment at the alternate site.  Should include connectivity testing, including voice and/or data connectivity.  Alternate site testing must include network connectivity testing, as appropriate.  Technical support participation dependent on extent of testing as defined by exercise objectives. 10

Types of Exercises Integrated  Exercises multiple components of a plan, in conjunction with each other, typically under simulated operating conditions.  Workarea involves recovery of multiple critical business functions and related onsite systems that would be lost in the event of a site disaster.  Systems involves testing of recovery of multiple applications running on a single component or within a single site, i.e., data center environment.  Where appropriate, upstream/downstream interfaces should be exercised. 11

Types of Exercises Live  Senior Management approval should be required for this type of exercise.  Perform production work at alternate recovery site.  High level of risk involved.  Selected associates, clients, vendors, technical support personnel, business continuity support personnel, and other dependent business units should participate. 12

Establish A Testing Strategy 1.Identify critical components of the recovery plan. 2.Identify frequency of testing based on risk rating determined through completion of BIA, i.e. quarterly, annually, bi-annually. 3.Select test type to most adequately validate all critical components.  Several different test types may need to be conducted to address all critical components to remain compliant. 4.When possible, conduct fully integrated exercises, requiring testing of all critical components. 13

Exercise Phases  Planning  Preparing  Executing / Conducting  Follow-up / Resolution  Closure / Next Exercise Date 14

Planning the Exercise 1.Identify resources 2.Select a test coordinator 3.Select the type of test 4.Define the test scope 5.Develop test goals and objectives 6.Define the disaster scenario 7.Document test assumptions 8.Set test date and duration 9.Define test team and participants 10.Schedule meetings 15

Preparing for the Exercise 1.Conduct preparatory meetings with participants 2.Develop tasks and issues lists 3.Identify equipment and site requirements 4.Document high-level test scripts 5.Develop exercise packet 6.Obtain approvals 16

Executing / Conducting the Exercise 1.Facilitate communication among test teams/participants. 2.Ensure activities occur in order published in exercise packet / scripts. Document deviations. 3.Ensure appropriate participants in the command center or appropriate alternate sites. 4.Work with sequence of events to log timeframes, issues, and any pertinent notations regarding activities. 5.Ensure issues documented and turned into test coordinator. 6.Compile issues into Issues List Report for tracking/resolution purposes. 7.Issues resolved during the test should be noted so. 8.Unresolved issues documented, assigned and tracked to resolution following the exercise. 9.Conduct periodic executive and test team status meetings and issue status updates throughout the exercise. 10.Document all costs associated with conducting the exercise. 11.Update appropriate telephone status resources. 17

Follow-up / Resolution 1.Schedule and conduct post-test review meeting shortly after concluding exercise. 2.Assign appropriate associates to work on resolving outstanding issues. 3.Follow up on resolution status. 4.Distribute test results and outstanding issues list report to Management, appropriate personnel. 5.Obtain validation sign-off forms from participant groups. 6.Retain exercise packets and test results for audit and regulatory reviews. 7.Follow up with participant groups to ensure recovery plans are updated based on test results / observations. 18

Closure / Next Exercise Date 1.Draft Final Summary Report and review with team in preparation for submission to Management:  Final Report is a summary of actual date, time, and results of the exercise.  Include recent upgrades or changes to the workarea/units, systems, or equipment.  List exercise objectives  Briefly note outstanding issues with resolution status and target final resolution date. 2.Finalize Final Summary Report. 3.Submit Summary Report to Management. 4.Ensure all issues are resolved prior to next test. 5.Determine and communicate next exercise date. 19

Future Testing Considerations 1.End-to-end process testing. 2.Integration of different types of plans:  Regional with workarea implications  Regional impacting numerous systems, workareas, vendors 3.Inclusion of new associates in process. 4.Participation in vendor contingency testing. 5.New regulatory concerns impacting recovery strategies. 6.Cyber-threat scenarios. 7.Others??? 20

??????? Questions ??????? 21