David M. Crosby Disaster Recovery Planning ……. Business Contingency Planning A Business Model For Continuity Planning David M. Crosby Information Assurance and Business Sustainability

Introductions David M. Crosby Former VP of Information Security, Venture Bank 35 Years Experience in IT 15 Years Experience in Information Security and Business Sustainability Finance, Aerospace, Insurance and Energy Industry; and Technology and Services Company Principal

Our World is Changing

The Business Continuity Management Program Service To Our Customers Institutional Best Practices County Regs. HIPAA GLB Notice Disaster Recovery and Contingency Operations Protect Information and Processes Int. Audit Federal Regs. Ext Audit SB 1386 State Regs.

The Business Continuity Management Program The interruption of fundamental business processes for any extended period of time could have a debilitating affect on our basic infrastructure…….and our way of life E-Commerce Private and Business Online Trading Cash Advances At ATM Machines Personal and Commercial Online Banking Purchases By Credit Cards Just In Time Inventories Communications Student Services Grants and Endowments General Administration & Finance

The Business Continuity Management Program ERP DRP BCP CMP ERP – Emergency Response Plan: Steps Taken To Immediately Respond To An Event, Ensure Personnel Safety, Minimize Further Impact To Assets, And Make Proper Notifications. DRP – Disaster Recovery Plan: Steps Taken To Restore Specified Infrastructure Requirements Such As Information Systems, Clinical Equipment Environments, Internal And External Network Connections, And Data Structures Utilizing Alternate Resources For Hardware, Software, Data, and Networks. BCP – Business Contingency Plan: Steps Taken To Restore Alternate Business Processes In The Event That Automated Processes Or Business Infrastructures Are Unavailable, Employing Documented Workaround And/Or Manual Procedures And Alternate Resources. CMP – Crisis Management Plan: Steps Taken To Manage The Event To Ensure That Order Is Maintained, Employee Assistance Is Being Provided, Proper Information Is Being Disseminated By Appropriate Representatives, Action Items Are Effectively Escalated, And Ongoing Internal And External Notifications Are Consistent.

The Business Continuity Management Program ERP DRP BCP CMP Working Components Response - Notifications, assessments, escalations, declarations, etc. (established procedures) Recovery/Relocation - Mobilization, Quick-ship, Infrastructure, Network and Data recovery, etc.. Movement of staff, patients, and business units to alternate facilities (flexibility and adaptability) Resumption - of Business Operations and I.T. functionality (business units must synch up processes and resume operations at an alternate site) Re-assessment - of situation, strategies, planning, reactions (input from all involved parties) Restoration - Movement back to home site and/or normal operations (reconstituted at restored site by I.T. and/or Business Units In order to clarify roles and responsibilities, and better define the scope of various components, we have establish our own internal set of definitions

Components Of The Emergency Response Plan First Response Notification Assessment and Status Escalations Declarations Personnel Safety Damage Mitigation Local Authorities Evacuations Initial Notifications Telephone Trees Command Center Assembly Damage Assessment Initial Status Reporting Secondary Notifications Organizational Committees Local Authorities Vendors Customers Media Checklists Scripts Procedures Contact Lists Vendors Mobilization

Components Of The Disaster Recovery Plan Disaster Recovery Planning Steps taken to restore specified infrastructure requirements such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate resources for hardware, software, data, and networks. What To Do When The Computer Goes Down

Components Of The Disaster Recovery Plan Disaster Recovery Is…… The successful recovery of mission-critical I.T. services to the customer community in response to a crisis Flexible Response To A Crisis Place to Recover (Location/Equipment/Network) Defined “Recovery Set” (Critical Components) Reliable Backups Test – Maintain – Test Service Continuation Disaster Recovery is NOT….. Recovery of full environment A business continuity plan A replacement for conventional service plans A trivial decision

Components Of The Disaster Recovery Plan Infrastructure Applications Analysis Network Infrastructure Opens Systems Documentation Hardware Systems Databases TSO/CICS Test Criteria/Objectives Questionnaires Interviews Analysis Documented Profiles Test Criteria/Objectives Recovery Plans LDAP DNS Email Intranet/Internet Gateway Servers Test Criteria/Objectives Owned Equipment DR Vendor Equipment Connectivity Requirements Test Criteria/Objectives Remote Access Parameters Define ‘rogue’ FTPs Identified Network Services Checklists Scripts Procedures Contact Lists Test Criteria/Objectives

Components Of The Disaster Recovery Plan I.T. Requirements RECOVERY TIME OBJECTIVE: (RTO) The period of time in which systems, applications, or I.T. functions must be recovered after an outage. RTO's are often used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation. RECOVERY POINT OBJECTIVE: (RPO) The point in time to which systems and data must be restored after an outage. RPO's are often used as the basis for the development of backup strategies, and as a determinant of the amount of data that may need to be recreated after the systems or functions have been recovered.

Components Of The Business Contingency Plan DRP BCP DRP – Disaster Recovery Plan: Steps taken to restore specified infrastructure requirements such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate resources for hardware, software, data, and networks. - Hardware - System Software - Data and Data Structures - Applications - Networks - Desktop Services - Production Support We established a set of working definitions for I.T. disaster recovery planning BCP – Business Contingency Plan: Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are unavailable, employing documented workaround and/or manual procedures and alternate resources. - Relocation of Personnel - Availability of remote support services and network connections - Contingency office space

What To Do While The Computer Is Down Components Of The Business Contingency Plan Business Contingency Planning Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are unavailable, employing documented workaround and/or manual procedures and alternate resources. What To Do While The Computer Is Down

Components Of The Business Contingency Plan Business Contingency Planning Is…… The successful response to an interruption in normal operating procedures and thus services to the customer community Flexible Response To A Crisis Place to Initiate Contingency Operations (Systems/Network/Location/Personnel/Equipment) Documented Systems Workaround Procedures Alternate Resources Business Continuity is NOT….. Disaster Recovery, Emergency Preparedness, or Crisis Management A Permanent Solution An I.T. Issue

Manual Business Processes Alternate Data Capture Components Of The Business Contingency Plan Mobilization Alternate Processes Alternate Resources Personnel & Skill Sets Facilities Vendors Hardware/Software Communications Business Resumption Logistics Transition Back To I.T. Validation/Audit Normal Operations Business Cycles Documentation Procedures Logistical Support Forms Contact Lists I.T. Workarounds Manual Business Processes Alternate Data Capture Logistics Location(s) Transportation Personnel

Business Continuity Planning Scenarios Components Of The Business Contingency Plan Business Continuity Planning Scenarios Loss of I.T Services or Resources Loss of Functional Support Personnel Loss of Facility Loss of Network Connectivity Loss of Voice Communications Loss of 3rd Party Suppliers Loss of Business Partners

Build Contingency Plans Components Of The Business Contingency Plan Build Contingency Plans Identify key functional components to establish the business environment Define the alternate process requirements for each component Ensure interdependent business processes are identified and can be synched up Define minimal processing requirements for each component TEST - TEST - TEST - TEST

Components Of The Business Contingency Plan Business Recovery Requirements RECOVERY TIME OBJECTIVE: (RTO) When do I have to have an alternate process in place to address loss of primary functions (I.T. and otherwise) ? RECOVERY POINT OBJECTIVE: (RPO) How current does my information have to be when normal processes are resumed ?

Components Of The Business Contingency Plan Centralized Administration and Coordination Decentralized Development, Maintenance and Execution Web-Enabled – 24 x 7 x 365 access from anywhere with VPN connection Automated progress reporting during Plans development, maintenance, and execution Define relationship between BCPs and DRPs (RTO and RPO) Capable of expanding to include ERP and CMP Real-time updating to a single database, not multiple Plans Version Control on all Plans Concurrent Plan development Issue Templates Import Templates Develop BCPs Flexibility when producing BCPs…………..or executing BCPs “Show me all Plans by Department….” “Show me all Plans by Building…..” “Show me all Plans by Building, by Floor…..” “Show me all Plans by Building, by Floor, by Department For execution and “system” testing, different views of recovery and continuity plans can be established to allow flexibility and accurate reporting

Components Of The Business Contingency Plan Negotiate The Service Level Agreement Between I.T. And Business Operations Use Both The I.T. And Business RTO & RPO As The Basis Disaster Recovery Plan Test Results Quantify Timelines Business Contingency Plan Exercises Qualify Impact I.T. Capabilities Improve Timelines – But At A Cost Business Contingencies Reduce Impact - But Require I.T. Capabilities This will require managing to the expectations of the organization with clearly defined SLAs including RTOs, RPOs, priority sequences, etc. Criticality Rankings Systems Recovery Sequencing Business Process Prioritization I.T. and Business Process Timelines Negotiated RTO and RPO

Components Of The Business Contingency Plan Results I.T. Better Understands The Customers’ Issues and Requirements I.T. Obtains A Clearly Documented Set Of Customer Expectations For DRP’s - Clarify and Justify Budget Forecasts - Establishes Specific Test Objectives - Ensure Active Customer Involvement In Testing & Recovery Processes Business Units Better Understand The Role Of I.T. In The Contingency Process Business Units Obtain A Set Of Parameters From Which To Develop their BCP’s - Workaround Procedures During Downtime - Procedures For Capturing Lost Transactions From Downtime and During Recovery - Restoration Of Normal Environments

Components Of The Crisis Management Plan Event Analysis Reaction Planning Communications Documentation Catastrophic Events Criminal Events Disease/Epidemics Technological or Safety Utility or Structural Weather Personal vs. Professional Emotional Assistance Addressing Traumatic Stress Family Assistance Pgms Professional Assistance Provide Information & Counseling Post Incident Follow-up Local Media Employees Local Authorities Openness Accuracy Balance Designate a point person Continuous Flow Employee Checklists And Action Plans Press Release Data Employee Notification Mechanisms

Crisis Management Preparedness Key Elements Identification of vulnerabilities Performance of regional threat assessment Assessment of system resources Communications infrastructure Standardization of plans Dissemination of information Analysis of system Surge Capacity Collaboration with federal, state, local agencies Crisis Management Preparedness Key Elements Components Of The Crisis Management Plan

Regional Collaboration Components Of The Crisis Management Plan Regional Collaboration Who does what?? Who calls whom?? Local Fire/EMS/OES Law Enforcement Health Dept./Hazmat Hospitals State State Health Dept. State OES/DHS Federal Federal Emergency Mgmt Agency CDC Military Private Sector Collaboration Individual Plans Supplement/Complement Broader Plans Clinical Care Response Public Health Response

The Business Continuity Management Program When the issues surrounding both I.T. Disaster Recovery Plans and Business Unit Business Contingency Plans come together what is at stake becomes much clearer, and each can understand the others objectives and expectations. Only then can a total Business Continuation Program be effective. And if the organization has an effective Business Continuation Program, not only can it assure that its goals and objectives will be met…..but will also become a valued partner in the protection of the larger infrastructure.…. The Business Continuity Management Program

Questions.....Comments ????

Helping Others