Disaster Recovery Testing

Before we start – Hot topic! WannaCry Ransomware Atttack

Highlights Only 55k$ of ransom paid Only 3 bitcoin wallets – no possiblity to track users who paid Windows machines affected – mostly Windows XP and Windows CE Most machines had no critical data but had a critical role – cash registers, ATMs, kiosks, terminals Kill switch detected but new versions emerged Remedies: Windows update, restore from external (!) backup, antivirus software NHS MRI case Customer Service Desks alerted last week Only several Incidents for Windows Server A large number of end-user Incidents No Disaster Recovery Invocation SLAs Breached for many accounts

Hello! Mateusz Holewski IT Service Continuity Manager at IBM

What is a disaster? Fire, flood, hurricane, tornado, earthquake, volcano Plane crash, vandalism, terrorism, sabotage, loss of personnel Anything that destroys normal processing capabilities

”... It is not a bad day at the office or a (business as usual) hiccup – this is when there is a fundamental change to the environment in which you are working” “Managing Openness: Trade and Outward-oriented Growth After the Crisis” Mona Haddad, Ben Shepherd

Why? FUNCTIONALITY + CONSTRAINT - AND VALUE AVAILABILITY CAPACITY AND CONTINUITY CONTINUITY SECURITY

Data always safe and available Successful audit Business Compliance Restore business in minutes Minimize downtime No loss in revenue Practicing being prepared Always staying in business ● Data always safe and available Successful audit SLA, SLO met and achieved Key functions and essential services continue operating Customers Confidence of customers Protecting reputation and brand Services uninterrupted

IT Service Continuity IT Service that consists of: Documents, Procedures, People, Architecture, Skills, Testing, Continual Improvement Part of Business Continuity Holy Grail for IT Specialists

Is paper test a real test?

Provider: Infrastructure provider Customer / Business Business Process Recovery Layers Application support Applications Middleware ITSCM Provider: Infrastructure provider Databases Filesystems Operating Systems Storage – disk management Hardware – servers, tape libraries Networking – LAN, WAN, Storage networks Facilities - building access, electricity, cooling, server racks etc. Personnel assigned to perform recovery

Health & safety Regulations What comes first? IT Business Processes Health & safety Regulations Emergency Services

Crisis Management Framework

RTO RPO? RPO: How much data are you prepared to lose? RPO RTO Disaster RTO: How long do you think recovery will take? RPO RTO Disaster

Architecture DR Solutions 2½ types of architecture: High Availability Disaster Recovery - restoring machines from backup Cloud Solutions….?

High Availability Load Balancer Objects Synchronization Data Synchronization Secondary or Recovery Site Primary Site Storage Synchronization Machine Synchronization Pros: Easy to operate, fast, reliable Cons: Expensive, requires maintenance

Disaster Recovery Offsite Storage Recovery Site Primary Site Pros: Easy to implement, cost effective Cons: Slower than HA solution

Architecture Backup and Restore is crucial for a successful recovery Even if we have full HA solution - tapes and offsite storage are used There are two types of people - the ones that make backups and those who will… 3-2-1 Rule: 3 copies – 2 types of media – 1 offsite location Cloud solutions transfer the DR to the cloud provider… Do you know the architecture of your system?

”... It’s too expensive to shut down the system” Customer

Offsite Storage Disaster Recovery Recovery Site Primary Site

Planning a DR Test Alert Documents Scope Timeline Communications Success criteria - objectives People Procedures Hardware and Network Software and Data Testers/QA area Business Processes Rollback phase

What to prepare? Documents – plan, technical instructions People – assign roles, plan communication Architecture – prepare DR environment, Backup&Restore solution Test – perform trainings, identify gaps Areas to check: 3rd party vendors: network, telephony etc. Application configuration vs. Infrastructure RTO vs RPO Offsite Storage

DR and Software Testing “Test Everything in 2 hours” Focus on Business Processes, Functionalities, Data Consistency Usually out of scope or limited scope: Performance, Security

Possible Areas of DR Application Testing Scripted functional tests - previously approved UAT scripts Data Consistency - reconciliation between systems, SQL queries, message queues etc. Smoke Testing Successful path scenario External connections Printing - printing queues Batch jobs, scheduled tasks, maintenance tasks Data Flow Isolation tests - separate registers, preparation to rollback phase

Environments in a project DEV TEST DR? STAGE/PRE-PROD DR PRODUCTION DR

Testing applications The application is already tested… System is up and running for 20 years now... Hardware is almost the same… OS and DB are the same… We can log in to the system, all seems fine… Easy, let’s go home...

Challenges for software testing on DR Environment Documentation Mismatch Lack of Test Cases Change Requests - IT Service Processes mismatch Is this DR or production environment? (serious issue during a test)

Challenges for software testing on DR Environment Hardware mismatch Network setup - firewalls, redirection - isolated environment Different configuration - interfaces to other systems Security configuration Licences for commercial software Limited performance Data consistency

Challenges for software testing on DR Environment Time constraint - DR tests are extremely expensive for customer Rollback phase !!!!

What is expected? Test script Contact info, number of people - logistics Technical Support Timeline - planning Issue Log, Issue reporting Initial issue assessment (depending on skill level) DR Education

Thank you