1. Disaster Recovery
Testing

5. Before we start –
Hot topic!
WannaCry Ransomware Atttack

7. Highlights Only 55k$ of ransom paid
Only 3 bitcoin wallets – no possiblity to track users who paid
Windows machines affected – mostly Windows XP and Windows CE
Most machines had no critical data but had a critical role – cash registers, ATMs, kiosks, terminals
Kill switch detected but new versions emerged
Remedies: Windows update, restore from external (!) backup, antivirus software
NHS MRI case
Customer Service Desks alerted last week
Only several Incidents for Windows Server
A large number of end-user Incidents
No Disaster Recovery Invocation
SLAs Breached for many accounts

8. Hello!
Mateusz Holewski
IT Service Continuity Manager at IBM

9. What is a disaster?
Fire, flood, hurricane, tornado, earthquake, volcano
Plane crash, vandalism, terrorism, sabotage, loss of personnel Anything that destroys normal processing capabilities

10. ”... It is not a bad day at the office
or a (business as usual) hiccup – this is when there is a fundamental change to the environment in which you are working”
“Managing Openness: Trade and Outward-oriented Growth After the Crisis”
Mona Haddad, Ben Shepherd

11. Why? FUNCTIONALITY + CONSTRAINT - AND VALUE AVAILABILITY CAPACITY AND
CONTINUITY
CONTINUITY
SECURITY

12. Data always safe and available Successful audit
Business
Compliance
Restore business in minutes
Minimize downtime
No loss in revenue
Practicing being prepared
Always staying in business ●
Data always safe and available
Successful audit
SLA, SLO met and achieved
Key functions and essential services continue operating
Customers
Confidence of customers
Protecting reputation and brand
Services uninterrupted

13. IT Service Continuity IT Service that consists of: Documents,
Procedures,
People,
Architecture,
Skills,
Testing,
Continual Improvement
Part of Business Continuity
Holy Grail for IT Specialists

17. Is paper test a real test?

19. Provider: Infrastructure provider
Customer / Business
Business
Process
Recovery Layers
Application support
Applications
Middleware
ITSCM
Provider: Infrastructure provider
Databases
Filesystems
Operating Systems
Storage – disk management
Hardware – servers, tape libraries
Networking – LAN, WAN, Storage networks
Facilities - building access, electricity, cooling, server racks etc.
Personnel assigned to perform recovery

20. Health & safety Regulations
What comes first? IT
Business Processes
Health & safety Regulations
Emergency Services

21. Crisis Management Framework

22. RTO RPO? RPO: How much data are you prepared to lose? RPO RTO Disaster
RTO: How long do you
think recovery
will take?
RPO
RTO
Disaster

23. Architecture DR Solutions
2½ types of architecture:
High Availability
Disaster Recovery - restoring machines from backup
Cloud Solutions….?

24. High Availability
Load Balancer
Objects Synchronization
Data Synchronization
Secondary or Recovery Site
Primary Site
Storage Synchronization
Machine Synchronization
Pros: Easy to operate, fast, reliable Cons: Expensive, requires maintenance

25. Disaster Recovery Offsite Storage Recovery Site Primary Site
Pros: Easy to implement, cost effective Cons: Slower than HA solution

26. Architecture Backup and Restore is crucial for a successful recovery
Even if we have full HA solution - tapes and offsite storage are used
There are two types of people - the ones that make backups and those who will…
3-2-1 Rule: 3 copies – 2 types of media – 1 offsite location
Cloud solutions transfer the DR to the cloud provider…
Do you know the architecture of your system?

27. ”... It’s too expensive to shut down the system”
Customer

28. Offsite Storage
Disaster Recovery
Recovery
Site
Primary Site

29. Planning a DR Test Alert Documents Scope Timeline
Communications Success criteria - objectives
People
Procedures
Hardware and Network
Software and Data
Testers/QA area
Business Processes
Rollback phase

30. What to prepare? Documents – plan, technical instructions
People – assign roles, plan communication
Architecture – prepare DR environment, Backup&Restore solution
Test – perform trainings, identify gaps
Areas to check:
3rd party vendors: network, telephony etc.
Application configuration vs. Infrastructure
RTO vs RPO
Offsite Storage

31. DR and Software Testing
“Test Everything in 2 hours”
Focus on Business Processes, Functionalities, Data Consistency
Usually out of scope or limited scope: Performance, Security

32. Possible Areas of DR Application Testing
Scripted functional tests - previously approved UAT scripts
Data Consistency - reconciliation between systems,
SQL queries, message queues etc.
Smoke Testing
Successful path scenario
External connections
Printing - printing queues
Batch jobs, scheduled tasks, maintenance tasks
Data Flow
Isolation tests - separate registers, preparation to
rollback phase

33. Environments in a project
DEV
TEST
DR?
STAGE/PRE-PROD DR
PRODUCTION DR

34. Testing applications The application is already tested…
System is up and running for 20 years now...
Hardware is almost the same…
OS and DB are the same…
We can log in to the system, all seems fine… Easy, let’s go home...

35. Challenges for software testing on DR Environment
Documentation Mismatch
Lack of Test Cases
Change Requests - IT Service Processes
mismatch
Is this DR or production environment?
(serious issue during a test)

36. Challenges for software testing on DR Environment
Hardware mismatch
Network setup - firewalls, redirection - isolated
environment
Different configuration - interfaces to other systems
Security configuration
Licences for commercial software
Limited performance
Data consistency

37. Challenges for software testing on DR Environment
Time constraint - DR tests are extremely
expensive for customer
Rollback phase !!!!

38. What is expected? Test script
Contact info, number of people - logistics
Technical Support
Timeline - planning
Issue Log, Issue reporting
Initial issue assessment (depending on skill
level)
DR Education

39. Thank you