1. Planning for Testing
In a DevOps World

2. Planning for a DevOps World Developing for Operations
Performance Information & Collecting Metrics
Telemetry Information & Performing Diagnostics
Configuration Management
Planning for Testing

3. Planning for a DevOps World
What are some of the important things to keep in mind if you want to live in a DevOps World?
Continuous Deployment – How can I get my product to market more quickly?
Continuous Integration – How can I build my product more quickly?
Continuous Testing – Did I test that product before it went out the door? Do I continue to test my product while it is in production?
Continuous Monitoring – Is my product working? How healthy is it?
Configuration Management – How do I know what components are used / were used to build my product?
Planning for Testing

4. Developing for Operations
What if you had an integrated test harness that you could plug into and get all sorts of good information about your system?
Performance Information – How many widgets has the system processed over the last hour? How many users are connected to the system? On average, how long does it take to process a user’s request?
Telemetry Information – How are your system resources doing? What is the error rate on the network interfaces? Is the system swapping memory? Is your system I/O bound? Memory bound? CPU bound?
Diagnostic Information – Is everything hooked up correctly? Is everything working alright?
User Behavior Information – Is the path through the application that the user takes anything like the path that we designed for the user? Do we need to make changes to better suit the needs of the user?
Planning for Testing

5. Performance Information & Collecting Metrics
Does your application capture metrics?
How do you quantify the performance of your application?
Are you meeting your service level agreements?
Are you using this information to anticipate growth needs and to plan for scaling out or up?
Planning for Testing

6. Adaptive Behavior
Example: Existing resources are totally consumed by the current demand.
Does your application have the ability to spin up new resources to meet the demand?
Does your application throttle back on the resources when the demand subsides?
Is this behavior automatically performed by the application?
Planning for Testing

7. Planning for Testing

8.
Planning for Testing

9. Telemetry Information & Performing Diagnostics
What are you measuring and logging?
Where are you recording your log entries and measurements?
How do you filter your records?
How do you analyze and interpret your records?
What benefit does operations get from the analysis?
Planning for Testing

10. Adaptive Behavior
Example: Disk write-errors are occurring at an unacceptable rate.
How do you know?
Does your application automatically send out a service request message?
Does your application automatically route data to other disks?
Planning for Testing

11. Diagnostic Information
Is the system operating?
Is the system operating properly?
Are there components that are not operating properly?
Does the “mis-operation” threaten the integrity of the overall system results?
Should the system halt processing rather than produce questionable results?
Are there “self-healing” steps that could be taken automatically?
Planning for Testing

12. Adaptive Behavior
Example: Your application looses connectivity with its database.
Does your application know?
Does your application reconnect?
Does your application display a “temporarily out of service” message for the user?
Does your application failover to an alternate site?
Is this behavior automatically performed?
Planning for Testing

13. Configuration Management
If you have to fix something then how do you know where to begin?
Configuration Audit – What are the components/subcomponents that make up the system? What is the version for each of the components? Can you answer the question “What code is running?” with confidence?
Roll Call – Are all of the components connected? What is a “safe state” if they are not?
Planning for Testing

14. Planning for Testing

15. Planning for Testing in a DevOps World
As we build our systems, we should be asking ourselves these questions:
How do I test my product in production?
Plan for testing.
How do I know that my product is working and healthy?
Build in tests for monitoring.
If there is something wrong then how do I find out what components are being used in production so that I know where to start looking for answers?
Build in configuration management support.
Automate “Everything”
Adaptive Behavior
Planning for Testing