1. The Myth of Continuous Performance Testing
Stephen Townshend

2. Introduction
What is CPT?
The concerns

4. 6/05/2019

5. What is CPT?
The concerns
When is it appropriate?

7. Fail
Pass

8. Moving in the right direction
The concerns
When is it appropriate?
Moving in the right direction

9. The concerns The limited value of component tests
6/05/2019
The concerns
The limited value of component tests
Losing sight of the big picture
Test asset maintenance
Automated pass/fail outcomes
Production-like environments

10. The limited value of component testing

11. Performance risk

12. What component testing tells us
What it does NOT tell us
Response Time
Component specific response time
User/customer experience in the real world
Capacity
Component capacity
Component resource usage in isolation
Capacity of integrated solution under real conditions
Issues relating to the way components integrate
Stability
Nothing
Stability of the solution over prolonged periods of use

15. Credit card example

16. Losing sight of the big picture
Keeping sight of the business purpose is incredibly important. It helps us make better decisions about what and how to performance test appropriately for each situation.
Performance is not a problem we can necessarily break into smaller parts. The sum of the parts is not the same as the whole.

20. Test asset maintenance

21. Test asset maintenance

22. Automated pass/fail outcome
Is meaningful to the business
Makes mathematical sense
Is not overly complicated
Is maintainable/configurable

23. Automated pass/fail outcome

24. Automated pass/fail outcome

25. Production-like environments
In an ideal world we would always test in prod (or prod-like) environments
Principle of DevOps
But in reality – this is almost never the case

26. Production
Environment
Test
Environment

27. Production-like environments
How non-production environments differ from production:
Lower spec (physical) or different hardware configuration (virtual, cloud)
Different numbers of nodes or servers
Missing integration points
Different application or database configuration
The absence of realistic volumes of data
Background jobs which run in production but not test
All of this eats into the confidence we can have in the results

28. Production-like environments
Difference from production
Risk/impact
Lower spec hardware
Environment may not scale to meet production peak load
Different number of nodes
Load balancing related issues may not be picked up
Missing integrations
End user experience not measured accurately
Performance issues relating to external integrations missed
Different app or DB configuration
Any test results are unlikely to reflect real world performance (response time, capacity, or stability)
Absence of realistic data volumes
Artificially quick response times (databases are faster without data)
Background jobs not running
Conflicts between online users and background jobs not assessed (user experience, server resources, stability)

29. Moving in the right direction
When is it appropriate?
Moving in the right direction

30. When is continuous performance testing most appropriate?

31. Moving in the right direction

32. Moving in the right direction

33. FIN