1. Non-functional requirements
Piotr Liss Jeppesen Poland
Non-functional requirements
Why performance testing without requirements is a bad idea

2. Introduction Approach Examples No NFRs Rules What is a NFR?

3. Functional vs Non-functional requirement
Example application
Mobile car reservation application
Functional requirements
Defines what application should be able to do.
Defines how application should work.
Non-functional requirements
Example
user is able to reserve a car
filter available cars by its location and car type
charge credit card; etc.
multiple users in same time are able to reserve cars
application will be available by 99% of time
android support since version 6.0; etc.
Example
Requirements for code
Requirements for environment/system

4. Non-functional requirements types
Accessibility
Adaptability
Auditability and control
Availability
Backup
Capacity, current and forecast
Certification
Compliance
Configuration management
Cost, initial and Life-cycle cost
Data integrity
Data retention
Dependency on other parties
Deployment
Development environment
Disaster recovery
Documentation
Durability
Efficiency
Effectiveness
Emotional factors
Environmental protection
Escrow
Exploitability
Extensibility
Failure management
Fault tolerance
Integrability ability to integrate components
Internationalization and localization
Interoperability
Open source
Operability
Performance / response time
Platform compatibility
Privacy (laws)
Portability
Quality (faults discovery)
Readability
Reliability
Reporting
Resilience
Resource constraints
Response time
Reusability
Robustness
Safety or Factor of safety
Scalability (horizontal, vertical)
Security (cyber and physical)
Software, tools, standards etc. Compatibility
Stability
Supportability
Testability
Throughput
Transparency
Usability (Human Factors) by target user community
Volume
Source:

5. Performance basics Value that describes performance
Performance metrics
Value that describes performance
Performance test types
Examples:
Response time (ms)
Responses per minute
Error rate (%)
Number of concurrent users
CPU utilization
Machines responsible for generating load
Load environment
Distributed load environment
Multiple connected machines responsible for generating load controlled by server

6. Introduction Approach Examples No NFRs Rules
Deployment process vs NFRs

7. Performance tests without NFRs
Plan
Create
Implement
Adjust
What plan?
Let’s do this!
“Too much data” and “not reliable”? What do you mean?!
I’ve changed... everything. Now is OK?
Test plans: most important ones used in functional testing
Test environment: existing one is good enough
Planned load: let’s say… user at same time?
Test scripts: 10 scripts, responsible for testing 5 services
Test environment: done
Load generating env: single machine
Results vary significantly after every execution and whole procedure generates huge amount of data to analyze.
Load environment doesn’t gives possibility to check capacity of the product.
Test plans: 2 scenarios covering most user stories
Test env: new one exclusively for performance tests
Planned load: necessary to use distributed environment and generate border load

8. Performance tests with NFRs
Plan
Create
Implement
Adjust
We need to talk about it… again!?
A lot of work ahead of us
Hey! It works!
What adjustments?
Test plans: those responsible for most of load
Test environment: comparable with production env, exclusively for performance testing
Planned load: high enough to judge capacity of product
Test scripts: 2 scenarios covering most user stories
Test environment: as close as production one
Load generating env: distributed environment
Able to simulate valuable load which results gives us information how production will work.
High performance test environment deployed. Load environment can be easily extended.
Test plans: maybe think about new ones
Test env: no need to adjust as long as production doesn’t change
Planned load: an existing environment is good enough for current needs

9. Performance tests with “real-life” NFRs
Plan
Create
Implement
Adjust
This is all we need for now.
Looks easy!
Works! But for how long?
Will it ever ends?!
Test plans: check one fully developed service
Test environment: we just need comparison between versions (performance degradation check)
Planned load: necessary to simulate typical load
Test scripts: 1 script with standard user story
Test environment: any that will not change between executions
Load generating env: single machine
Able to compare performance between releases of one existing service.
Test and load environment appropriate for current needs.
Test plans: adjust when requirements changes
Test env: adjust when requirements changes
Planned load: adjust when requirements changes

10. Solution comparison No NFRs All NFRs Real-life PROS CONS
Quick to provide solution...
Provides the most mature and “ready-to-use” product...
Quite quick solution
Meets current requirements
Provides good test product...
CONS
...which will need a lot of adjustments before it even start to provide valuable informations
...that will need changes in future
Requires a lot of preparations and experienced team
...for current requirements that will change soon

11. Introduction Approach Examples No NFRs Rules
NFRs in tests implementation

12. NFRs example A - published application
Every service should have comparable performance. Typical user stories based on real usage statistics.
Average request-response time can’t be higher than 2sec. In case of reports 8sec.
Number of users: (based on current number of registered users)
Application will increase its market share (verify current environment limit)
Multiple simple single-service backend scenarios based on usage
Automated test triggering would be helpful in case of multiple test scripts
Clear values exceeding of which means test fail
Partially inform us about possible load.
Suggested type of tests: load and stress
Capacity testing requires environment comparable with prod one
Scenarios ✓ ✓
Performance metrics –
Load ✓
Type of tests ✓
CI/CD ✓
Environment

13. NFRs example B - early stage of development
Core service can not generate errors during load tests
Application performance can not decrease
Scenarios based on one service, no information about user stories
Basic load test would be enough
Stable, isolated, not necessarily comparable with production
Process triggered after every product build
Scenarios – ✓
Type of tests ✓
Environment
CI/CD ✓
Performance metrics
irrelevant
irrelevant
Load

14. NFRs example C - replace of old application
Application should work faster than the application we replace (no perf tests)
Response time can not exceed 3 seconds
More than 10 concurrent users capacity is crucial (implement in existing CI with virtual environment)
Active number of users per week on current application: 500
Stress and load tests required
Comparison possible only if we test both applications
Dangerous assumption causing often failed tests
Adding to CI a stress test that can crash whole environment is bad idea
Perf test on virtual machines will not give comparable results
Not necessary all users will move to new application
Type of tests ✓ –
Performance metrics
CI/CD – ✘
Environment –
Load ✘
Scenarios

15. NFRs example D ✘ ✘ ✘ ✘ ✘ ✘ Environment Load Scenarios Type of tests
Performance metrics ✘
CI/CD

16. Introduction Examples Approach No NFRs Rules How to live without NFRs

17. Do we need to compare test environment with production one?
Which scenarios are crucial in perspective of performance tests? (Most used, most important, most problematic)
Do we need to compare test environment with production one?
TEST SCENARIOS
ENVIRONMENT
What kind of solution should I prepare? I need requirements!
What is the purpose of performance tests? (Verify possible degradation, capacity, disaster recovery, simulate real usage)
Do we know how many users uses our application? (or are there any border limits defined for our application)
TYPE OF TESTS
LOAD
What is acceptable response time for given scenarios? (or number of concurrent users)
How often we would like to run tests? (CI/CD)
Do we want to indicate possible source of problems? (Monitoring)
PERFORMANCE METRICS
OTHERS

18. Introduction Examples Approach No NFRs Rules What else to remember?

19. NFRs rules
Performance test results
NFR regarding performance metrics is necessary to judge test result
NFRs might be divided and applies to type of product release:
CI merge request
release candidate
application publishing
NFRs leveling
PASS
FAIL
Average response time: 2440ms
Errors: 0,21%
Average response time: 3521ms
Errors: 1,37%
NFRs should be:
testable
negotiable
easy to implement
easy to understand
Performance test results analysis
It’s helpful to define requirements per service/element/page/feature to make performance analysis easier
testable - requirement have to be described that way, that we application will be able to met them or not
negotiable - not necessary mandatory

20. Even poor quality requirements are better than no requirements.
Assume that requirements will change during product development.
Thanks!
Any questions?
linkedin.com/in/piotrliss
Copyright © 2019 Piotr Liss All Rights Reserved.
Do not distribute and use without the author's consent.