1. Writing a Test Strategy
Tor Stålhane

2. Why a testing strategy We need a testing strategy to help
System and software testers plus Test-and- Evaluation staff to determine the overall test strategy for development or modification of a software intensive system
Project stakeholders – customers and senior management – approve the test strategy
Testers and system and software analysis to determine
Test objectives
Qualification requirements
Verification and validation criteria

3. Testing strategy concepts
We will discuss the following concepts:
Purpose of a test strategy
Testing focus
Contents of a test strategy
Software integrity levels
Test objectives and priorities

4. Purpose of a test strategy – 1
The test strategy is important to
Obtain consensus on test goals and objectives from stakeholders – e.g. management, developers, testers, customers and users
Mange expectations right from the start
Be sure that we are heading in the right direction
Identify the type of tests to be conducted at all test levels

5. Purpose of a test strategy – 2
When we write a test strategy is important to remember that:
Whatever we do some kind of test strategy will emerge. Thus, we might as well specify the one we think is best
A documented strategy is the most effective way to get an early agreement on goals and objectives
We need to address:
Human factors – usability
Interoperability, except for stand-alone systems

6. Testing focus
Our focus will depend on which stakeholder we are considering at the moment:
Users – acceptance test and operational tests
Analysis – systems test, qualification tests
Designer – integration tests
Programmer – unit tests
The main point is that we need to define the stakeholder first – then the tests to be run.

7. Contents of a test strategy – 1
The following is a list of what can be specified in a test strategy. Not all of it is needed in all cases – only use what is necessary.
Project plan, risks and activities
Relevant regulations – depending of application area
Required processes and standards
Supporting guide lines

8. Contents of a test strategy – 2
Stakeholders – e.g. users, testers, maintainers – and their objectives
Necessary resources – people, computers
Test levels and phases
Test environment
Completion criteria for each phase
Required documentation and review method for each document

9. Software integrity level
There are several ways to define software integrity levels. When we choose an integrity level this will strongly influence the way we do testing.
We will look at three definitions of integrity levels:
IEEE 1012 – general software
ISO – automotive software
IEC – general safety critical software

10. IEEE 1012 – general software
4, High – some functions affect critical system performance.
3, Major – some functions affects important system performance
2, Moderate – some functions effect system performance but workarounds can be implemented to compensate.
1, Low – some functions have noticeable effect on system performance but creates only inconveniences

11. Development requirements level
V&v Activities
V&V activity
Development requirements level
Design level
Implementation level
Test level
SW Integrity Level 4 3 2 1
Acceptance test execution X
Acceptance test plan
Interface analysis
Management and review support
Management review of V&V

12. ISO 26262 – automotive software
The ASIL level – A, B, C or D – is the outcome of the combination of three factors:
S – Severity. How dangerous is a event
E – Probability. How likely is the event
C – Controllability. How easy the event to control if it occurs

13. Finding the ASIL level Severity Probability C1 C2 C3 S1 E1 QM E2 E3 A

14. Methods for software integration testing
Methods and Measures
According
to req.
ASIL A B C D 1
Requirements based test
9.4.4 ++ 2
External interface test + 3
Fault injection test 4
Error guessing test

15. Methods for software unit testing
Methods and Measures
According
to req.
ASIL A B C D 1
Functional tests
8.4.2
See table 8.2 2
Structural coverage
See table 8.3 3
Resource usage measurement + ++ 4
Back-to-back test between simulation model and code, if applicable

16. IEC 61508 – safety critical software
integrity
level
High demand or continuous mode of
operation
(Probability of a dangerous failure per
hour) 4  10 - 9 to  8 3 7 2 6 1 5
PFDavg = Ft / Fnp . The table above, together with this
value decides the SIL level.

17. Detailed design Technique/Measure Ref SIL1 SIL2 SIL3 SIL4 1a
Structured methods including for example, JSD, MASCOT, SADT and Yourdon
C.2.1 HR 1b
Semi-formal methods
Table B.7 R 1c
Formal methods including for example, CCS, CSP, HOL, LOTOS, OBJ, temporal logic, VDM and Z
C.2.4
--- 2
Computer-aided design tools
B.3.5 3
Defensive programming
C.2.5 4
Modular approach
Table B.9 5
Design and coding standards
Table B.1 6
Structured programming
C.2.7 7
Use of trusted/verified software modules and components (if available)
C.2.10
C.4.5
Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied. 17

18. Module testing and integration
Technique/Measure
Ref
SIL1
SIL2
SIL3
SIL4 1
Probabilistic testing
C.5.1
--- R HR 2
Dynamic analysis and testing
B.6.5
Table B.2 3
Data recording and analysis
C.5.2 4
Functional and black box testing
B.5.1
B.5.2
Table B.3 5
Performance testing
C.5.20
Table B.6 6
Interface testing
C.5.3
a) Software module and integration testing are verification activities (see table A.9).
b) A numbered technique/measure shall be selected according to the safety integrity level.
c) Appropriate techniques/measures shall be selected according to the safety integrity level.

19. Test objectives and priorities
Only in rather special cases can we test all input – binary input – output and few parameters. Thus, we need to know
The overall objective of testing
The objective of every test case
The test case design techniques needed to achieve our goals in a systematic way.
The test objectives are our requirements specification for testing.

20. Test data selection
One of the important decisions in selecting a test strategy is how to select test data. We will look at five popular methods
Random testing
Domain partition testing
Risk based testing
User profile testing
Bach’s heuristic risk-based testing

21. Random testing The idea of random testing is simple:
Define all input parameters – e.g. integer, real, string
Use a random test / number generator to produce inputs to the SUT
The main problem with this method is lack of an oracle to check the results against. Thus, manual checking is necessary.
The method is mostly used for crash testing (robustness testing) – will the system survive this input?

22. Domain partition testing – 1
Definitions:
A domain is a set of input values for which the program performs the same computation for every number of the set. We want maximal domains so that the program performs different computations on adjacent domains
A program is said to have a domain error if the program incorrectly performs input classification

23. Domain testing – simple example
If a >< 0, this equation has the following solution
Otherwise we have that

24. Testing domains
A = 0
a = 0
A = 0
a >< 0 D3 D2 D1

25. Risk based testing The idea of risk based testing is to
Identify the risk or cost of not delivering a certain functionality.
Use this info to prioritize tests. We will cover this in more details later under “Test prioritation”

26. User profile testing
The main idea with this type of testing is to generate tests that mirror the user’s way of using the system.
Consider a situation where we know that the users in 80% of all case
Fetch a table from the database
Update one or more info items
Save the table back to the database
Then 80% of all tests should test these three actions.

27. Bach’s risk-based testing
Bach’s heuristics is based on his experience as a tester. Based on this experience he has identified
A generic risk list – things that are important to test
A risk catalogue – things that often go wrong
We will give a short summary of the first of Bach’s lists.

28. Bach’s generic risk list – 1
Look out for anything that is:
Complex – large, intricate or convoluted
New – no past history in this product
Changed – anything that has been tampered with or “improved”
Upstream dependency – a failure here will cascade through the system
Downstream dependency – sensitive to failure in the rest of the system
Critical – a failure here will cause serious damage

29. Bach’s generic risk list – 2
Precise – must meet the requirements exactly
Popular – will be used a lot
Strategic – of special importance to the users or customers
Third-party – developed outside the project
Distributed – spread out over time or space but still required to work together
Buggy – known to have a lot of problems
Recent failure – has a recent history of failures.

30. Test and system level – 1

31. Test and system level – 2
From the diagram on the previous slide we see that we can test on the
Electronics level – e.g. DoorActuator sends the right signal
State / signal level – e.g. door is closed iff DoorStateClosed
Logical level – e.g. the door remain closed as long as the speed is non-zero
Safety level – e.g. the door remain closed as long as the train is moving

32. Acknowledgement
The first part of this presentation is mainly taken from Gregory T. Daich’s presentation “Defining a Software Testing Strategy”, 30 April 2002.