1. Software testing strategies 2
CS 560
Lecture 17

2. Building Reliable Systems: Two principles
For a software system to be reliable:
Each stage of development must be done well, with incremental verification and testing.
Testing and correction do not ensure quality, but reliable systems are not possible without testing.

3. Static and dynamic verification testing
Static verification testing:
Techniques of verification that do not include execution of the software.
May be manual or use computer tools.
Dynamic verification testing:
Testing the software with trial data.
Debugging to remove errors.

4. Static verification testing - Reviews
Reviews are a form of static verification testing that is carried out throughout the software development process

5. Benefits of reviews
Multiple people looking at design/code. Uncover mistakes, suggest improvements.
Developers share expertise which helps with training.
Incompatibilities between components can be identified.
Gives developers an incentive to tidy loose ends.
Helps scheduling and management control.

6. Static verification testing: pair design and pair programming
Two people work together as a team:
design and/or coding
testing and system integration
documentation and hand-over
Benefits include:
two people create better software with fewer mistakes
cross training
Many software development organizations report excellent productivity using this technique.

7. Static verification testing: Program Inspections
Formal program reviews whose objective is to detect faults
Code is read and reviewed line by line.
150 to 250 lines of code in 2 hour meeting.
Use checklist of common errors.
Requires team commitment and trained leaders.
So effective that it is claimed that it can replace unit testing.

8. Static verification testing: Analysis Tools
Static analysis tools
Cross-reference table:
Shows every use of a variable, procedure, object, etc.
Information flow analysis:
Identifies input variables on which an output depends.
Path analysis:
Identifies all possible paths through the program.

9. Defensive Programming
Murphy's Law:
If anything can go wrong, it will.
Defensive Programming:
Write simple code.
If code is difficult to read, rewrite it.
Develop modular components that are highly cohesive and low coupling.
Always include comments about each function/method and all complex lines of code.

10. Dynamic verification testing: stages of testing
Testing is most effective if divided into stages, with user interface testing carried out separately.
Unit testing (component)
unit test
System testing
integration test
performance test
installation test
Acceptance testing (carried out separately)

11. Dynamic testing: Unit testing
Tests on small sections of a system:
a single class or function
Emphasis is on accuracy of code against specification (requirement)
If unit testing is not thorough, system testing becomes almost impossible.
If your are working on a project that is behind schedule, do not rush the unit testing.

12. Dynamic testing: Unit testing, interface (documentation)
Interface types:
Parameter interfaces
Data passed from one procedure to another.
Shared memory interfaces
Block of memory is shared between procedures.
Procedural interfaces
Sub-system encapsulates a set of procedures to be called by other sub-systems.
Message passing interfaces
Sub-systems request services from other sub-systems.

13. Dynamic testing: Unit testing, interface errors
Interface misuse
A calling component calls another component and makes an error in its use of its interface.
Ex: parameters in the wrong order, wrong number of parameters, incorrect data format, wrong interface protocol, etc.
Interface misunderstanding
A calling component embeds assumptions about the behaviour of the called component which are incorrect.
Timing errors
The called and the calling component operate at different speeds and out-of-date information is accessed

14. Dynamic testing: Unit testing, Basis paths (documentation)
The objective of path testing is to build a set of test cases so that each path through the program is executed at least once.
Ensures statement/branch coverage.
If every condition in a compound condition is considered:
Condition coverage can be achieved as well.
Steps for basis path testing:
Draw a (control) flow graph using the source code.
Line numbers can dictate each node in the graph.
Calculate the cyclomatic complexity using the flow graph.
Determine the basis set of linearly independent paths.
Design test cases to exercise each path in the basis set.

15. Dynamic testing: Unit testing, Basis paths (documentation)
Flow Graphs
Used to depict program control structure.
Can be drawn from a piece of source code.
Flow Graph Notation: composed of edges and nodes.
An edge starts from one node and ends at another node.

16. Dynamic testing: Unit testing, Basis paths (documentation)
Flow graph for extreme algorithm

17. Dynamic testing: Unit testing, Basis paths (documentation)
Calculating cyclomatic complexity:
E = number of edges
N = number of nodes
P = set of connected components
CC = E – N + 2P
CC = 11 – = 3
Independent paths through the program:
12, 1, 2, 3, 5, 6, 7, 10 \\one element
12, 1, 2, 3, 5, 6, 7, 8, 7, 10\\max in 1st position
12, 1, 2, 3, 5, 6, 7, 8, 9, 7, 10\\searching for max
Test cases should be derived so that all of these paths are executed

18. Dynamic testing: Unit testing, Basis paths (documentation)
Designing the test cases:
Path 1 test case:
12, 1, 2, 3, 5, 6, 7, 10
Input data: [4]
Expected output: 4
Path 2 test case:
12, 1, 2, 3, 5, 6, 7, 8, 7, 10
Input data: [6, 2, 5, 1, 3]
Expected output: 6
Path 3 test case:
12, 1, 2, 3, 5, 6, 7, 8, 9, 7, 10
Input data: [5, 2, 1, 1, 8, 3, 4]
Expected output: 8

19. Dynamic testing: Unit testing, Basis paths Example 2 (documentation)
Flow graph for Fibonacci algorithm

20. Dynamic testing: Unit testing, Basis paths Example 2 (documentation)
Calculating cyclomatic complexity:
CC = E – N + 2P
CC = 13 – = 3
Independent paths through the program:
1, 2, 3, 4, 7, 8, 9 \\ n == 0
1, 2, 3, 4, 7, 8, 9, 10, 15 \\ n == 1
1, 2, 3, 4, 7, 8, 9, 10, 11, 12, 13 \\ n > 1

21. Dynamic testing: Unit testing, Basis paths Example 2 (documentation)
Designing the test cases: 0,1, 1, 2, 3, 5, 8, …
Path 1 test case:
1, 2, 3, 4, 7, 8, 9
Input data: 0
Expected output: 0
Path 2 test case:
1, 2, 3, 4, 7, 8, 9, 10, 15
Input data: 1
Expected output: 1
Path 3 test case:
1, 2, 3, 4, 7, 8, 9, 10, 11, 12, 13
Input data: 2

22. Dynamic testing: system testing, integration test (documentation)
Tests complete systems or subsystems composed of integrated components.
Main difficulty is localizing errors
Errors may not exist until components rely on each other to function properly.
Incremental integration testing reduces this problem.

23. Dynamic testing: system testing, performance/Stress test (documentation)
Exercises the system beyond its maximum design load.
Stressing the system often causes defects to come to light.
Stressing the system test failure behavior.
Systems should not fail catastrophically.
Stress testing also checks for unacceptable loss of service or data.
Particularly relevant to distributed systems which can exhibit severe degradation as a network becomes overloaded.

24. Dynamic testing: acceptance testing (documentation)
Used to determine if the requirements of a software product are met.
Gather key acceptance criteria
The list of features/functions that will be evaluated before testing the product.
Determine testing approaches
Types of acceptance tests: stress, timing, compliance, capacity
Testing levels: system level, component level, integration level
Test methods and tools provided by development/testing team
Test data recording
Description of how acceptance test will be recorded

25. Documentation of testing (documentation)
Every project needs a test plan that documents the testing procedures for thoroughness, visibility, and for future maintenance.
It should include:
Description of testing approach.
List of test cases and related bugs.
Procedures for running the tests.
Test analysis report.

26. Key points of software testing
Test parts of a system which are commonly used rather than those which are rarely executed.
Acceptance testing is based on the system specifications and requirements.
Flow graphs identify test cases which cause all paths through the program to be executed.
Interface defects arise because of specification misreading, misunderstanding, errors or invalid timing assumptions.