Software testing strategies 2 CS 560 Lecture 17

Building Reliable Systems: Two principles For a software system to be reliable: Each stage of development must be done well, with incremental verification and testing. Testing and correction do not ensure quality, but reliable systems are not possible without testing.

Static and dynamic verification testing Static verification testing: Techniques of verification that do not include execution of the software. May be manual or use computer tools. Dynamic verification testing: Testing the software with trial data. Debugging to remove errors.

Static verification testing - Reviews Reviews are a form of static verification testing that is carried out throughout the software development process

Benefits of reviews Multiple people looking at design/code. Uncover mistakes, suggest improvements. Developers share expertise which helps with training. Incompatibilities between components can be identified. Gives developers an incentive to tidy loose ends. Helps scheduling and management control.

Static verification testing: pair design and pair programming Two people work together as a team: design and/or coding testing and system integration documentation and hand-over Benefits include: two people create better software with fewer mistakes cross training Many software development organizations report excellent productivity using this technique.

Static verification testing: Program Inspections Formal program reviews whose objective is to detect faults Code is read and reviewed line by line. 150 to 250 lines of code in 2 hour meeting. Use checklist of common errors. Requires team commitment and trained leaders. So effective that it is claimed that it can replace unit testing.

Static verification testing: Analysis Tools Static analysis tools Cross-reference table: Shows every use of a variable, procedure, object, etc. Information flow analysis: Identifies input variables on which an output depends. Path analysis: Identifies all possible paths through the program.

Defensive Programming Murphy's Law: If anything can go wrong, it will. Defensive Programming: Write simple code. If code is difficult to read, rewrite it. Develop modular components that are highly cohesive and low coupling. Always include comments about each function/method and all complex lines of code.

Dynamic verification testing: stages of testing Testing is most effective if divided into stages, with user interface testing carried out separately. Unit testing (component) unit test System testing integration test performance test installation test Acceptance testing (carried out separately)

Dynamic testing: Unit testing Tests on small sections of a system: a single class or function Emphasis is on accuracy of code against specification (requirement) If unit testing is not thorough, system testing becomes almost impossible. If your are working on a project that is behind schedule, do not rush the unit testing.

Dynamic testing: Unit testing, interface (documentation) Interface types: Parameter interfaces Data passed from one procedure to another. Shared memory interfaces Block of memory is shared between procedures. Procedural interfaces Sub-system encapsulates a set of procedures to be called by other sub-systems. Message passing interfaces Sub-systems request services from other sub-systems.

Dynamic testing: Unit testing, interface errors Interface misuse A calling component calls another component and makes an error in its use of its interface. Ex: parameters in the wrong order, wrong number of parameters, incorrect data format, wrong interface protocol, etc. Interface misunderstanding A calling component embeds assumptions about the behaviour of the called component which are incorrect. Timing errors The called and the calling component operate at different speeds and out-of-date information is accessed

Dynamic testing: Unit testing, Basis paths (documentation) The objective of path testing is to build a set of test cases so that each path through the program is executed at least once. Ensures statement/branch coverage. If every condition in a compound condition is considered: Condition coverage can be achieved as well. Steps for basis path testing: Draw a (control) flow graph using the source code. Line numbers can dictate each node in the graph. Calculate the cyclomatic complexity using the flow graph. Determine the basis set of linearly independent paths. Design test cases to exercise each path in the basis set.

Dynamic testing: Unit testing, Basis paths (documentation) Flow Graphs Used to depict program control structure. Can be drawn from a piece of source code. Flow Graph Notation: composed of edges and nodes. An edge starts from one node and ends at another node.

Dynamic testing: Unit testing, Basis paths (documentation) Flow graph for extreme algorithm

Dynamic testing: Unit testing, Basis paths (documentation) Calculating cyclomatic complexity: E = number of edges N = number of nodes P = set of connected components CC = E – N + 2P CC = 11 – 10 + 2 = 3 Independent paths through the program: 12, 1, 2, 3, 5, 6, 7, 10 \\one element 12, 1, 2, 3, 5, 6, 7, 8, 7, 10\\max in 1st position 12, 1, 2, 3, 5, 6, 7, 8, 9, 7, 10\\searching for max Test cases should be derived so that all of these paths are executed

Dynamic testing: Unit testing, Basis paths (documentation) Designing the test cases: Path 1 test case: 12, 1, 2, 3, 5, 6, 7, 10 Input data: [4] Expected output: 4 Path 2 test case: 12, 1, 2, 3, 5, 6, 7, 8, 7, 10 Input data: [6, 2, 5, 1, 3] Expected output: 6 Path 3 test case: 12, 1, 2, 3, 5, 6, 7, 8, 9, 7, 10 Input data: [5, 2, 1, 1, 8, 3, 4] Expected output: 8

Dynamic testing: Unit testing, Basis paths Example 2 (documentation) Flow graph for Fibonacci algorithm

Dynamic testing: Unit testing, Basis paths Example 2 (documentation) Calculating cyclomatic complexity: CC = E – N + 2P CC = 13 – 12 + 2 = 3 Independent paths through the program: 1, 2, 3, 4, 7, 8, 9 \\ n == 0 1, 2, 3, 4, 7, 8, 9, 10, 15 \\ n == 1 1, 2, 3, 4, 7, 8, 9, 10, 11, 12, 13 \\ n > 1

Dynamic testing: Unit testing, Basis paths Example 2 (documentation) Designing the test cases: 0,1, 1, 2, 3, 5, 8, … Path 1 test case: 1, 2, 3, 4, 7, 8, 9 Input data: 0 Expected output: 0 Path 2 test case: 1, 2, 3, 4, 7, 8, 9, 10, 15 Input data: 1 Expected output: 1 Path 3 test case: 1, 2, 3, 4, 7, 8, 9, 10, 11, 12, 13 Input data: 2

Dynamic testing: system testing, integration test (documentation) Tests complete systems or subsystems composed of integrated components. Main difficulty is localizing errors Errors may not exist until components rely on each other to function properly. Incremental integration testing reduces this problem.

Dynamic testing: system testing, performance/Stress test (documentation) Exercises the system beyond its maximum design load. Stressing the system often causes defects to come to light. Stressing the system test failure behavior. Systems should not fail catastrophically. Stress testing also checks for unacceptable loss of service or data. Particularly relevant to distributed systems which can exhibit severe degradation as a network becomes overloaded.

Dynamic testing: acceptance testing (documentation) Used to determine if the requirements of a software product are met. Gather key acceptance criteria The list of features/functions that will be evaluated before testing the product. Determine testing approaches Types of acceptance tests: stress, timing, compliance, capacity Testing levels: system level, component level, integration level Test methods and tools provided by development/testing team Test data recording Description of how acceptance test will be recorded

Documentation of testing (documentation) Every project needs a test plan that documents the testing procedures for thoroughness, visibility, and for future maintenance. It should include: Description of testing approach. List of test cases and related bugs. Procedures for running the tests. Test analysis report.

Key points of software testing Test parts of a system which are commonly used rather than those which are rarely executed. Acceptance testing is based on the system specifications and requirements. Flow graphs identify test cases which cause all paths through the program to be executed. Interface defects arise because of specification misreading, misunderstanding, errors or invalid timing assumptions.