Software Testing Reference: Software Engineering, Ian Sommerville, 6 th edition, Chapter 20

CMSC 345, Version 4/032 Topics Covered l The testing process l Defect testing Black box testing »Equivalence partitions White box testing »Equivalence partitions »Path testing l Integration testing Top-down, bottom-up, sandwich, thread Interface testing Stress testing l Object-oriented testing

CMSC 345, Version 4/033 Testing Goal l The goal of testing is to discover defects in programs. l A successful test is a test that causes a program to behave in an anomalous way. l Tests show the presence, not the absence, of defects. l Test planning should be continuous throughout the software development process.

CMSC 345, Version 4/034 The Testing Process l Component testing Testing of individual program components (e.g., functions, methods, or classes) Usually the responsibility of the component developer (except sometimes for critical systems) l Integration and system testing Testing of groups of components integrated to create a sub- system or entire system Usually the responsibility of an independent testing team Tests are based on system specification l Acceptance testing Run in presence of customer or by customer Used to validate all system requirements

CMSC 345, Version 4/035 l Test data - Inputs that have been devised to conduct the particular test l Expected output – Recorded before test is conducted l Actual output – The output actually received l Pass/fail criteria – Criteria to determine if test passed or failed when the actual results are compared to the expected results. Determined before test is conducted. The Testing Process (con’t)

CMSC 345, Version 4/036 Black Box Testing l An approach to testing where the program is considered as a “black box” (i.e., one cannot “see” inside of it) l The program test cases are based on the system specification, not the internal workings (e.g., algorithms), of the program. l Use equivalence partitions when conducting black box testing

CMSC 345, Version 4/037 Equivalence Partitioning l Input data and output results often fall into different classes where all members of a class are related. l Each of these classes is an equivalence partition where the program behaves in an equivalent way for each class member. l Test cases should be chosen from each partition, especially at the boundaries.

CMSC 345, Version 4/038 l Partition system inputs into groups (partitions) that should cause equivalent behavior. Include both valid and invalid inputs. If input is a 5-digit integer between 10,000 and 99,999, equivalence partitions are: 99,999 l Choose test cases at the boundaries of these partitions: 9,999 10,000 99, ,000 Equivalence Partitions Example 1

CMSC 345, Version 4/039 Equivalence Partitions Example 2

CMSC 345, Version 4/0310 Example - Search Routine procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ; Pre-condition -- the array has at least one element T’FIRST <= T’LAST Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

CMSC 345, Version 4/0311 l Inputs that conform to the pre-conditions l Inputs where a pre-condition does not hold* l Inputs where the key element is a member of the array l Inputs where the key element is not a member of the array *According to Sommerville Search Routine Input Categories

CMSC 345, Version 4/0312 What Tests Should We Use?

CMSC 345, Version 4/0313 Search Routine Test Cases

CMSC 345, Version 4/0314 l Sometimes called structural testing l Derivation of test cases according to program structure (can “see” inside) l Objective is to exercise all program statements at least once l Usually applied to relatively small program units such as functions or class methods White Box Testing

Binary search (Java)

CMSC 345, Version 4/0316 Any New Tests?

CMSC 345, Version 4/0317 Binary Search Test Cases

CMSC 345, Version 4/0318 Path Testing l The objective of path testing is to ensure that the set of test cases is such that each path through the program is executed at least once. l The starting point for path testing is a program flow graph.

Binary search flow graph

CMSC 345, Version 4/0320 l 1, 2, 8, 9 l 1, 2, 3, 8, 9 l 1, 2, 3, 4, 6, 7, 2 l 1, 2, 3, 4, 5, 7, 2 l Test cases should be derived so that all of these paths are executed. l A dynamic program analyser may be used to check that paths have been executed (e.g., LINT). Independent Paths

CMSC 345, Version 4/0321 l The minimum number of tests needed to test all statements equals the cyclomatic complexity. l CC = number_edges – number_nodes + 2 l In the case of no goto’s, CC = number_decisions + 1 l Although all paths are executed, all combinations of paths are not executed. Cyclomatic Complexity

CMSC 345, Version 4/0322 Integration Testing l Tests the complete system or subsystems composed of integrated components l Integration testing is black box testing with tests derived from the requirements and design specifications. l Main difficulty is localizing errors l Incremental integration testing reduces this problem.

CMSC 345, Version 4/0323 Incremental Integration Testing

CMSC 345, Version 4/0324 Approaches to Integration Testing l Top-down testing Start with high-level system and integrate from the top-down, replacing individual components by stubs where appropriate l Bottom-up testing Integrate individual components in levels until the complete system is created l In practice, most integration testing involves a combination of both of these strategies: Sandwich testing (outside-in) Thread testing

CMSC 345, Version 4/0325 Top-down Testing

CMSC 345, Version 4/0326 Bottom-up Testing

CMSC 345, Version 4/0327 Method “Pro’s” l Top-down l Bottom-up l Sandwich l Thread

CMSC 345, Version 4/0328 Interface Testing l Interface misuse A calling component calls another component and makes an error in its use of its interface; e.g., parameters in the wrong order or of the wrong type. l Interface misunderstanding A calling component embeds assumptions about the behavior of the called component which are incorrect.

CMSC 345, Version 4/0329 Interface Types l Parameter interfaces Data passed from one procedure to another l Shared memory interfaces Block of memory is shared between procedures l Procedural interfaces Sub-system encapsulates a set of procedures to be called by other sub-systems l Message passing interfaces Sub-systems request services from other sub-systems

CMSC 345, Version 4/0330 Some Interface Testing Guidelines l Design tests so that parameters to a called procedure are at the extreme ends of their ranges (boundaries). l Always test pointer parameters with null pointers. l Design tests that cause the component to fail. l In shared memory systems, vary the order in which components are activated.

CMSC 345, Version 4/0331 Stress Testing l Exercises the system beyond its maximum design load. Stressing the system often causes defects to come to light. l Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or data. l Particularly relevant to distributed systems which can exhibit severe degradation as a network becomes overloaded.

CMSC 345, Version 4/0332 l The components to be tested are object classes that are instantiated as objects. l No obvious “top” or “bottom” to the system for top-down or bottom-up integration and testing. l Levels: Testing class methods Testing the class as a whole Testing clusters of cooperating classes Testing the complete OO system Object-oriented Testing

CMSC 345, Version 4/0333 Object Class Testing l Complete test coverage of a class involves Testing all operations associated with an object Setting and interrogating all object attributes Exercising the object in all possible states l Inheritance makes it more difficult to design object class tests as the information to be tested is not localized.

CMSC 345, Version 4/0334 Object Integration l Levels of integration are less distinct in object- oriented systems. l Cluster testing is concerned with integrating and testing clusters of cooperating objects. l Identify clusters using knowledge of the operation of objects and the system features that are implemented by these clusters

CMSC 345, Version 4/0335 Approaches to Cluster Testing l Use case or scenario testing Testing is based on user interactions with the system Has the advantage that it tests system features as experienced by users l Thread testing Tests the system’s response to events as processing threads through the system l Object interaction testing Tests sequences of object interactions that stop when an object operation does not call on services from another object

CMSC 345, Version 4/0336 Scenario-based Testing l Identify scenarios from use cases and supplement these with sequence diagrams that show the objects involved in the scenario

CMSC 345, Version 4/0337 Sample Sequence Diagram