Chapter 14 Testing Tactics

14.1 Software Testing Fundamentals What is a “Good” test? A good test has a high probability of finding an error A good test is not redundant A good test should be “best of breed” A good test should be neither too simple nor too complex Test Case Design OBJECTIVE: to uncover errors CRITERIA: in a complete manner CONSTRAINT: with a minimum of effort and time 1/21

14.1 Software Testing Fundamentals Testability Operability - the better it works the more efficiently it can be tested Observability - what you see is what you test Controllability - the better software can be controlled the more testing can be automated and optimized Decomposability - by controlling the scope of testing, the more quickly problems can be isolated and retested intelligently Simplicity - the less there is to test, the more quickly we can test Stability - the fewer the changes, the fewer the disruptions to testing Understandability - the more information known, the smarter the testing 2/21

14.2 Black-box an White-box Testing Why not try exhaustive testing? loop = 20 X There are 5201014 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this program!! White-Box Testing Note: Testing cannot show the absence of errors and defects. Black-Box Testing 3/21

Definition: White-box or Glass-box Testing 14.3 White-box Testing Definition: White-box or Glass-box Testing Knowing the internal workings of a product, tests are performed to check the workings of all possible logic paths -- testing in the small. Questions Can you guarantee that all independent paths within a module will be executed at least once? Can you exercise all logical decisions on their true and false branches? Will all loops execute at their boundaries and within their operational bounds? Can you exercise internal data structures to ensure their validity? 4/21

Goal: execute every statement at least once 14.4 Basis Path Testing Goal: execute every statement at least once Defines the number of independent paths in the basis set of a program and provides us with an upper bound for the number of tests that must be conducted. a b c d e f g h i j k a（begin） b（in） c d e f g h i j（out） k（stop） TOTAL=TOTAL+A K++ Input A Output K, L, TOTAL Stop T Begin K=0 L=0 TOTAL=0 Do While TOTAL<1000 and A0 F A>0 L++ Compound conditions flow graph Predicate node d’ R1 R2 R3 R4 Cyclomatic Complexity = # of regions E  N + 2 P + 1 5/21

14.4 Basis Path Testing Prepare test cases that will force execution of each path in the basis set – only to execute every statement at least once is NOT enough Example: in A > 1 AND B=0 T A=2 OR X > 1 X = X / A X = X + 1 out F CC = ? 5 Test case : A=3, B=0, X=4. Note: If AND is written as OR, the error will not be detected by the above test case. 6/21

14.5 Control Structure Testing Condition Testing: test every condition at least once Examples: Test ( E1 > E2 ) AND ( E3 = E4 ) B1 AND B2 { ( t, t ), ( f, t ), ( t, f ) } B1 AND ( E3 = E4 ) { ( t, = ), ( f, = ), ( t, < ), ( t, > ) } ( E1 > E2 ) AND ( E3 = E4 ) { ( >, = ), ( =, = ), ( <, = ), ( >, < ), ( >, > ) } 7/21

14.5 Control Structure Testing Data Flow Testing: selects test paths according to the locations of variable definitions and uses in the program DEF(S) = { X| S contains a definition of X }; USE(S) = { X| S contains a use of X }. DU chain of X = [X, S, S’] where XDEF(S) and XUSE(S’), and the definition of X in S is live at S’ ---- the flow of X. Every DU chain is covered once. 8/21

14.5 Control Structure Testing Loop Testing Unstructured Loops Concatenated Loops 0,1 Nested Loops Simple loop 1 2 2<m<n n-1 n n+1 9/21

Definition: Black-box or Behavioral Testing 14.6 Black-box Testing Definition: Black-box or Behavioral Testing Knowing the specified function a product is to perform and demonstrating correct operation based solely on its specification without regard for its internal logic – testing in the large. Questions How is functional validity tested? How is system behavior and performance tested? What classes of input will make good test cases? Is the system particularly sensitive to certain input values? How are the boundaries of a data class isolated? What data rates and data volume can the system tolerate? What effect will specific combinations of data have on system operation? 10/21

Equivalence Partitioning Boundary Value Analysis 14.6 Black-box Testing Techniques Graph-Based Testing Equivalence Partitioning Boundary Value Analysis Orthogonal Array Testing 11/21

14.6 Black-box Testing Guidelines Equivalence Partitioning An equivalence class represents a set of valid or invalid states for input conditions, so that there is no particular reason to choose one element over another as a class representative. Guidelines If an input condition specifies a range, one valid and two invalid equivalence classes are defined. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined. If an input condition specifies a member of a set, one valid and one invalid equivalence classes are defined. If an input condition is Boolean, one valid and one invalid equivalence classes are defined. 12/21

14.6 Black-box Testing Guidelines Boundary Value Analysis (BVA) "Bugs lurk in corners and congregate at boundaries ..." Boris Beizer Guidelines If input condition specifies a range bounded by values a and b, test cases should include a and b, values just above and just below a and b. If an input condition specifies and number of values, test cases should be exercise the minimum and maximum numbers, as well as values just above and just below the minimum and maximum values. Apply guidelines 1 and 2 to output conditions, test cases should be designed to produce the minimum and maxim output reports. If internal program data structures have boundaries (e.g., size limitations), be certain to test the boundaries. 13/21

14.7 Object-Oriented Testing Methods Test Case Design 1. Each test case should be uniquely identified and should be explicitly associated with the class to be tested, 2. The purpose of the test should be stated, 3. A list of testing steps should be developed for each test and should contain [BER94]: a. a list of specified states for the object that is to be tested b. a list of messages and operations that will be exercised as a consequence of the test c. a list of exceptions that may occur as the object is tested d. a list of external conditions e. supplementary information that will aid in understanding or implementing the test. Unlike conventional test case design, which is driven by an I-P-O view of software or the algorithmic detail of individual modules, OO testing focuses on designing appropriate sequences of operations to exercise the states of a class. 14/21

14.7 Object-Oriented Testing Methods Fault-Based Testing The tester looks for plausible faults (i.e., aspects of the implementation of the system that may result in defects). To determine whether these faults exist, test cases are designed to exercise the design or code. Class Testing and the Class Hierarchy Inheritance does not obviate the need for thorough testing of all derived classes. In fact, it can actually complicate the testing process. Scenario-Based Test Design Scenario-based testing concentrates on what the user does, not what the product does. This means capturing the tasks (via use-cases) that the user has to perform, then applying them and their variants as tests. 15/21

14.8 Class Level Testing Methods Random testing identify operations applicable to a class define constraints on their use identify a minimum test sequence an operation sequence that defines the minimum life history of the class (object) generate a variety of random (but valid) test sequences exercise other (more complex) class instance life histories  Requires large number of data permutations and combinations and can be inefficient. 16/21

14.8 Class Level Testing Methods Partition Testing reduces the number of test cases required to test a class in much the same way as equivalence partitioning for conventional software state-based partitioning categorize and test operations based on their ability to change the state of a class attribute-based partitioning categorize and test operations based on the attributes that they use category-based partitioning categorize and test operations based on the generic function each performs 17/21

14.9 Inter-class Test Case Design Multiple Class Testing (random) For each client class, use the list of class operators to generate a series of random test sequences. The operators will send messages to other server classes. For each message that is generated, determine the collaborator class and the corresponding operator in the server object. For each operator in the server object (that has been invoked by messages sent from the client object), determine the messages that it transmits. For each of the messages, determine the next level of operators that are invoked and incorporate these into the test sequence. 18/21

14.9 Inter-class Test Case Design Behavior Testing – The tests to be designed should achieve all state coverage [KIR94]. e m p t y a c o n s u A d i ( l ) w r k g h f b I F 1 4.12 S [ K R 9 4 ] 19/21

14.10 Specialized Testing 14.11 Testing Patterns Graphical User Interfaces Client/Server Architectures Documentation and Help Facilities Real-Time Systems 14.11 Testing Patterns Define the patterns and REUSE 20/21

Final Words About Testing Testing never ends, it just gets transferred from you [ the software engineer ] to your customer. Every time your customer uses the program, a test is being conducted. 21/21