Control Structure Testing Some Alternative Techniques This should follow Testing_Flow Copyright © 2016-2018 Curt Hill

Introduction The purpose of the Cyclomatic Complexity metric was to: Put an upper bound on basis path needed tests Give a measure of module complexity There are other methods for determining tests These all revolve around control flow This presentation considers some of them Copyright © 2016-2018 Curt Hill

Motivation Basis path testing is satisfied that every statement executed This may not be sufficient Control flow is one of the places bugs hide By examining closely all of these we should find a host of other bugs Some of the alternatives to basis path testing should cover those test cases and provide others that are useful Copyright © 2016-2018 Curt Hill

Condition Testing The focus of this approach is to examine predicates We look specifically at relational operations (comparisons) and Boolean operators Thus, in C we only examine the parentheses following an if, while or switch These are the units that drive all of the control flow of the module Copyright © 2016-2018 Curt Hill

Boolean/Relational Errors The Boolean result that drives our control flow may have several possible errors These may be in a: Comparison Boolean operator Arithmetic We then generate test cases looking for the possible errors Copyright © 2016-2018 Curt Hill

Error Types Arithmetic expression error Relational operator error Usually an equal left out or put in, but sometimes a reversal of less/greater Boolean operator Wrong, extra or missing operator Wrong Boolean variable Precedence error Wrong or missing parenthesis Copyright © 2016-2018 Curt Hill

Test Case Generation We set up the test data considering these errors When done systematically we generally get more complete coverage than basis path testing The computation of Cyclomatic Complexity or the construction of a flow graph is not needed Copyright © 2016-2018 Curt Hill

Branch Testing A form of condition testing For every Boolean affecting flow we must generate tests that force the Boolean to be true and false Consider: if(a==b && c<d) then four tests are required – each comparison to be true or false in every combination An if or while with N conditions needs 2N tests Copyright © 2016-2018 Curt Hill

Domain Testing Includes branch testing and but increases the test cases for comparisons For any comparison such as a>b we should test three times a<b a=b a>b This should find relational errors Copyright © 2016-2018 Curt Hill

Combinatorics It does not take too many decision statements to make the number of tests large Whenever we can we do wish to reduce the number of tests without reducing effectiveness We typically do this by limiting the tests in a way similar to short circuit evaluation Copyright © 2016-2018 Curt Hill

Limits Suppose we have an if with a condition of: a && b where a and b are comparisons of Boolean variables We would normally expect to do 4 tests We may be able to omit the test where both a and b is false without losing effective coverage Copyright © 2016-2018 Curt Hill

Example Consider the following: if(x>4 || y < 2) Each variable > = < We could generate 9 tests (x = 5, y = 1) => t, t, t (x = 5, y = 2) => t, f, t (x = 5, y = 3) => t, f, t (x = 4, y = 1) => f, t, t (x = 4, y = 2) => f, f, f (x = 4, y = 3) => f, f, f (x = 3, y = 1) => f, t, t (x = 3, y = 2) => f, f, f (x = 3, y = 3) => f, f, f Copyright © 2016-2018 Curt Hill

Data Flow Analysis In data flow testing we consider the definition and access of variables along a path Start with a flow graph and construct definition and use chains Data flow analysis should lead us to develop test cases Often better than basis path testing with fewer tests than every path This is covered more fully in its own presentation Copyright © 2016-2018 Curt Hill

Static Analysis Data flow analysis is a form of static analysis We know that static analysis cannot tell us the whole story There may be error paths that are not possible to actually execute Arrays and dynamic data structures are problematic because of one name with many values Copyright © 2016-2018 Curt Hill

Loop testing The whole point of computer software is looping It is the ability to repeat some action very many times that makes programs useful Unfortunately, bugs like to hide in and around loops Loop testing is a white box technique to exterminate some of those bugs Copyright © 2016-2018 Curt Hill

Common Loop Errors The most common loop error is the off by one error We want to go through M times and instead went through M-1 or M+1 Usually the condition has an extra equal or needs an equal or reversed condition Uninitialized loop control variable Often skips the loop Loop control variable not changed in a loop Usually gives an infinite loop Copyright © 2016-2018 Curt Hill

Common Loop Errors Of those only the off by one has much of a chance of making it to unit testing The others are usually detected in initial or ad hoc testing Therefore we need tests that will detect the off by one loop Copyright © 2016-2018 Curt Hill

Loop Structures There are about four categories of loop Simple Nested The approach depends on the category Simple Nested Concatenated Unstructured Copyright © 2016-2018 Curt Hill

Simple Loop Just what you might think One loop construct Single entry, single exit Not in or adjacent to another loop From a testing viewpoint you want tests that exercise: Skipping the loop completely If that is possible Executing the loop once Normal loop counts Looking for off by one Copyright © 2016-2018 Curt Hill

Nested A loop in a loop Strategy: Perhaps several deep Strategy: Minimize outer loops and test the innermost loop like before Work outward until outermost loop is tested Combinatorics may prevent us to applying the simple loop strategy Copyright © 2016-2018 Curt Hill

Concatenated Loops Two loops in succession Independent loops can just use simple loop strategy Related loops have something in common – often the control variable These need a variation of the nested loop strategy Verify the coupling between the two loops is properly handled Copyright © 2016-2018 Curt Hill

Unstructured Not the problem it once was Characterized by either multiple entrances or multiple exits Multiple entrances Typically a GOTO has bypassed the loop initialization – hard to do otherwise Multiple exits Done with GOTOs, breaks, throws and returns Copyright © 2016-2018 Curt Hill

Unstructured Again The multiple entrance, multiple exit loop is the most insidious control structure known Extremely hard to analyze and test Don’t do it Rewrite into something structured The multiple entrances are much worse than multiple exits Breaks are most benign GOTOs are the worst Returns are somewhere in between Copyright © 2016-2018 Curt Hill

Finally Black box and white box testing complement each other Use both Often a black box test will cover one or more white box tests For time’s sake eliminate the redundant tests Copyright © 2016-2018 Curt Hill