Introduction to Software Testing Chapter 3.1 Logic Coverage Paul Ammann & Jeff Offutt

2 Covering Logic Expressions  Logic expressions show up in many situations  Covering logic expressions is required by the US Federal Aviation Administration for safety critical software  Logical expressions can come from many sources – Decisions in programs – FSMs and statecharts – Requirements  Tests are intended to choose some subset of the total number of truth assignments to the expressions

3 Logic Predicates and Clauses  A predicate is an expression that evaluates to a boolean value  Predicates can contain – boolean variables – non-boolean variables that contain >, =, <=, != – boolean function calls  Internal structure is created by logical operators – ¬ – the negation operator –  – the and operator –  – the or operator –  – the implication operator –  – the exclusive or operator –  – the equivalence operator  A clause is a predicate with no logical operators

4 Example (a = n*o) Includes four clauses: (a < b) – relational expression f (z) – boolean-valued function D – boolean variable (m >= n*o) – relational expression  Most predicates have few clauses – It would be nice to quantify that claim!  Sources of predicates – Decisions in programs – Guards in finite state machines – Decisions in UML activity graphs – Requirements, both formal and informal – SQL queries

Problem: List all the clauses for the predicate below: ((f 0))  (M  (e < d + c)) Solution: There are four: f 0, M, and e < d + c.

Problem: 2 Write the predicate to represent the requirement: List all the wireless mice that either retail for more than $100 or for which the store has more than 20 items. Also list non-wireless mice that retail for more than $50. The predicate describing whether to list a given mouse is: ((mouseType = wireless)  ((retail > 100)  (stock > 20)))  (  (mouseType = wireless)  (retail > 50))

Clause  A clause is a predicate that does not contain any of the logical operators.  For example, the predicate (a = b) ∨ C ∧ p(x) contains three clauses: a relational expression (a = b), a boolean variable C the function call p(x).  Because they may contain a structure of their own, relational expressions require special treatment.

The Predicate A predicate may be written in a variety of logically equivalent ways. For example, the predicate ((a = b) ∨ C) ∧ ((a = b) ∨ p(x)) is logically equivalent to the predicate given in the previous slide  but ((a = b) ∧ p(x)) ∨ (C ∧ p(x)) is not.  The usual rules of boolean algebra may be used to convert boolean expressions into equivalent forms.

Testing and Covering Predicates  We use predicates in testing as follows : – Developing a model of the software as one or more predicates – Requiring tests to satisfy some combination of clauses  Abbreviations: – P is the set of predicates – p is a single predicate in P – C is the set of clauses in P – C p is the set of clauses in predicate p – c is a single clause in C Introduction to Software Testing (Ch 3) © Ammann & Offutt9

10 Predicate and Clause Coverage  Each predicate and each clause be evaluated to both true and false Predicate Coverage (PC) : For each p in P, TR contains two requirements: p evaluates to true, and p evaluates to false. Clause Coverage (CC) : For each c in C, TR contains two requirements: c evaluates to true, and c evaluates to false.

Predicate Coverage ((a > b) ∨ C) ∧ p(x) two tests that satisfy predicate coverage are (a = 5, b = 4, C = true, p(x) = true) and (a = 5, b = 6, C = false, p(x) = false). Predicate coverage for the above clause could also be satisfied with the two tests  (a = 5, b = 4, C = true, p(x) = true) and  ( a = 5, b = 4, C = true, p(x) = false) the first two clauses, a = 5, b = 4, never have the value false!

Clause Coverage  Predicate ((a > b) ∨ C) ∧ p(x) requires different values to satisfy Clause Coverage Clause coverage requires that (a > b) = true and false, C = true and false, p(x) = true and false. These requirements can be satisfied with two tests: ((a =5, b = 4), (C = true), p(x) = true) and ((a = 5, b = 6), (C = false), p(x) = false).

Active Clause Coverage p= a  b ends up with total four requirements in TR. Two for clause a and two for clause b.  For clause a, a determines p if and only if b is false. So we have two test requirements: (a= true, b= false), (a= false, b= false)  For clause b, b determines p if and only if a is false. So we have two requirements (a = false, b= true), (a= false, b= false)) This is summarized in the partial truth table (bold face)

 Clause coverage does not subsume predicate coverage,  Predicate coverage does not subsume clause coverage,  Example: The predicate p = a ∨ b. The clauses C are {a, b}. The four test inputs that enumerate the combinations of logical values for the clauses:  Test set T 23 = {2, 3} satisfies clause coverage, but not predicate coverage, because p is never false.  Test set T 24 = {2, 4} satisfies predicate coverage, but not clause coverage, because b is never true. FINALLY: These two test sets demonstrate that neither predicate coverage nor clause coverage subsumes the other.

15 Predicate Coverage Another Example ((a = n*o) predicate coverage Predicate = true a = 5, b = 10, D = true, m = 1, n = 1, o = 1 = (5 = 1*1) = true  true  TRUE = true Predicate = false a = 10, b = 5, D = false, m = 1, n = 1, o = 1 = (10 = 1*1) = false  false  TRUE = false

Introduction to Software Testing (Ch 3) © Ammann & Offutt 16 Two tests (a < b) = true a = 5, b = 10 (a < b) = false a = 10, b = 5 D = true D = false m >= n*o = true m = 1, n = 1, o = 1 m >= n*o = false m = 1, n = 2, o = 2 true cases 1) a = 5, b = 10, D = true, m = 1, n = 1, o = 1 false cases 2) a = 10, b = 5, D = false, m = 1, n = 2, o = 2 ((a = n*o) Clause coverage

Problems with Predicate and Clause Coverage Introduction to Software Testing (Ch 3) © Ammann & Offutt 17  Predicate Coverage does not fully exercise all the clauses, especially in the presence of short circuit evaluation  Clause Coverage does not always ensure Predicate Coverage –That is, we can satisfy Clause Coverage without causing the predicate to be both true and false –This is definitely not what we want !  The simplest solution is to test all combinations

18 Combinatorial Coverage Combinatorial Coverage requires every possible combination Sometimes called Multiple Condition Coverage Combinatorial Coverage (CoC) : For each p in P, TR has test requirements for the clauses in Cp to evaluate to each possible combination of truth values. a < bDm >= n*o ((a = n*o) 1TTTT 2TTFF 3TFTT 4TFFF 5FTTT 6FTFF 7FFTF 8FFFF

Combinatorial Coverage A predicate p with n independent clauses has 2 n possible assignments of truth values.  The general idea is simple and comprehensive  Combinatorial coverage is impractical for predicates with more than a few clauses.  A powerful collection of test criteria that are based on the notion of making individual clauses “active”  We check to see that if we vary a clause in a situation where the clause should affect the predicate. Ifact, the clause does affect the predicate.  Finally: Getting the details right is hard What exactly does “independently” mean ? This idea is simplified as making clauses active … 19

Determination  A clause c i in predicate p, called the major clause, determines p if and only if the values of the remaining minor clauses c j are such that changing c i changes the value of p  This is considered to make the clause active

Determining Predicates 21  Goal : Find tests for each clause when the clause determines the value of the predicate P = A  B if B = true, p is always true. so if B = false, A determines p. if A = false, B determines p. P = A  B if B = false, p is always false. so if B = true, A determines p. if A = true, B determines p.

22 p = a  b 1)a = true, b = false 2)a = false, b = false 3)a = false, b = true 4)a = false, b = false Active Clause Coverage  Ambiguity : Do the minor clauses have to have the same values when the major clause is true and false? Active Clause Coverage (ACC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. Duplicate a is major clauseb is major clause

23 Example : p = (a ∨ b) ∧ c p = (a ∨ b) ∧ c Finding values for predicate coverage is easy and was already shown in Section 3.2. Two test requirements are TR PC = {p = true, p = false} Finding values for predicate coverage is easy. Test Requiremet PC = {p = true, p = false} and they can be satisfied with the following values for the clauses: Suppose that clauses a, b, and c were defined in terms of Java program variables as follows The complete expanded predicate is : p = (x < y ∨ done) ∧ list.contains(str)

24 Test Requirements for Predicate Coverage.  The values for the program variables need not be the same in a particular test case if the goal is to set a clause to a particular value.  For example clause a is true in both tests, even though program variables x and y have different values.

25. The Same Example as Clause Coverage TR = {a = true, a = false,b = true, b = false, c = true, c = false} They can be satisfied with the following values for the clauses (blank cells represent are « don’t-care values»):

26  The test requirements can be satisfied with the following table for the clauses.  These can be the same as with clause coverage The exception is that the blank cells from clause coverage are replaced with the values  Values for major clauses are indicated with upper case letters in boldface. The duplication; the first and fifth rows are identical, and the second and fourth are identical. Four tests General Active Clause Cov. Four tests are needed to satisfy General Active Clause Cov.