1. Introduction to Software Testing Chapter 8.1 Logic Coverage Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwaretest/

2. Ch. 8 : Logic Coverage Introduction to Software Testing, Edition 2 (Ch 07) © Ammann & Offutt 2 Four Structures for Modeling Software GraphsLogic Input Space Syntax Input Models Integ Source Applied to DNF Specs FSMs Source Use cases Specs Design Source Applied to

3. Semantic Logic Criteria (8.1) Logic expressions show up in many situations Covering logic expressions is required by the US Federal Aviation Administration for safety critical software Logical expressions can come from many sources –Decisions in programs –FSMs and statecharts –Requirements Tests are intended to choose some subset of the total number of truth assignments to the expressions Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 3

4. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 4 Logic Predicates and Clauses A predicate is an expression that evaluates to a boolean value Predicates can contain –boolean variables –non-boolean variables that contain >, =, <=, != –boolean function calls Internal structure is created by logical operators –¬ – the negation operator –  – the and operator –  – the or operator –  – the implication operator –  – the exclusive or operator –  – the equivalence operator A clause is a predicate with no logical operators

5. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 5 Examples (a = n*o) Four clauses: –(a < b) – relational expression – f (z) – boolean-valued function – D – boolean variable – (m >= n*o) – relational expression Most predicates have few clauses Sources of predicates –Decisions in programs –Guards in finite state machines –Decisions in UML activity graphs –Requirements, both formal and informal –SQL queries

6. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 6 Humans have trouble translating from English to Logic Translating from English “I am interested in SWE 637 and CS 652” course = swe637 OR course = cs652 “If you leave before 6:30 AM, take Braddock to 495, if you leave after 7:00 AM, take Prosperity to 50, then 50 to 495” (time 7:00  path = Prosperity) Hmm … this is incomplete ! (time = 6:30  path = Prosperity)

7. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 7 Logic Coverage Criteria (8.1.1) We use predicates in testing as follows : –Developing a model of the software as one or more predicates –Requiring tests to satisfy some combination of clauses Abbreviations: –P is the set of predicates –p is a single predicate in P –C is the set of clauses in P –C p is the set of clauses in predicate p –c is a single clause in C

8. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 8 Predicate and Clause Coverage The first (and simplest) two criteria require that each predicate and each clause be evaluated to both true and false Predicate Coverage (PC) : For each p in P, TR contains two requirements: p evaluates to true, and p evaluates to false. Clause Coverage (CC) : For each c in C, TR contains two requirements: c evaluates to true, and c evaluates to false. When predicates come from conditions on edges, this is equivalent to edge coverage PC does not evaluate all the clauses, so …

9. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 9 Predicate Coverage Example ((a = n*o) predicate coverage Predicate = true a = 5, b = 10, D = true, m = 1, n = 1, o = 1 = (5 = 1*1) = true  true  TRUE = true Predicate = false a = 10, b = 5, D = false, m = 1, n = 1, o = 1 = (10 = 1*1) = false  false  TRUE = false

10. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 10 Clause Coverage Example ((a = n*o) Clause coverage Two tests (a < b) = true a = 5, b = 10 (a < b) = false a = 10, b = 5 D = true D = false m >= n*o = true m = 1, n = 1, o = 1 m >= n*o = false m = 1, n = 2, o = 2 true cases 1) a = 5, b = 10, D = true, m = 1, n = 1, o = 1 false cases 2) a = 10, b = 5, D = false, m = 1, n = 2, o = 2

11. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 11 Problems with PC and CC PC does not fully exercise all the clauses, especially in the presence of short circuit evaluation CC does not always ensure PC –That is, we can satisfy CC without causing the predicate to be both true and false –This is definitely not what we want ! The simplest solution is to test all combinations …

12. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 12 Combinatorial Coverage CoC requires every possible combination Sometimes called Multiple Condition Coverage Combinatorial Coverage (CoC) : For each p in P, TR has test requirements for the clauses in Cp to evaluate to each possible combination of truth values. a < bDm >= n*o ((a = n*o) 1TTTT 2TTFF 3TFTT 4TFFF 5FTTT 6FTFF 7FFTF 8FFFF

13. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 13 Combinatorial Coverage This is simple, neat, clean, and comprehensive … But quite expensive! 2 N tests, where N is the number of clauses –Impractical for predicates with more than 3 or 4 clauses The literature has lots of suggestions – some confusing The general idea is simple: Test each clause independently from the other clauses Getting the details right is hard What exactly does “independently” mean ? The book presents this idea as “making clauses active” …

14. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 14 Active Clauses (8.1.2) Clause coverage has a weakness : The values do not always make a difference Consider the first test for clause coverage, which caused each clause to be true: –(5 = 1*1) Only the first clause counts ! To really test the results of a clause, the clause should be the determining factor in the value of the predicate Determination : A clause c i in predicate p, called the major clause, determines p if and only if the values of the remaining minor clauses c j are such that changing c i changes the value of p This is considered to make the clause active

15. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 15 Determining Predicates Goal : Find tests for each clause when the clause determines the value of the predicate This is formalized in a family of criteria that have subtle, but very important, differences P = A  B if B = true, p is always true. so if B = false, A determines p. if A = false, B determines p. P = A  B if B = false, p is always false. so if B = true, A determines p. if A = true, B determines p.

16. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 16 p = a  b 1)a = true, b = false 2)a = false, b = false 3)a = false, b = true 4)a = false, b = false Active Clause Coverage This is a form of MCDC, which is required by the FAA for safety critical software Ambiguity : Do the minor clauses have to have the same values when the major clause is true and false? Active Clause Coverage (ACC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. Duplicate a is major clauseb is major clause

17. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 17 Resolving the Ambiguity This question caused confusion among testers for years Considering this carefully leads to three separate criteria : –Minor clauses do not need to be the same –Minor clauses do need to be the same –Minor clauses force the predicate to become both true and false p = a  (b  c) Major clause : a a = true, b = false, c = true a = false, b = false, c = false c = false Is this allowed ?

18. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 18 General Active Clause Coverage This is complicated ! It is possible to satisfy GACC without satisfying predicate coverage We really want to cause predicates to be both true and false ! General Active Clause Coverage (GACC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that ci determines p. TR has two requirements for each ci : c i evaluates to true and c i evaluates to false. The values chosen for the minor clauses c j do not need to be the same when c i is true as when c i is false, that is, c j (c i = true) = c j (c i = false) for all c j OR c j (c i = true) != c j (c i = false) for all c j.

19. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 19 Restricted Active Clause Coverage This has been a common interpretation by aviation developers RACC often leads to infeasible test requirements There is no logical reason for such a restriction Restricted Active Clause Coverage (RACC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. The values chosen for the minor clauses cj must be the same when c i is true as when c i is false, that is, it is required that c j (c i = true) = c j (c i = false) for all c j.

20. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 20 Correlated Active Clause Coverage A more recent interpretation Implicitly allows minor clauses to have different values Explicitly satisfies (subsumes) predicate coverage Correlated Active Clause Coverage (CACC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. The values chosen for the minor clauses c j must cause p to be true for one value of the major clause c i and false for the other, that is, it is required that p(c i = true) != p(c i = false).

21. CACC and RACC Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 21 abc a  (b  c) 1TTTT 2TTFT 3TFTT 4TFFF 5FTTF 6FTFF 7FFTF 8FFFF TTTTTTTT FFFFFFFF a major clause P a : b=true or c = true CACC can be satisfied by choosing any of rows 1, 2, 3 AND any of rows 5, 6, 7 – a total of nine pairs abc a  (b  c) 1TTTT 2TTFT 3TFTT 4TFFF 5FTTF 6FTFF 7FFTF 8FFFF TTTTTTTT FFFFFFFF a RACC can only be satisfied by row pairs (1, 5), (2, 6), or (3, 7) Only three pairs

22. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 22 Inactive Clause Coverage (8.1.3) The active clause coverage criteria ensure that “major” clauses do affect the predicates Inactive clause coverage takes the opposite approach – major clauses do not affect the predicates Inactive Clause Coverage (ICC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i does not determine p. TR has four requirements for each c i : (1) c i evaluates to true with p true, (2) c i evaluates to false with p true, (3) c i evaluates to true with p false, and (4) c i evaluates to false with p false.

23. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 23 General and Restricted ICC Unlike ACC, the notion of correlation is not relevant –ci does not determine p, so cannot correlate with p Predicate coverage is always guaranteed General Inactive Clause Coverage (GICC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i does not determine p. The values chosen for the minor clauses c j do not need to be the same when c i is true as when c i is false, that is, c j (c i = true) = c j (c i = false) for all c j OR c j (c i = true) != c j (c i = false) for all c j. Restricted Inactive Clause Coverage (RICC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i does not determine p. The values chosen for the minor clauses cj must be the same when c i is true as when c i is false, that is, it is required that c j (c i = true) = c j (c i = false) for all c j.

24. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 24 Infeasibility & Subsumption (8.1.4) Consider the predicate: (a > b  b > c)  c > a (a > b) = true, (b > c) = true, (c > a) = true is infeasible As with graph-based criteria, infeasible test requirements have to be recognized and ignored Recognizing infeasible test requirements is hard, and in general, undecidable Software testing is inexact – engineering, not science

25. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 25 Logic Criteria Subsumption Clause Coverage CC Predicate Coverage PC Combinatorial Clause Coverage COC Restricted Active Clause Coverage RACC Restricted Inactive Clause Coverage RICC General Active Clause Coverage GACC Correlated Active Clause Coverage CACC General Inactive Clause Coverage GICC

26. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 26 Making Clauses Determine a Predicate Finding values for minor clauses c j is easy for simple predicates But how to find values for more complicated predicates ? Definitional approach: –p c=true is predicate p with every occurrence of c replaced by true –p c=false is predicate p with every occurrence of c replaced by false To find values for the minor clauses, connect p c=true and p c=false with exclusive OR p c = p c=true  p c=false After solving, p c describes exactly the values needed for c to determine p (8.1.5)

27. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 27 Examples p = a  b p a = p a=true  p a=false = (true  b) XOR (false  b) = true XOR b = ¬ b p = a  b p a = p a=true  p a=false = (true  b)  (false  b) = b  false = b p = a  (b  c) p a = p a=true  p a=false = (true  (b  c))  (false  (b  c)) = true  (b  c) = ¬ (b  c) = ¬ b  ¬ c “NOT b  NOT c” means either b or c can be false RACC requires the same choice for both values of a, CACC does not

28. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 28 Repeated Variables The definitions in this chapter yield the same tests no matter how the predicate is expressed (a  b)  (c  b) == (a  c)  b (a  b)  (b  c)  (a  c) –Only has 8 possible tests, not 64 Use the simplest form of the predicate, and ignore contradictory truth table assignments

29. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 29 A More Subtle Example p = ( a  b )  ( a  ¬ b) p a = p a=true  p a=false = ((true  b)  (true  ¬ b))  ((false  b)  (false  ¬ b)) = (b  ¬ b)  false = true  false = true a always determines the value of this predicate b never determines the value – b is irrelevant ! p = ( a  b )  ( a  ¬ b) p b = p b=true  p b=false = ((a  true)  (a  ¬ true))  ((a  false)  (a  ¬ false)) = (a  false)  (false  a) = a  a = false

30. b & c are the same, a differs, and p differs … thus TTT and FTT cause a to determine the value of p Again, b & c are the same, so TTF and FTF cause a to determine the value of p Tabular Method for Determination The math sometimes gets complicated A truth table can sometimes be simpler Example Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 30 abc a  (b  c) 1TTTT 2TTFT 3TFTT 4TFFF 5FTTF 6FTFF 7FFTF 8FFFF papa pbpb pcpc In sum, three separate pairs of rows can cause a to determine the value of p, and only one pair each for b and c Finally, this third pair, TFT and FFT, also cause a to determine the value of p For clause b, only one pair, TTF and TFF cause b to determine the value of p Likewise, for clause c, only one pair, TFT and TFF, cause c to determine the value of p

31. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 31 Logic Coverage Summary Predicates are often very simple—in practice, most have less than 3 clauses –In fact, most predicates only have one clause ! –With only clause, PC is enough –With 2 or 3 clauses, CoC is practical –Advantages of ACC and ICC criteria significant for large predicates CoC is impractical for predicates with many clauses Control software often has many complicated predicates, with lots of clauses Question … why don’t complexity metrics count the number of clauses in predicates?

32. Disjunctive Normal Form Common Representation for Boolean Functions –Slightly Different Notation for Operators –Slightly Different Terminology Basics: –A literal is a clause or the negation (overstrike) of a clause Examples: a, a –A term is a set of literals connected by logical “and” “and” is denoted by adjacency instead of  Examples: ab, ab, ab for a  b, a  ¬ b, ¬ a  ¬ b –A (disjunctive normal form) predicate is a set of terms connected by “or” “or” is denoted by + instead of  Examples: abc + ab + ac Terms are also called “implicants” –If a term is true, that implies the predicate is true Introduction to Software Testing, Edition 2 (Ch 8) 32 © Ammann & Offutt

33. Implicant Coverage (8.2.1) Obvious coverage idea : Make each implicant evaluate to “true” –Problem : Only tests “true” cases for the predicate –Solution : Include DNF representations for negation Implicant Coverage (IC) : Given DNF representations of a predicate f and its negation f, for each implicant in f and f, TR contains the requirement that the implicant evaluate to true. Example: f = ab + bc f = b + ac Implicants: { ab, bc, b, ac } Possible test set: {TTF, FFT} Observation: IC is relatively weak Introduction to Software Testing, Edition 2 (Ch 8) 33 © Ammann & Offutt

34. Improving on Implicant Coverage Additional Definitions : –A proper subterm is a term with one or more clauses removed Example: abc has 6 proper subterms: a, b, c, ab, ac, bc –A prime implicant is an implicant such that no proper subterm is also an implicant Example: f = ab + abc Implicant ab is a prime implicant Implicant abc is not a prime implicant (due to proper subterm ac) –A redundant implicant is an implicant that can be removed without changing the value of the predicate Example: f = ab + ac + bc ab is redundant Predicate can be written: ac + bc Introduction to Software Testing, Edition 2 (Ch 8) 34 © Ammann & Offutt (8.2.2)

35. Unique True Points A minimal DNF representation is one with only prime, non- redundant implicants A unique true point with respect to a given implicant is an assignment of truth values so that –The given implicant is true, and –All other implicants are false A unique true point test focuses on just one implicant A minimal representation guarantees the existence of at least one unique true point for each implicant Multiple Unique True Point Coverage (MUTP) : Given minimal DNF representations of a predicate f, for each implicant i, choose unique true points (UTPs) such that clauses not in i take on values T and F. Introduction to Software Testing, Edition 2 (Ch 8) 35 © Ammann & Offutt

36. Unique True Point Example Consider again : f = ab + bc –Implicants : { ab, bc, b, ac } –Each of these implicants is prime –None of these implicants is redundant Unique true points : –ab: {TTT} –bc: {FTF} –b: {FFF, TFF, TFT} –ac: {FTT} Note: One possible (minimal) test set satisfying MUTP –For third clause, need first and third tests, but not second –Note that MUTP is infeasible for the other three clauses not enough UTPs for clauses not in implicant to take on all truth values Introduction to Software Testing, Edition 2 (Ch 8) 36 © Ammann & Offutt

37. Near False Points (8.2.3) A near false point with respect to a clause c in implicant i is an assignment of truth values such that f is false, but if c is negated (and all other clauses left as is), i (and hence f) evaluates to true Relation to determination: at a near false point, c determines f –Hence we should expect relationship to ACC criteria Note that definition only mentions f, and not f Clearly, CUTPNFP subsumes RACC Unique True Point and Near False Point Pair Coverage (CUTPNFP) : Given a minimal DNF representation of a predicate f, for each clause c in each implicant i, TR contains a unique true point for i and a near false point for c such that the points differ only in the truth value of c. Introduction to Software Testing, Edition 2 (Ch 8) 37 © Ammann & Offutt

38. CUTPNFP Example Consider f = ab + cd –Implicant ab has 3 unique true points : {TTFF, TTFT, TTTF} For clause a, we can pair unique true point TTFF with near false point FTFF For clause b, we can pair unique true point TTFF with near false point TFFF –Implicant cd has 3 unique true points : {FFTT, FTTT, TFTT} For clause c, we can pair unique true point FFTT with near false point FFFT For clause d, we can pair unique true point FFTT with near false point FFTF CUTPNFP set : {TTFF, FFTT, TFFF, FTFF, FFTF, FFFT} –First two tests are unique true points; others are near false points Rough number of tests required: # implicants * # literals Introduction to Software Testing, Edition 2 (Ch 8) 38 © Ammann & Offutt

39. The MNFP and MUMCUT Criteria (8.2.3) The next two criteria provide enough scaffolding to make guarantees about fault detection (see later slides) Multiple Near False Point Coverage (MNFP) : Given a minimal DNF representation of a predicate f, for each literal c in each implicant i, TR choose near false points (NFPs) such that clauses not in i take on values T and F. Introduction to Software Testing, Edition 2 (Ch 8) 39 © Ammann & Offutt MUMCUT : Given a minimal DNF representation of a predicate f, apply MUTP, CUTPNFP, and MNFP.

40. DNF Fault Classes ENF: Expression Negation Fault f = ab+c f’ = ab+c TNF: Term Negation Fault f = ab+c f’ = ab+c TOF: Term Omission Fault f = ab+c f’ = ab LNF: Literal Negation Fault f = ab+c f’ = ab+c LRF: Literal Reference Fault f = ab + bcd f’ = ad + bcd LOF: Literal Omission Fault f = ab + c f’ = a + c LIF: Literal Insertion Fault f = ab + c f’ = ab + bc ORF+: Operator Reference Fault f = ab + c f’ = abc ORF*: Operator Reference Fault f = ab + c f’ = a + b + c Key idea is that fault classes are related with respect to testing : Test sets guaranteed to detect certain faults are also guaranteed to detect additional faults Introduction to Software Testing, Edition 2 (Ch 8) 40 © Ammann & Offutt

41. Fault Detection Relationships Expression Negation Fault ENF Literal Insertion Fault LIF Term Omission Fault TOF Literal Reference Fault LRF Literal Negation Fault LNF Operator Reference Fault ORF+ Literal Omission Fault LOF Term Negation Fault TNF Operator Reference Fault ORF* Introduction to Software Testing, Edition 2 (Ch 8) 41 © Ammann & Offutt

42. Karnaugh Maps for Testing Logic Expressions Fair Warning –We use, rather than teach, Karnaugh Maps –Newcomers to Karnaugh Maps probably need a tutorial Suggestion: Google “Karnaugh Map Tutorial” Our goal: Apply Karnaugh Maps to concepts used to test logic expressions –Identify when a clause determines a predicate –Identify the negation of a predicate –Identify prime implicants and redundant implicants –Identify unique true points –Identify unique true point / near false point pairs No new material here on testing –Just fast shortcuts for concepts already presented Introduction to Software Testing, Edition 2 (Ch 8) 42 © Ammann & Offutt (8.2.4)

43. K-Map : A Clause Determines a Predicate Consider the predicate : f = b + ac + ac Suppose we want to identify when b determines f The dashed line highlights where b changes value –If two cells joined by the dashed line have different values for f, then b determines f for those two cells –b determines f: ac + ac (but NOT at ac or ac ) Repeat for clauses a and c ttt1 ttt0 10110100 ab c Introduction to Software Testing, Edition 2 (Ch 8) 43 © Ammann & Offutt

44. K-Map: Negation of a predicate Consider the predicate: f = ab + bc Draw the Karnaugh Map for the negation –Identify groups –Write down negation: f = b + a c tt1 t0 10110100 ab c tt1 ttt0 10110100 ab c Introduction to Software Testing, Edition 2 (Ch 8) 44 © Ammann & Offutt

45. K-Map: Prime and Redundant Implicants Consider the predicate: f = abc + abd + abcd + abcd + acd Draw the Karnaugh Map Implicants that are not prime: abd, abcd, abcd, acd Redundant implicant: abd Prime implicants –Three: ad, bcd, abc –The last is redundant –Minimal DNF representation f = ad + bcd 01 00 10 11 0100 ab cd t t tt t t 11 10 Introduction to Software Testing, Edition 2 (Ch 8) 45 © Ammann & Offutt

46. K-Map: Unique True Points Consider the predicate: f = ab + cd Three unique true points for ab –TTFF, TTFT, TTTF –TTTT is a true point, but not a unique true point Three unique true points for cd –FFTT, FTTT, TFTT Unique true points for f f = ac + bc + ad + bd –FTFT,TFFT, FTTF, TFTF 01 00 10110100 ab cd t t tt 11 10 t t t Introduction to Software Testing, Edition 2 (Ch 8) 46 © Ammann & Offutt

47. For each implicant find unique true points (UTPs) so that –Literals not in implicant take on values T and F Consider the DNF predicate: – f = ab + cd For implicant ab –Choose TTFT, TTTF For implicant cd –Choose FTTT, TFTT MUTP test set –{TTFT, TTTF, FTTT, TFTT} MUTP: Multiple Unique True Points 01 00 10110100 ab cd t t tt 11 10 t t t Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 47

48. CUTPNFP: Corresponding Unique True Point Near False Point Pairs Consider the DNF predicate: f = ab + cd For implicant ab –For a, choose UTP, NFP pair TTFF, FTFF –For b, choose UTP, NFP pair TTFT, TFFT For implicant cd –For c, choose UTP, NFP pair FFTT, FFFT –For d, choose UTP, NFP pair FFTT, FFTF Possible CUTPNFP test set –{TTFF, TTFT, FFTT //UTPs FTFF, TFFT, FFFT, FFTF} //NFPs 01 00 10110100 ab cd t t tt 11 10 t t t Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 48

49. Find NFP tests for each literal such that all literals not in the term attain F and T Consider the DNF predicate: – f = ab + cd For implicant ab –Choose FTFT, FTTF for a –Choose TFFT, TFTF for b For implicant cd –Choose FTFT, TFFT for c –Choose FTTF, TFTF for d MNFP test set –{TFTF, TFFT, FTTF, TFTF} Example is small, but generally MNFP is large MNFP : Multiple Near False Points 01 00 10110100 ab cd t t tt 11 10 t t t Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 49

50. 50 Minimal-MUMCUT Criterion Kaminski/Ammann (ICST 2009) Minimal-MUMCUT uses low level criterion feasibility analysis –Adds CUTPNFP and MNFP only when necessary Minimal-MUMCUT guarantees detecting LIF, LRF, LOF –And thus all 9 faults in the hierarchy CUTPNFP feasible? MNFP Test Set = MUTP + MNFP For Each Literal In Term Test Set = MUTP + CUTPNFP MUTP feasible? Test Set = MUTP + NFP For Each Term Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt

51. 51 Logic Expressions from Source Predicates are derived from decision statements In programs, most predicates have less than four clauses –Wise programmers actively strive to keep predicates simple When a predicate only has one clause, COC, ACC, ICC, and CC all collapse to predicate coverage (PC) Applying logic criteria to program source is hard because of reachability and controllability: –Reachability : Before applying the criteria on a predicate at a particular statement, we have to get to that statement –Controllability : We have to find input values that indirectly assign values to the variables in the predicates –Variables in the predicates that are not inputs to the program are called internal variables Illustrated through an example in the following slides … Introduction to Software Testing, Edition 2 (Ch 8)

52. Thermostat (pg 1 of 2) Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 52 1 // Jeff Offutt & Paul Ammann—September 2014 2 // Programmable Thermostat 6 import java.io.*; 10 public class Thermostat 11 { 12 private int curTemp; // Current temperature reading 13 private int thresholdDiff; // Temp difference until heater on 14 private int timeSinceLastRun; // Time since heater stopped 15 private int minLag; // How long I need to wait 16 private boolean Override; // Has user overridden the program 17 private int overTemp; // OverridingTemp 18 private int runTime; // output of turnHeaterOn–how long to run 19 private boolean heaterOn; // output of turnHeaterOn – whether to run 20 private Period period; // morning, day, evening, or night 21 private DayType day; // week day or weekend day 23 // Decide whether to turn the heater on, and for how long. 24 public boolean turnHeaterOn (ProgrammedSettings pSet) 25 {

53. Thermostat (pg 2 of 2) Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 53 26 int dTemp = pSet.getSetting (period, day); 28 if (((curTemp < dTemp - thresholdDiff) || 29 (Override && curTemp < overTemp - thresholdDiff)) && 30 (timeSinceLastRun > minLag)) 31 { // Turn on the heater 32 // How long? Assume 1 minute per degree (Fahrenheit) 33 int timeNeeded = curTemp - dTemp; 34 if (Override) 35 timeNeeded = curTemp - overTemp; 36 setRunTime (timeNeeded); 37 setHeaterOn (true); 38 return (true); 39 } 40 else 41 { 42 setHeaterOn (false); 43 return (false); 44 } 45 } // End turnHeaterOn The full class is in the book and on the book website.

54. © Ammann & Offutt 54 Two Thermostat Predicates 28-30 : (((curTemp < dTemp - thresholdDiff) || (Override && curTemp < overTemp - thresholdDiff)) && timeSinceLastRun > minLag)) 34 : (Override) Introduction to Software Testing, Edition 2 (Ch 8) Simplify a : curTemp < dTemp - thresholdDiff b : Override c : curTemp < overTemp - thresholdDiff d : timeSinceLastRun > minLag) 28-30 : (a || (b && c)) && d 34 : b

55. © Ammann & Offutt 55 Reachability for Thermostat Predicates 28-30 : True 34 : (a || (b && c)) && d curTemp < dTemp - thresholdDiff Introduction to Software Testing, Edition 2 (Ch 8) Need to solve for the internal variable dTemp pSet.getSetting (period, day); setSetting (Period. MORNING, DayType. WEEKDAY, 69); setPeriod (Period. MORNING ); setDay (DayType. WEEKDAY );

56. © Ammann & Offutt 56 Predicate Coverage (true) Introduction to Software Testing, Edition 2 (Ch 8) (a || (b && c)) && d a: curTemp < dTemp – thresholdDiff : true b: Override : true c: curTemp < overTemp – thresholdDiff : true d: timeSinceLastRun > (minLag) : true a : true b : true c : true d : true thermo = new Thermostat(); // Needed object settings = new ProgrammedSettings(); // Needed object settings.setSetting (Period.MORNING, DayType.WEEKDAY, 69); // dTemp thermo.setPeriod (Period.MORNING); // dTemp thermo.setDay (DayType.WEEKDAY); // dTemp thermo.setCurrentTemp (63); // clause a thermo.setThresholdDiff (5); // clause a thermo.setOverride (true); // clause b thermo.setOverTemp (70); // clause c thermo.setMinLag (10); // clause d thermo.setTimeSinceLastRun (12); // clause d assertTrue (thermo.turnHeaterOn (settings)); // Run test (8.3.1)

57. Correlated Active Clause Coverage Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 57 P a = ((a || (b && c)) && d)  ((a || (b && c)) && d) (T && d)  ((b && c) && d) ((T || (b && c)) && d)  ((F || (b && c)) && d) !(b && c) && d ( !b || !c ) && d Check with the logic coverage web app http://cs.gmu.edu:8080/offutt/coverage/LogicCoverage d  ((b && c) && d) (1 of 6) (8.3.3) T  ((b && c) && T)

58. Correlated Active Clause Coverage Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 58 (a || (b && c)) && d a b c d P a : T t f t F t f t P b : f T t t f F t t P c : f t T t f t F t P d : t t t T t t t F duplicates Six tests needed for CACC on Thermostat (2 of 6)

59. Correlated Active Clause Coverage Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 59 curTemp dTemp thresholdDiff a=t : curTemp < dTemp - thresholdDiff 63 69 5 a=f : !(curTemp < dTemp - thresholdDiff) 66 69 5 dTemp: settings.setSettings (Period.MORNING, DayType.WEEKDAY, 69) thermo.setPeriod (Period.MORNING); thermo.setDay (Daytype.WEEKDAY); Override b=t : Override T b=f : !Override F curTemp overTemp thresholdDiff c=t : curTemp < overTemp - thresholdDiff 63 72 5 c=f : !(curTemp < overTemp - thresholdDiff) 66 67 5 timeSinceLastRun minLag d=t : timeSinceLastRun > minLag 12 10 d=f : !(timeSinceLastRun > minLag) 8 10 These values then need to be placed into calls to turnHeaterOn() to satisfy the 6 tests for CACC (3 of 6)

60. Correlated Active Clause Coverage dTemp = 69 (period = MORNING, daytype = WEEKDAY ) 1. T t f t thermo.setCurrentTemp (63); thermo.setThresholdDiff (5); thermo.setOverride (true); thermo.setOverTemp (67); // c is false thermo.setMinLag (10); thermo.setTimeSinceLastRun (12); 2. F t f t thermo.setCurrentTemp (66); // a is false thermo.setThresholdDiff (5); thermo.setOverride (true); thermo.setOverTemp (67); // c is false thermo.setMinLag (10); thermo.setTimeSinceLastRun (12); Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 60 (4 of 6)

61. Correlated Active Clause Coverage dTemp = 69 (period = MORNING, daytype = WEEKDAY ) 3. f T t t thermo.setCurrentTemp (66); // a is false thermo.setThresholdDiff (5); thermo.setOverride (true); thermo.setOverTemp (72); // to make c true thermo.setMinLag (10); thermo.setTimeSinceLastRun (12); 4. F f T t thermo.setCurrentTemp (66); // a is false thermo.setThresholdDiff (5); thermo.setOverride (false); // b is false thermo.setOverTemp (72); thermo.setMinLag (10); thermo.setTimeSinceLastRun (12); Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 61 (5 of 6)

62. Correlated Active Clause Coverage dTemp = 69 (period = MORNING, daytype = WEEKDAY ) 5. t t t T thermo.setCurrentTemp (63); thermo.setThresholdDiff (5); thermo.setOverride (true); thermo.setOverTemp (72); thermo.setMinLag (10); thermo.setTimeSinceLastRun (12); 6. t t t F thermo.setCurrentTemp (63); thermo.setThresholdDiff (5); thermo.setOverride (true); thermo.setOverTemp (72); thermo.setMinLag (10); thermo.setTimeSinceLastRun (8); // d is false Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 62 (6 of 6)

63. Program Transformation Issues Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 63 if ((a && b) || c) { S1; } else { S2; } if (a) { if (b) S1; else { if (c) S1; else S2; } else { if (c) S1; else S2; } Transform ( 1 ) ? (8.3.4)

64. Problems With Transformation 1 We trade one problem for two problems : –Maintenance becomes harder –Reachability becomes harder Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 64 abc (a  b)  c CACCPC T TTTTX TTFTX TFTTXX TFFFXX FTTTX FTFFX FFTT FFFFX n Consider coverage : –CACC on the original requires four rows marked in the table –PC on the transformed version requires five different rows n PC on the transformed version has two problems : 1.It does not satisfy CACC on the original 2.It is more expensive (more tests)

65. Program Transformation Issue 2 Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 65 if ((a && b) || c) { S1; } else { S2; } d = a && b; e = d || c; if (e) { S1; } else { S2; } Transform ( 2 ) ?

66. Problems With Transformation 2 We move complexity into computations –Logic criteria are not effective at testing computations Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 66 abc (a  b)  c CACCPC T TTTTX TTFTX TFTTX TFFFX FTTT FTFFX FFTT FFFFX n Consider coverage : –CACC on the original requires four rows marked in the table –PC on the transformed version requires only two n PC on the transformed version becomes equivalent to clause coverage on the original –Not an effective testing technique

67. Transforming Does Not Work Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 67 Logic coverage criteria exist to help us make better software Circumventing the criteria is unsafe

68. Side Effects in Predicates Side effects occur when a value is changed while evaluating a predicate –A clause appears twice in the same predicate –A clause in between changes the value of the clause that appears twice Example : –Evaluation : Runtime system checks A, then B, if B is false, check A again –But now A has a different value! –How do we write a test that has two different values for the same predicate? No clear answers to this controllability problem Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 68 (8.3.5) A && (B || A)B is : changeVar (A) We suggest a social solution : Go ask the programmer

69. Summary : Logic Coverage for Source Code Predicates appear in decision statements ( if, while, for, etc.) Most predicates have less than four clauses –But some programs have a few predicates with many clauses The hard part of applying logic criteria to source is usually resolving the internal variables –Sometimes setting variables requires calling other methods Non-local variables (class, global, etc.) are also input variables if they are used If an input variable is changed within a method, it is treated as an internal variable thereafter Avoid transformations that hide predicate structure Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 69

70. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 70 Specifications in Software Specifications can be formal or informal –Formal specs are usually expressed mathematically –Informal specs are usually expressed in natural language Lots of formal languages and informal styles are available Most specification languages include explicit logical expressions, so it is very easy to apply logic coverage criteria Implicit logical expressions in natural-language specifications should be re-written as explicit logical expressions as part of test design –You will often find mistakes One of the most common is preconditions …

71. Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 71 Preconditions Programmers often include preconditions for their methods The preconditions are often expressed in comments in method headers Preconditions can be in javadoc, “requires”, “pre”, … Example – Saving addresses // name must not be empty // state must be valid // zip must be 5 numeric digits // street must not be empty // city must not be empty Rewriting to logical expression name != “”  state in stateList  zip >= 00000  zip <= 99999  street != “”  city != “” Conjunctive Normal Form

72. Shortcut for Predicates in Conjunctive Normal Form A predicate is in conjunctive normal form (CNF) if it consists of clauses or disjuncts connected by the and operator –A  B  C  … –(A  B)  (C  D) A major clause is made active by making all other clauses true ACC tests are “all true” and then a “diagonal” of false values: Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 72 ABC… 1TTT 2FTT… 3TFT 4TTF......

73. Shortcut for Predicates in Disjunctive Normal Form A predicate is in disjunctive normal form (DNF) if it consists of clauses or conjuncts connected by the or operator –A  B  C  … –(A  B)  (C  D) A major clause is made active by making all other clauses false ACC tests are “all false” and then a “diagonal” of true values: Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 73 ABC… 1FFF 2TFF… 3FTF 4FFT......

74. Summary : Logic Coverage for Specs Logical specifications can come from lots of places : –Preconditions –Java asserts –Contracts (in design-by-contract development) –OCL conditions –Formal languages Logic specifications can describe behavior at many levels : –Methods and classes (unit and module testing) –Connections among classes and components –System-level behavior Many predicates in specifications are in disjunctive normal or conjunctive normal form—simplifying the computations Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 74

75. © Ammann & Offutt 75 Covering Finite State Machines FSMs are graphs –Nodes represent state –Edges represent transitions among states Transitions often have logical expressions as guards or triggers As we said : Find a logical expression and cover it Introduction to Software Testing, Edition 2 (Ch 8)

76. © Ammann & Offutt 76 Example—Subway Train All Doors Open Left Doors Open Right Doors Open All Doors Closed trainSpeed = 0  platform=right  (location = inStation  (emergencyStop  overrideOpen  location = inTunnel)) trainSpeed = 0  platform=left  (location = inStation  (emergencyStop  overrideOpen  location = inTunnel)) secondPlatform = right secondPlatform= left ¬ emergencyStop  ¬ overrideOpen  doorsClear  closeDoorPressed (applies to all three transitions) Introduction to Software Testing, Edition 2 (Ch 8)

77. © Ammann & Offutt 77 Determination of the Predicate trainSpeed = 0  platform=left  (location = inStation  (emergencyStop  overrideOpen  location = inTunnel)) P trainSpeed = 0 : platform = left  (location = inStation  (emergencyStop  overrideOpen  location = inTunnel)) P platform = left : trainSpeed = 0  (location = inStation  (emergencyStop  overrideOpen  location = inTunnel)) P location = inStation : trainSpeed = 0  platform = left  (¬ emergencyStop  ¬ overrideOpen  ¬ location = inTunnel) P emergencyStop : trainSpeed = 0  platform = left  (¬ location = inStation  overrideOpen  location = inTunnel) P overrideOpen : trainSpeed = 0  platform = left  (¬ location = inStation  emergencyStop  location = inTunnel) Introduction to Software Testing, Edition 2 (Ch 8) P location = inTunnel : trainSpeed = 0  platform = left  (¬ location = inStation  emergencyStop  overrideOpen)

78. Test Truth Assignments (CACC) Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 78 trainSpeed = 0  platform=left  (location = inStation  (emergencyStop  overrideOpen  location = inTunnel)) Speed=0platform=leftinStationemergStopoverrideOpeninTunnel trainSpeed = 0 Tttttt trainSpeed != 0 Fttttt platform = left tTtttt platform != left tFtttt inStation ttTfff ¬ inStation ttFfff emergencyStop ttfTtt ¬ emergStop ttfFtt overrideOpen ttftTt ¬ overrideOpen ttftFt inTunnel ttfttT ¬ inTunnel ttfttF

79. An Accidental Transition Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 79 Expected outputs are easy : When the prime clause is true, the transition is taken When false, the transition is not taken Expected Result trainSpeed = 0Left Doors Open trainSpeed != 0All Doors Closed platform = leftLeft Doors Open platform != leftAll Doors Closed inStationLeft Doors Open ¬ inStationAll Doors Closed emergencyStopLeft Doors Open ¬ emergencyStopAll Doors Closed overrideOpenLeft Doors Open ¬ overrideOpenAll Doors Closed inTunnelLeft Doors Open ¬ inTunnelAll Doors Closed Oops … if platform !=left, then platform must equal right So the expected output of this test is to take a different transition Accidental transitions must be recognized when designing expected results during test automation

80. Problem With a Predicate? Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 80 Notice anything funny about these test requirements ? trainSpeed=0platform=leftinStationemergencyStopoverrideOpeninTunnel inStation ttTfff ¬ inStation ttFfff Cannot both be false. If the train is not in the station (location != inStation), then it must be in a tunnel (location = inTunnel)! Possible solutions : 1.Rewrite the predicate to eliminate dependencies Check with the developer for mistakes 2.Check each truth assignment, change the values as needed

81. Early Identification is a Win! Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 81 The process of modeling software artifacts for test design can help us find defects in the artifacts This is a very powerful side-effect of the model-driven test design process

82. © Ammann & Offutt 82 Complicating Issues Some buttons must be pressed simultaneously to have effect – so timing must be tested Reachability : The tests must reach the state where the transition starts (the prefix) Exit : Some tests must continue executing to an end state Expected output : The expected output is the state that the transition reaches for true values, or same state for false values Accidental transitions : Sometimes a false value for one transition happens to be a true value for another –The alternate expected output must be recognized Introduction to Software Testing, Edition 2 (Ch 8)

83. © Ammann & Offutt 83 Test Automation Mapping problem : The names used in the FSMs may not match the names in the program Examples –platform = left requires the train to go to a specific station –trainspeed = 0 probably requires the brake to be applied multiple times The solution to this is implementation-specific –Sometimes a direct name-to-name mapping can be found –Sometimes more complicated actions must be taken to assign the appropriate values –Simulation : Directly inserting value assignments into the middle of the program This is an issue of controllability Introduction to Software Testing, Edition 2 (Ch 8)

84. Summary FSM Logic Testing FSMs are widely used at all levels of abstraction Many ways to express FSMs –Statecharts, tables, Z, decision tables, Petri nets, … Predicates are usually explicitly included on the transitions –Guards –Actions –Often represent safety constraints FSMs are often used in embedded software Introduction to Software Testing, Edition 2 (Ch 8) © Ammann & Offutt 84