Course Software Testing & Verification 2013/14 Wishnu Prasetya Logical Coverage (ch. 3) Course Software Testing & Verification 2013/14 Wishnu Prasetya

How to test this? m(p) { if (!member(p) || (p.age>17 && !admin(p)) S0 ; else S1 ; } m(p) { if (!member(p)) goto L0 if (!p.age17) goto L1 if (!admin(p)) goto L1 L0: S0 ; L1: S1 ; } The code to the left is often transformed to the “short-circuiting” version above by a compiler. In most cases, we do not want to try all combinations, but does point out that the two most obvious choices of cov criteria are not the same, and we are left with the dillema what to choose; or to perhaps come up with something in between. Another point to make here is that path-based coverage does not in priciple take the semantics of logical formulas into account. Edge coverage or even full path coverage on the original code may feel inadequate. The short circuit version do have more paths (4), but requires us to know what the compiled code is.

Member(p)  (p.age>17  admin(p)) The obvious ones Member(p)  (p.age>17  admin(p)) a  (b  c) (C3.12) Predicate coverage (C3.13) Clause coverage; Note that AO call “Member(p)” a clause instead of e.g. proposition. |TR| = ... ? feels too weak strangely does not subsume predicate coverage, nor the other way around. (C3.14) All combinations  explodes if the formula is large |TR| = 2 will do to do clause coverage, for any formula. Example to show that CC does not subsume PC: a \/ b t1: a=1, b=0 t2: a=0, b=1 covers CC, but not PC. Conversely: t1: a=1,b=0 t2: a=0,b=0 covers PC but not CC

Using active clauses a  (b  c) Major and minor clause. 1 a  (b  c) duplicates Major and minor clause. (Def 3.42) a major clause c of p determines p if it is possible to make it active : to assign values to the minor clauses so that changing the value of c changes the value of p. (Def3.43, Active Clause Covrg, ACC) For each major clause c which determines p, TR includes the cases c=1 and c=0. Minor clauses must be given values that would make p active. |T|  2*n, but duplicates can be removed, ending up with |T|  n+1, if n is number of clauses. Sounds good, but.... If the whole formula is a contradiction or a tautology, then obviously no clause can be a determinant. AO use the term “active clause”, but does not define what it is! I assume here it means a determining clause. |T| <=n + 1 Note that it is an upperbound. In the next example a<->b, n+1 = 3, and yet 2 test-cases can cover ACC.

You need to fix the def. of ACC! b a  b 1 a  b Active Clause Covrg (ACC): for each major clause c which determines p, TR includes the cases c=1 and c=0; etc  need refinement! does not specify if you should use the same values of minor clauses or not. does not require p to actually change when you change c does not subsume predicate covrg. See example above. In the example a <-> b, a is determinant, for any value of b. (and conversely too of course). So, the test: t1 : {a = 0, b = 0 } t2 : {a =1, b= 1} will cover ACC for a as major, as well as b as major. But we haven’t test p = false !!

Correlated and Restricted ACC b a  b 1 a  b (C3.15) General ACC: for each major clause c which determines p, TR includes the cases c=1 and c=0. Minor clauses must be given values that would make p active. The values of the minors in both cases do not have to be the same.  actually this is just the same as ACC. (C3.16) Correlated ACC: as above, but for those two cases of c, one must cause p to be 1 and the other 0. (C3.17) Restricted ACC: as in ACC but the values of the minors chosen in both cases of c must be the same. OA mean ACC as an intermediate definition. They know it leaves some thing unspecified (as we discussed), but they then call ACC ambiguous, with a reference to the concept of MCDC which were used elsewhere. ACC and MCDC are the same, but the latter is indeed known to be ambiguous. So, based on this historical reason ACC can be called ambiguous. From my perspective, it is a good definition, but leaves some things unspecified. In itself, there is nothing wrong with that. Anyway, the difference is perhaps nothing more than words play. But in anycase, if you look it from my perspetive, then General ACC is just equivalent to ACC. It is just being defined with some of the unspecified aspects of ACC being spelled out. For CACC, these tests will do: For a: { a=0, b=0 } and { a=1,b=0 } For b: {a=0, b=0 } and { a=0, b=1} One is redundant. The above test will also satisfes RACC

Correlated vs Restricted ACC a  ( b  c ) isStudent(p)  ( isMasterStudent(p)  c ) a b c whole 1 a b c whole 1 a : a : b : b : In blue we see an example of additional semantical constraint. It is not possible for p to be a master-student if she is not a student. Now looking at the tests wrt a as the active clause in RACC (the red rows), we see that the first one is feasible, but the second one { isStudent=1, isMaster=0, c=1} is actually not possible. In CACC we are allowed to choose a different values for minor clauses, e.g. (see the left table); both red tests in the left table is feasible. Also to note: CACC obviously subsumes predicate coverage; but the other way around is not the case. E.g. the tests: {a=1,b=1,c=1} and {a=0,b=1,c=1} gives full predicate coverage, but not CACC. c : c : Left shows tests that cover CACC; after removing duplicates we get |T| = 4. Right covers RACC; after removing duplicates |T| = 4. RACC subsumes CACC Keep in mind some combinations may actually be impossible

Inactive-clause approach ACC-variants makes sure that each situation where a clause affects the whole predicate is tested. Inactive clause is the opposite. A major clause c is in inactive if it is possible to assign values to minor clauses so that changing the value of c won’t change the value of p. (Def3.44, Inactive Clause Coverage, ICC) For each major clause c, if it is possible to make it inactive, TR includes: (1) a test where c=0 and p=1, (2) a test where c=1 and p=1, (3) a test where c=0 and p=0, (4) a test where c=1 and p=0. Minor clauses must be given values that make c inactive. (C3.18, GICC) if the values of minor clauses in 1,2,3,4 do not have to be the same. (C3.19, RICC) if the values of minors in 1/2 and 3/4 have to be te same. Their definitions suggest that ACC and ICC are complementary.

Overview Combinatorial Clause Coverage COC Restricted Active Clause Coverage RACC Restricted Inactive Clause Coverage RICC Correlated Active Clause Coverage CACC General Inactive Clause Coverage GICC General Active Clause Coverage GACC Clause Coverage CC Predicate Coverage PC

Finding values to set a clause active/inactive In ACC-variants we need to figure out how to assign values to minor clauses to make a major clause c determines p. Is there an algorithm for this!?  yes; simple, namely by solving (finding values of the variables that make the formula true) : pc=0  pc=1 Equivalently, solve: pc=0 xor pc=1 = 1 How to solve such a formula? You can imagine how; but if p is large, use e.g. a SAT solver. Well, all we need is to solve the equation: p[c=0]  p[c=1]. Equivalently, we solve (p[c=0] XOR p[c=1]) = 1. For ICC approach we need to solve p[c=0] = p[c=1]

Constructing the TR We’ll take CACC as an example. Given a p, and a constraint B over the clauses (e.g. isMasterStd(p)  isStd(p)). For each clause c, solve these: (1) Bc=1  (pc=1 = 1)  ( pc=0 xor pc=1) (2) Bc=0  (pc=0 = 0)  ( pc=0 xor pc=1) Add the solutions to TR. Remove duplicates from TR.

Going back to program p.pt+1 < 1000 awardPt(p) { p.pt ++ ; if (birthday(p) && p.pt <1000 ) notify(p) ; } Apply a logic-coverage at each decision point. This produces a TR for each. But : we still have to calculate the corresponding program’s inputs  calculate it by hand? Can we automate the calculation?  what to do with loops/recursions?

Disjunctive Normal Form (DNF) ((a  c)  b)  (a  c) (a  c)  (b  c) Rather than investigating each clause of p, we can opt to investigate different “logical scenarios” that affect p. Every proposition logic formula f can be rewritten to an equivalent DNF. A DNF consists of implicants, connected by . Each implicant consists of literals connected by . A literal is a clause, or a negation of a clause. The right formula above is the DNF of the left one.

Implicants of p and p p = ab  bc  p =  (ab  bc) = b  ac We’ll write p to abbreviate p Implicants of p  scenarios leading to p becomes true Implicants of p  scenarios leading to p become false. Idea: cover all those scenarios.

Implicant Coverage p = ab  bc p =  (ab  bc) = b  ac implicant a b c ab 1 bc ac The above tests set reduces to just two tests, which do not satisfy GACC. After reduction above only give two test cases: t1 = { a=1, b=1, c=0 } t2 = { a=0, b=0, c=1 } Notice wrt to a as major clause, the second test-case (the 3rd and 4th rows in the table above) assign values to b and c that do NOT make a active! Objection with IC: it feels a bit weak; the same test can satisfy the requirement wrt multiple implicants. (C3.20, IC) Let p be a DNF, and p be a DNF of p; for each implicant i in p and p TR includes the case that i = 1. IC subsumes pred. cov (PC); but not necessarily subsumes any ACC variant.

Unique true point p = abc  abc  bc p = ab  ac  bc (p140) a unique true point of a DNF p wrt implicant i is values assignment that makes i = 1 and other implicants (of p) = 0. (C3.21, Unique True Point Covrg, UTPC) Given minimal DNF p and its negation p; for every implicant i in p and p the TR includes a unique true point for i. (p140) Prime implicant, e.g. abc in red; abc is not. (p140) Redundant implicant, e.g ab in blue. A DNF is minimal if every implicant is prime and not redundant. In such a DNF, there exists a test satisfying an implicant that does not satisfy the others. Prime implicant: if no subterm of the implicant is also an implicant. This implies dependency of that other implicant to the prime implicant. Redundant: if removing the implicant does not change the meaning of p. To show that ab is redundant in blue, we show that ab \/ ac \/ b~c = ac \/ b~c We show this in case split. (1) ab = 1 , and (2) ab=0 If ab = 1, then lhs=1. Now rhs = ac \/ b~c = 1c \/ 1~c = c \/ ~c = 1 DONE Now if ab=0, then trivially lhs = rhs.

IC vs UTPC p = ab  bc p = b  ac |T| = 2 |T| = 4 implicant a b c ab 1 bc ac |T| = 2 Unfortnately UTPC does not subsume GACC let alone the other two ACC. Example: see book. implicant a b c ab 1 bc ac |T| = 4

Converting to DNF ab a b c Rewriting, using Boolean algebra 1 00 01 11 10 1 b c Rewriting, using Boolean algebra For small number of clauses we can also use Karnaugh map (p144 -145). Important property: the combinations that belong to two adjacent cells, differ only in one bit. (“adjancent” in donut sense.) This allows simplication over those cells. Karnaugh gives you minimal DNF.  if you make sure that every group has at least one non-shared cell. The map on the left produces: ~a~b \/ ~ab \/ a~b But if we use simplification: ~a \/ ~b The second formula: a \/ ~ab~c \/ ~a~bc; but this can actually be simpler: a \/ b~c \/ ~bc

Near False Point Not suprisingly, UTPC does not subsume GACC  it just disregard clauses. Let p be a DNF with implicant i. Let c be a clause in i. A near-false point is a values assignment that makes p=0, but if we flip the value of c, it will make i=1 (thus also p). NFP gives us a concept of how a clause influences an implicant. Hey, this 111 is actually also a UTP of ab ! Near-false points for a in ab : 011 Flipping the value of a : 111  makes ab true (and p too) p = ab  bc

Near False Point Near-false points for a in ab : 011 Flipping the value of a : 111  makes ab true (and p too) is actually also a UTP of ab p = ab  bc (C3.22, Corresponding Unique True Point and Near False Point Pair Covrg) Given a minimalDNF p, for every clause c and implicant i, TR contains a unique true point for i and a near-false point wrt c and i, such that these points only differ in the value of c. Does not subsume UTPC (simply because you dont have the DNF for p) , but it subsumes RACC.

UTPC vs CUTPNFP How strong are they compared to each other? Well, we just said they are unrelated. Still... let’s define a new metric: their ability to reveal certain types of faults  let’s take hypothetical faults you can make when you manually write down a DNF

Some mistakes when coding down a DNF ENF: Expression Negation Fault f = ab+c f’ = ab+c TNF: Term/Implicant Neg. Fault f = ab+c f’ = ab+c TOF: Term Omission Fault f = ab+c f’ = ab LNF: Literal Negation Fault f = ab+c f’ = ab+c LRF: Literal Reference Fault f = ab + bcd f’ = ad + bcd LOF: Literal Omission Fault f = ab + c f’ = a + c LIF: Literal Insertion Fault f = ab + c f’ = ab + bc ORF+: Operator Reference Fault f = ab + c f’ = abc ORF*: Operator Reference Fault f = ab + c f’ = a + b + c Key idea: fault classes are related to testing. A test case “reveals” a fault if it results f’ is different than the expected result on f. A test-case that reveals certain faults may reveal additional faults.

Literal Insertion Fault LIF Literal Reference Fault LRF Literal Omission Fault LOF Term Omission Fault TOF Literal Negation Fault LNF Operator Reference Fault ORF* Operator Reference Fault ORF+ Term Negation Fault TNF Expression Negation Fault ENF UTPC can detect TOF. CUTPNFP detects TOF as well, and detects LOF too.