Software Testing Logic Coverage. Introduction to Software Testing (Ch 3) © Ammann & Offutt 2 Logic Coverage Four Structures for Modeling Software Graphs.

Slides:



Advertisements
Similar presentations
Geometry Logic.
Advertisements

Introduction to Software Testing Chapter 1
Constructing a Truth Table
Introduction to Software Testing Chapter 3.3 Logic Coverage from Source Code Paul Ammann & Jeff Offutt.
Introduction to Software Testing Chapter 3.1, 3.2 Logic Coverage Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 1 Paul Ammann & Jeff Offutt SUMMARY OF PARTS 1 AND 2 FROM LAST WEEK.
Logic ChAPTER 3 1. Truth Tables and Validity of Arguments
CITS5501 Software Quality and Testing Logic and Path Coverage From Introduction to Software Testing, 2 nd ed, Ammann & Offutt.
Introduction to Software Testing Chapter 3.1 Logic Coverage Paul Ammann & Jeff Offutt.
Introduction to Software Testing Chapter 8.2
Truth Tables for Negation, Conjunction, and Disjunction.
3.2 – Truth Tables and Equivalent Statements
Normal or Canonical Forms Rosen 1.2 (exercises). Logical Operators  - Disjunction  - Conjunction  - Negation  - Implication p  q   p  q  - Exclusive.
CSE 20 DISCRETE MATH Prof. Shachar Lovett Clicker frequency: CA.
Chapter 1 The Logic of Compound Statements. Section 1.1 Logical Form and Logical Equivalence.
Today’s Agenda  Quick Review  Finish Graph-Based Testing  Predicate Testing Software Testing and Maintenance 1.
Introduction to Software Testing Chapter 8.1 Logic Coverage Paul Ammann & Jeff Offutt
Logical Form and Logical Equivalence Lecture 2 Section 1.1 Fri, Jan 19, 2007.
CSNB143 – Discrete Structure LOGIC. Learning Outcomes Student should be able to know what is it means by statement. Students should be able to identify.
Introduction to Software Testing Chapter 3.6 Disjunctive Normal Form Criteria Paul Ammann & Jeff Offutt
Boolean Algebra. Logical Statements A proposition that may or may not be true:  Today is Monday  Today is Sunday  It is raining.
Boolean Logic. Boolean Operators (T/F) xyx AND y FFF FTF TFF TTT xyx OR y FFF FTT TFT TTT xNOT x FT TF.
Introduction to Software Testing Chapter 3.1, 3.2 Logic Coverage Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapters 1-5 Coverage Summary Paul Ammann & Jeff Offutt
Chapter 7 Logic, Sets, and Counting
First delivery of the course Software Quality and Testing Katerina Zdravkova, Anastas Mišev
Introduction to Software Testing Paul Ammann & Jeff Offutt Updated 24-August 2010.
CS1022 Computer Programming & Principles Lecture 1 Logic and Proof.
Warmup Answer the following True/False questions in your head: I have brown hair AND I am wearing glasses I am male OR I am wearing sneakers I am NOT male.
How do I show that two compound propositions are logically equivalent?
Logic. Statements, Connectives, and Quantifiers In symbolic logic, we only care whether statements are true or false – not their content. In logic, a.
Logical Form and Logical Equivalence Lecture 1 Section 1.1 Wed, Jan 12, 2005.
Boolean Algebra Monday/Wednesday 7th Week. Logical Statements Today is Friday AND it is sunny. Today is Friday AND it is rainy. Today is Monday OR it.
Introduction to Software Testing Chapter 8.1 Logic Coverage Paul Ammann & Jeff Offutt
1 Propositional Logic Introduction. 2 What is propositional logic? Propositional Logic is concerned with propositions and their interrelationships. 
Boolean Logic.
Using Logic Criterion Feasibility to Reduce Test Set Size While Guaranteeing Double Fault Detection Gary Kaminski and Paul Ammann Software Engineering.
Notes - Truth Tables fun, fun, and more fun!!!!. A compound statement is created by combining two or more statements, p and q.
Introduction to Software Testing Chapter 3.6 Disjunctive Normal Form Criteria Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 3.6 Disjunctive Normal Form Criteria Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 3.1 Logic Coverage Paul Ammann & Jeff Offutt.
Slot machines A healthy way to learn about connectives!
Introduction to Software Testing Chapter 3.2 Logic Coverage
Mathematics for Computing Lecture 2: Computer Logic and Truth Tables Dr Andrew Purkiss-Trew Cancer Research UK
Introduction to Software Testing Chapter 3.4 Logic Coverage for Specifications Paul Ammann & Jeff Offutt
Introduction to Software Testing (2nd edition) Chapter 5 Criteria-Based Test Design Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 8.2
Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 8.1 Logic Coverage
Paul Ammann & Jeff Offutt
Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 3, Sec# 1 & 2 Logic Coverage
Paul Ammann & Jeff Offutt
Introduction to Software Testing Syntactic Logic Coverage Criteria
Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 3.1, 3.2 Logic Coverage
Logic Coverage CS 4501 / 6501 Software Testing
Paul Ammann & Jeff Offutt
Moonzoo Kim School of Computing KAIST
Introduction to Software Testing Chapter 3.2 Logic Coverage
Logic Coverage CS 4501 / 6501 Software Testing
Paul Ammann & Jeff Offutt
Logic Coverage Active Clause Coverage CS 4501 / 6501 Software Testing
Logic Coverage for Source Code CS 4501 / 6501 Software Testing
Introduction to Software Testing Chapter 5.1 Syntax-based Testing
[Ammann and Offutt, “Introduction to Software Testing,” Ch. 8]
Introduction to Software Testing Chapter 8.1 Logic Coverage
Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 3.2 Logic Coverage
Introduction to Software Testing Chapter 3.1, 3.2 Logic Coverage
Presentation transcript:

Software Testing Logic Coverage

Introduction to Software Testing (Ch 3) © Ammann & Offutt 2 Logic Coverage Four Structures for Modeling Software Graphs Logic Input Space Syntax Use cases Specs Design Source Applied to DNF Specs FSMs Source Input Models Integ Source

3 Logic Predicates A predicate is an expression that evaluates to a boolean value Predicates can contain –boolean variables –non-boolean variables that contain >, =, <=, != –boolean function calls Internal structure is created by logical operators –¬ – the negation operator –  – the and operator –  – the or operator –  – the implication operator –  – the exclusive or operator –  – the equivalence operator

4 Predicate Coverage The first (and simplest) two criteria require that each predicate and each clause be evaluated to both true and false Predicate Coverage (PC) : For each p in P, TR contains two requirements: p evaluates to true, and p evaluates to false. When predicates come from conditions on edges, this is equivalent to edge coverage

5 Examples (a = n*o) Four clauses: –(a < b) – relational expression – f (z) – Boolean-valued function – D – Boolean variable – (m >= n*o) – relational expression Sources of predicates –Decisions in programs –Guards in finite state machines –Decisions in UML activity graphs –Requirements, both formal and informal –SQL queries

Shortcut for Predicates in Conjunctive Normal Form Conjunctive clauses are connected only by the and operator –A  B  C  … Each major clause is made active by making all other clauses true ACC tests are “all true” and then a “diagonal” of false values: 6 ABC… 1TTT 2FTT… 3TFT 4TTF......

Shortcut for Predicates in Disjunctive Normal Form Disjunctive clauses are connected only by the or operator –A  B  C  … Each major clause is made active by making all other clauses false ACC tests are “all false” and then a “diagonal” of true values: 7 ABC… 1FFF 2TFF… 3FTF 4FFT......

8 Predicate Coverage Example ((a = n*o) predicate coverage Predicate = true a = 5, b = 10, D = true, m = 1, n = 1, o = 1 = (5 = 1*1) = true  true  TRUE = true Predicate = false a = 10, b = 5, D = false, m = 1, n = 1, o = 1 = (10 = 1*1) = false  false  TRUE = false

9 Clause Coverage Example ((a = n*o) Clause coverage Two tests (a < b) = true a = 5, b = 10 (a < b) = false a = 10, b = 5 D = true D = false m >= n*o = true m = 1, n = 1, o = 1 m >= n*o = false m = 1, n = 2, o = 2 true cases 1) a = 5, b = 10, D = true, m = 1, n = 1, o = 1 false cases 2) a = 10, b = 5, D = false, m = 1, n = 2, o = 2

10 Combinatorial Coverage CoC requires every possible combination Sometimes called Multiple Condition Coverage Combinatorial Coverage (CoC) : For each p in P, TR has test requirements for the clauses in Cp to evaluate to each possible combination of truth values. a < bDm >= n*o ((a = n*o) 1TTTT 2TTFF 3TFTT 4TFFF 5FTTT 6FTFF 7FFTF 8FFFF

Combinatorial Coverage This is simple, neat, clean, and comprehensive … But quite expensive! 2 N tests, where N is the number of clauses –Impractical for predicates with more than 3 or 4 clauses The literature has lots of suggestions – some confusing The general idea is simple: Test each clause independently from the other clauses

12 Determining Predicates Goal : Find tests for each clause when the clause determines the value of the predicate This is formalized in several criteria that have subtle, but very important, differences P = A  B if B = true, p is always true. so if B = false, A determines p. if A = false, B determines p. P = A  B if B = false, p is always false. so if B = true, A determines p. if A = true, B determines p.

13 p = a  b 1)a = true, b = false 2)a = false, b = false 3)a = false, b = true 4)a = false, b = false Active Clause Coverage Ambiguity : Do the minor clauses have to have the same values when the major clause is true and false? Active Clause Coverage (ACC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. Duplicate a is major clauseb is major clause

14 A More Subtle Example p = ( a  b )  ( a  ¬ b) p a = p a=true  p a=false = ((true  b)  (true  ¬ b))  ((false  b)  (false  ¬ b)) = (b  ¬ b)  false = true  false = true a always determines the value of this predicate b never determines the value – b is irrelevant ! p = ( a  b )  ( a  ¬ b) p b = p b=true  p b=false = ((a  true)  (a  ¬ true))  ((a  false)  (a  ¬ false)) = (a  false)  (false  a) = a  a = false

Condition Coverage (cont’d) if A or B then s1 else s2 end_if_then_el se ABBranch test 1TFtrue test 2FFfalse

Compound Condition Coverage (cont’d) input(X,Y) if (Y<=0) or (X=0) then Y := -Y end_if Combinations of condition values: TT, TF, FT, FF

Thermostat (pg 1 of 2) 17 1 // Jeff Offutt--October // Programmable Thermostat 3 import java.io.*; 4 class thermostat 5 { 6 private Heater myHeater; 7 // Decide whether to turn the heater on, and for how long. 8 public boolean turnHeaterOn ( 9 int curTemp, /* Current temperature reading */ 10 int thresholdDiff, /* Temp difference until we turn heater on */ 11 Minutes timeSinceLastRun, /* Time since heater stopped */ 12 Minutes minLag, /* How long I need to wait */ 13 Time timeOfDay, /* current time (Hours and minutes) */ 14 Day dayOfWeek, /* Monday, Tuesday,... */ 15 Settings programmedSettings [], /* User's program, by day */ 16 boolean Override, /* Has user overridden the program */ 17 int overTemp /* OverridingTemp */ 18 )

Thermostat (pg 2 of 2) { 20 int desiredTemp; 21 // getPeriod() translates time into Morning, Day, Evening, Night 22 desiredTemp = programmedSettings [dayOfWeek].getDesiredTemp 23 (getPeriod [TimeOfDay]); 24 if (((curTemp < desiredTemp - thresholdDiff) || 25 (Override && curTemp < overTemp - thresholdDiff)) && 26 timeSinceLastRun.greaterThan (minLag)) 27 { // Turn on the heater 28 // How long? Assume 1 minute per degree (Fahrenheit) 29 int timeNeeded = curTemp - desiredTemp; 30 if (Override) 31 timeNeeded = curTemp - overTemp; 32 myHeater.setRunTime (timeNeeded); 33 return (true); 33 } 34 else 35 return (false); 36 } // End turnHeaterOn 37 } // End class

19 Two Thermostat Predicates 24-26: (((curTemp < desiredTemp - thresholdDiff) || (Override && curTemp < overTemp - thresholdDiff)) && timeSinceLastRun.greaterThan (minLag)) 30: (Override) Simplify a : curTemp < desiredTemp - thresholdDiff b : Override c : curTemp < overTemp - thresholdDiff d : timeSinceLastRun.greaterThan (minLag) 24-26: (a || (b && c)) && d 30: b

20 Predicate Coverage (true) (a || (b && c)) && d curTemp < desiredTemp – thresholdDiff : true Override : true curTemp < overTemp – thresholdDiff : true timeSinceLastRun.greaterThan (minLag) : true a : true b : true c : true d : true programmedSettings [Monday].setDesiredTemp (Morning, 69) // dayOfWeek = Monday // timeOfDay = 8:00 curTemp = < 69 – 5 Override : true overTemp = < 70 – 5 timeSinceLastRun.setValue (12) minLag = 10

21 Predicate Coverage (false) (a || (b && c)) && d curTemp < desiredTemp – thresholdDiff : false Override : false curTemp < overTemp – thresholdDiff : false timeSinceLastRun.greaterThan (minLag) : false a : false b : false c : false d : false programmedSettings [Monday].setDesiredTemp (Morning, 69) // dayOfWeek = Monday // timeOfDay = 8:00 curTemp = < 69 – 5 Override : false overTemp = < 70 – 5 timeSinceLastRun.setValue (8) minLag = 10

Correlated Active Clause Coverage (1 of 5) 22 P a = ((a || (b && c)) && d)  ((a || (b && c)) && d) (T && d)  ((b && c) && d) ((T || (b && c)) && d)  ((F || (b && c)) && d) !(b && c) && d ( !b || !c ) && d d  ((b && c) && d)

Correlated Active Clause Coverage (2 of 5) 23 (a || (b && c)) && d a : T t f t (row 3) F t f t (row 13) b : f T t t (row 9) f F t t (row 13) c : f t T t (row 9) f t F f (row 11) d : t t t T (row 1) t t t F (row 2) duplicates Six tests needed for CACC on Thermostat

Correlated Active Clause Coverage (3 of 5) 24 curTemp desiredTemp thresholdDiff curTemp < desiredTemp - thresholdDiff !(curTemp < desiredTemp - thresholdDiff) desiredTemp = programmedSettings [Monday].setDesiredTemp (Morning, 69) dayOfWeek = Monday timeOfDay = 8:00 Override Override T !Override F curTemp overTemp thresholdDiff curTemp < overTemp - thresholdDiff !(curTemp < overTemp - thresholdDiff) timeSinceLastRun minLag timeSinceLastRun.greaterThan (minLag) !(timeSinceLastRun.greaterThan (minLag)) 8 10 These values then need to be placed into calls to turnHeaterOn() to satisfy the 6 tests for CACC

Correlated Active Clause Coverage (4 of 5) desiredTemp = programmedSettings [Monday].setDesiredTemp (Morning, 69) 1. T t f t a = T : curTemp = 63; c = f : curTemp = 66 turnHeaterOn ( 63/66, 5, 12, 10, 8:00, Monday, programmedSettings, true, 70 ) 2. F t f t turnHeaterOn ( 66, 5, 12, 10, 8:00, Monday, programmedSettings, true, 70 ) 3. f T t t a = f : curTemp = 66; c = t : curTemp = 63 turnHeaterOn ( 63/66, 5, 12, 10, 8:00, Monday, programmedSettings, true, 70 ) 4. f t F f turnHeaterOn ( 66, 5, 8, 10, 8:00, Monday, programmedSettings, true, 70 ) 5. t t t T turnHeaterOn ( 63, 5, 12, 10, 8:00, Monday, programmedSettings, true, 70 ) 6. t t t F turnHeaterOn ( 63, 5, 8, 10, 8:00, Monday, programmedSettings, true, 70 ) 25

Correlated Active Clause Coverage (5 of 5) curTemp is fixed by the solution to clause a thresholdDiff is also fixed by the solution to clause a So we choose different values for overtemp T t f t turnHeaterOn ( 63, 5, 12, 10, 8:00, Monday, programmedSettings, true, 62 ) 3. f T t t turnHeaterOn ( 66, 5, 12, 10, 8:00, Monday, programmedSettings, true, 66 ) 26

Program Transformation Issues 27 if ((a && b) || c) { S1; } else { S2; } if (a) { if (b) S1; else { if (c) S1; else S2; } else { if (c) S1; else S2; } Transform (1)? Transform (2)? d = a && b; e = d || c; if (e) { S1; } else { S2; }

Problems with Transformed Programs Maintenance is certainly harder with Transform (1) –Not recommended! Coverage on Transform (1) –PC on transform does not imply CACC on original –CACC on original does not imply PC on transform Coverage on Transform (2) –Structure used by logic criteria is “lost” –Hence CACC on transform 2 only requires 3 tests –Note: Mutation analysis (Chapter 5) addresses this problem Bottom Line: Logic coverage criteria are there to help you! abc (a  b)  c CACC PC CACC (2) TTTTX TTFTXX TFTTXXX TFFFXX FTTTX FTFFXX FFTT FFFFX 28

float exam, course, average; char trailer[20]; cin>>exam; cin>>course; if ( (exam 100.0) || (course ) ) cout<<"Marks out of range - program terminating"; else { if ( (exam<50.0) || (course<50.0)) cout<<"Fail"; else if (exam < 60) cout<<"Pass - Grade C"; else if ( ((exam+course)/2.0) >= 70.0) { average = (exam+course)/2.0; printf("Pass - Grade A with an average mark of %5.2f\n", average); cout<<"Pass - Grade A with an average mark of "<< average <<endl;} else cout<<"Pass - Grade B"; } 29