Using Logic Criterion Feasibility to Reduce Test Set Size While Guaranteeing Fault Detection Gary Kaminski and Paul Ammann ICST 2009 March 24 Version

Motivation Current logic criteria: –generate large test sets (Combinatorial) or –do not guarantee detecting logic faults (RACC) Goal: - generate smaller test sets while still guaranteeing fault detection Assumption: - restrict attention to minimal Disjunctive Normal Form (DNF) Boolean predicates tested in isolation

A Word About Infeasibility Infeasible Test Requirements are a hassle! –They can bloat test sets –They can thwart subsumption hierarchies –Example: RACC and CACC May be infeasible to satisfy RACC, but feasible to satisfy CACC RACC subsumes CACC, yet for a literal in a predicate, CACC may yield a test case when RACC does not Coverage Criteria for Detecting Logic Faults –If all test requirements feasible, simple criteria are enough –More complex criteria needed to fill in the gaps –This paper analyzes feasibility at a “low” level –Result: Minimal, fault-detecting test sets

Building Test Sets Guaranteed to Detect Faults (Current) Predicate P: ab + a!c Test Set = T1 + T2 + T3 Apply Criterion 1 T1 Apply Criterion 2 T2 Apply Criterion 3 T3 Apply Each Criterion to P, Component by Component If criterion feasible on component, generate test If criterion infeasible on component, satisfy as much as possible Result: Tests from all Criteria on all Components Criteria are all necessary; but individual tests may be unnecessary

Analyzing Criterion Feasibility at Component Level Predicate P: ab + a!c Test Set = T1’ + T2’ + T3’ Criterion Feasibility Analyzed, Component by Component If criterion feasible on component, generate test and FINISH If criterion infeasible on component, partially satisfy and go to next criterion Criterion 1 Feasible? Extract Components Apply Criterion 1 T1’ Yes Criterion 2 Feasible? No Apply Criterion 2 Yes T2’ Apply Criterion 3 No T3’ Result: Every resulting test has a reason for being there Note: Some details glossed over in this figure…

Minimal DNF Terms separated by OR, literals by AND ab + a!c vs. a(b + !c) Make each term true and other terms false ab + ac vs. ab + abc Impossible to remove a literal or term without changing the predicate ab vs. abc + ab!c

Minimal DNF Logic Faults Original: ab + bc Literal Insertion Fault: abc + b!c Literal Insertion Fault: ab!c + b!c Literal Reference Fault: ac + b!c Literal Reference Fault: a!c + b!c Literal Omission Fault: b + b!c A test set detecting these faults also detects others

Lau and Yu’s Fault Hierarchy A test set that guarantees detection of a source fault guarantees detection of a destination fault Ignores effect of criterion feasibility LOF ORF. LRF LNF TNF ENF LIF TOF ORF+

Unique True Points and Near False Points UTP: An assignment of values such that only one term evaluates to true. ab + !ac: 110 and 111 are UTPs for ab NFP: An assignment of values such that the predicate evaluates to false but when a literal is omitted, it evaluates to true. ab + !ac: 100 and 101 are NFPs for b

MUTP Criterion Find UTP tests for each term such that all literals not in the term attain 0 and 1. Detects LIF and if feasible, detects LRF Inexpensive to satisfy Feasible for ab + !ac ab – 110, 111 !ac – 001, 011 Infeasible for ab + ac ab – 110

CUTPNFP Criterion Find a UTP - NFP pair such that only the literal of interest changes value. Detects LOF and if feasible, detects LRF More expensive to satisfy Feasible for b in ab + ac UTP for ab is 110 NFP for b in ab is 100 Infeasible for b in first term of ab + b!c + !bc UTP for ab is 111 NFP for b in ab 100 (101 makes !bc true)

MNFP Criterion Find NFP tests for each literal such that all literals not in the term attain 0 and 1. Detects LOF and if feasible, detects LRF Most expensive to satisfy Feasible for a in first term of ab + ac 010, 011 Infeasible for a in first term of ab + !ac 010 (011 makes !ac true)

MUMCUT Criterion Combine CUTPNFP, MNFP, and MUTP - detects LIF, LRF, and LOF but expensive - without considering feasibility need all 3 criteria to detect LRF Other criteria require less inputs but do not guarantee fault detection (RACC) Can we reduce MUMCUT test set size while still guaranteeing LRF detection?

MUTP Feasibility and LRF If MUTP is feasible for a term: Black – Green -MUTP detects LRF -CUTPNFP not needed to detect LRF -MNFP not needed to detect LRF CUTPNFP feasible? MNFP Test Set = MUTP + MNFP For Each Literal In Term Test Set = MUTP + CUTPNFP For Each Term MUTP feasible? Test Set = MUTP + NFP

CUTPNFP Feasibility and LRF If MUTP is infeasible for a term but CUTPNFP is feasible for a literal in the term: Black – Red – Black - Green -MUTP does not detect LRF -CUTPNFP detects LRF -MNFP not needed to detect LRF Test Set = MUTP + NFP MNFP Test Set = MUTP + MNFP For Each Term MUTP feasible? CUTPNFP feasible? For Each Literal In Term Test Set = MUTP + CUTPNFP

MNFP Feasibility and LRF If MUTP is infeasible for a term and CUTPNFP is infeasible for a literal in the term: Black – Red – Black – Red – Black -MUTP does not detect LRF -CUTPNFP does not detect LRF -MNFP will detect LRF For Each Term MUTP feasible? CUTPNFP feasible? MNFP Test Set = MUTP + MNFP For Each Literal In Term Test Set = MUTP + NFP Test Set = MUTP + CUTPNFP

Minimal-MUMCUT Criterion MUTP feasible  MUTP detects LRF CUTPNFP feasible  CUTPNFP detects LRF Both infeasible  MNFP detects LRF Minimal-MUMCUT: Always need MUTP tests to detect LIF CUTPNFP tests only when MUTP infeasible MNFP tests only when both are infeasible “Minimal” means that every test in the test set is needed to guarantee fault detection – not minimized

New Fault Hierarchy Black arrow: relation always holds Green arrow: relation holds if MUTP is feasible Red arrow: relation holds if MUTP is infeasible and CUTPNFP is feasible LOF ORF. LRF LNF TNF ENF LIF TOF ORF+

Case Study Analyzed 19 Boolean predicates in an avionics software system (Weyuker, Chen, Lau, and Yu) Number of unique literals range: 5 to 13 Determined MUTP feasibility for each term and CUTPNFP feasibility for each literal Examined test set size for MUMCUT vs. Minimal-MUMCUT

Case Study Results Minimal-MUMCUT size is 12% of MUMCUT size Savings in test set size comes from 1) CUTPNFP feasible for all 853 literals: no MNFP 2) For 24% of literals, MUTP detects LRF: no CUTPNFP 3) 16 of 19 predicates had a MUTP feasible term

Test Set Size vs. Number of Unique Literals

Conclusion Used criterion feasibility to reduce test set size without sacrificing fault detection Modification of fault detection relations in Lau and Yu’s hierarchy based on criterion feasibility Introduction of the Minimal-MUMCUT criterion based on minimal DNF Applications for software testing of programs with large predicates