A Theory of Predicate-complete Test Coverage and Generation Thomas Ball Testing, Verification and Measurement Microsoft Research FMCO Symposium November

Control-flow Coverage Criteria Statement/branch coverage widely used in industry 100% coverage ≠ a bug-free program!! More stringent criteria –modified-condition-decision, predicate, data- flow, mutation, path, …

Beyond Statement and Branch Coverage void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

Beyond Statement and Branch Coverage void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

Beyond Statement and Branch Coverage void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

Corrected Program void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (lo<=hi && a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

Corrected Program void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (lo<=hi && a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

Predicate-complete Testing Program predicates –relational expression such as (x<0) –the expression (x 0) has two predicates Program with m statements and n predicates –m x 2 n possible observable states S –finest partition of behavior based on programmer’s observations Goal –cover all reachable observable states R  S

Reachable Observable States L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

Upper and Lower Bounds m x 2 n possible states S Upper bound U Reachable states R Lower bound L Bound reachable observable states R – predicate abstraction – modal transition systems – |L| / |U| defines “goodness” of abstraction Test generation using L Increase |L| / |U| ratio

Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

Predicate Abstraction of Infinite-state Systems –Graf & Saïdi, CAV ’97 –Abstract Interpretation, Cousot & Cousot ‘77 Idea –Given set of predicates P = { P 1, …, P k } Formulas describing properties of system state Abstract State Space –Set of Abstract Boolean variables B = { b 1, …, b k } b i = true  Set of states where P i holds

a a’ may MCMC MAMA   a a’ total MCMC MAMA   a a’ total & onto   a a’ onto   Modal Transitions [Larsen]

Predicate Abstraction if Q  SP(P,s) then(P,Q)  onto P SP(P,s) Q Q WP(s,Q) P if P  WP(s,Q) then(P,Q)  may Q WP(s,Q) P if P  WP(s,Q) then(P,Q)  total

Example

Upper Bound: May-Reachability a b c may a b c

Upper Bound: May-Reachability a b c may a b c

c d total a b onto Lower Bound may

c d a b Lower Bound may onto total

c d a b Lower Bound may onto total

Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); } Example

Observation Vector [ lo pivot ] lo<hi  lo<=hi  lo pivot)  (  a[lo] pivot) Only 10/16 observations possible

13 labels x 10 observations = 130 observable states But, program constrains reachable observable states greatly. void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ; }

Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

Test Generation DFS of lower bound generates covering set of paths Symbolically execute paths to generate tests Run program on tests to find errors and compute coverage of observable states

{ 0,-7,-8 }

Array bounds violations Generated Inputs (L0:TTTT,L4:FTFT) { 0,-8,1 } (L0:TTTT,L4:TTFT) { 0,-8,2,1 } (L0:TTTT,L4:TTTT) { 0,-8,-8,1 } (L0:TTTF,L4:TTFF) { 1,-7,3,0 } (L0:TTTF,L4:FTTF) { 0,-7,-8 } (L0:TTTF,L4:TTTF) { 1,-7,-7,0 } (L0:TTFT,L7:TTFF) { 0,2,-8,1 } (L0:TTFT,L7:FTFT) { 0,1,2 } (L0:TTFT,L7:TTFT){ 0,3,1,2 } (L0:TTFF,L0:TTTT) { 1,2,-1,0 } void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ; }

Results Buggy partition function –U=49, L=43, Tested=42 Fixed partition function –U=56, L=37, Tested=43 What about the remaining 13 states?

Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

Refinement

New Observation Vector [ lo<hi, lo<=hi, lo=hi+1, a[lo] pivot, a[lo-1] pivot ] Only 48/128 observations possible For this set of predicates, L = U

Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

Discussion Comparison to bisimulation Completeness of abstractions Related work

Bisimulation

Abstraction Completeness

Related Work Predicate abstraction Modal transition systems Abstraction-guided test generation Symbolic execution/constraint satisfaction Test coverage criteria

PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

L1: if (p) L2: if (q) L3: x=0; L4: y=p+q; Path Coverage does not imply PCT Coverage

L1: if (p) L2: if (q) L3: x=0; L4: y=p+q; Path Coverage does not imply PCT Coverage

Conclusions PCT coverage –new form of state-based coverage –similar to path coverage but finite Upper and lower bounds –computed using predicate abstraction and modal transitions –use lower bound to guide test generation –refine bounds