1. A Theory of Predicate-complete Test Coverage and Generation Thomas Ball Testing, Verification and Measurement Microsoft Research FMCO Symposium November 2-5 2004

2. Control-flow Coverage Criteria Statement/branch coverage widely used in industry 100% coverage ≠ a bug-free program!! More stringent criteria –modified-condition-decision, predicate, data- flow, mutation, path, …

3. Beyond Statement and Branch Coverage void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

4. Beyond Statement and Branch Coverage void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

5. Beyond Statement and Branch Coverage void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

6. Corrected Program void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (lo<=hi && a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

7. Corrected Program void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (lo<=hi && a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

10. Predicate-complete Testing Program predicates –relational expression such as (x<0) –the expression (x 0) has two predicates Program with m statements and n predicates –m x 2 n possible observable states S –finest partition of behavior based on programmer’s observations Goal –cover all reachable observable states R  S

11. Reachable Observable States L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

12. Upper and Lower Bounds m x 2 n possible states S Upper bound U Reachable states R Lower bound L Bound reachable observable states R – predicate abstraction – modal transition systems – |L| / |U| defines “goodness” of abstraction Test generation using L Increase |L| / |U| ratio

13. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

14. Predicate Abstraction of Infinite-state Systems –Graf & Saïdi, CAV ’97 –Abstract Interpretation, Cousot & Cousot ‘77 Idea –Given set of predicates P = { P 1, …, P k } Formulas describing properties of system state Abstract State Space –Set of Abstract Boolean variables B = { b 1, …, b k } b i = true  Set of states where P i holds

15. a a’ may MCMC MAMA   a a’ total MCMC MAMA   a a’ total & onto   a a’ onto   Modal Transitions [Larsen]

16. Predicate Abstraction if Q  SP(P,s) then(P,Q)  onto P SP(P,s) Q Q WP(s,Q) P if P  WP(s,Q) then(P,Q)  may Q WP(s,Q) P if P  WP(s,Q) then(P,Q)  total

17. Example

18. Upper Bound: May-Reachability a b c may a b c

19. Upper Bound: May-Reachability a b c may a b c

20. c d total a b onto Lower Bound may

21. c d a b Lower Bound may onto total

22. c d a b Lower Bound may onto total

23. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

24. void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); } Example

25. Observation Vector [ lo pivot ] lo<hi  lo<=hi  lo pivot)  (  a[lo] pivot) Only 10/16 observations possible

26. 13 labels x 10 observations = 130 observable states But, program constrains reachable observable states greatly. void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ; }

28. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

29. Test Generation DFS of lower bound generates covering set of paths Symbolically execute paths to generate tests Run program on tests to find errors and compute coverage of observable states

32. { 0,-7,-8 }

33. Array bounds violations Generated Inputs (L0:TTTT,L4:FTFT) { 0,-8,1 } (L0:TTTT,L4:TTFT) { 0,-8,2,1 } (L0:TTTT,L4:TTTT) { 0,-8,-8,1 } (L0:TTTF,L4:TTFF) { 1,-7,3,0 } (L0:TTTF,L4:FTTF) { 0,-7,-8 } (L0:TTTF,L4:TTTF) { 1,-7,-7,0 } (L0:TTFT,L7:TTFF) { 0,2,-8,1 } (L0:TTFT,L7:FTFT) { 0,1,2 } (L0:TTFT,L7:TTFT){ 0,3,1,2 } (L0:TTFF,L0:TTTT) { 1,2,-1,0 } void partition(int a[]) { assume(a.length>2); int pivot = a[0]; int lo = 1; int hi = a.length-1; L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ; }

34. Results Buggy partition function –U=49, L=43, Tested=42 Fixed partition function –U=56, L=37, Tested=43 What about the remaining 13 states?

35. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

36. Refinement

37. New Observation Vector [ lo<hi, lo<=hi, lo=hi+1, a[lo] pivot, a[lo-1] pivot ] Only 48/128 observations possible For this set of predicates, L = U

39. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

40. Discussion Comparison to bisimulation Completeness of abstractions Related work

41. Bisimulation

43. Abstraction Completeness

45. Related Work Predicate abstraction Modal transition systems Abstraction-guided test generation Symbolic execution/constraint satisfaction Test coverage criteria

46. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

47. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

48. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

49. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

50. L1: if (p) L2: if (q) L3: x=0; L4: y=p+q; Path Coverage does not imply PCT Coverage

51. L1: if (p) L2: if (q) L3: x=0; L4: y=p+q; Path Coverage does not imply PCT Coverage

52. Conclusions PCT coverage –new form of state-based coverage –similar to path coverage but finite Upper and lower bounds –computed using predicate abstraction and modal transitions –use lower bound to guide test generation –refine bounds