1. Software Testing – Lecture #2 Thomas Ball with material from M. Young, A. Memon and MSR’s FSE group

2. Testing - The Story So Far

3. Two Threads Specification-based test generation –Foundations of Software Engineering Implementation-based test generation –Testing, Verification and Measurement Unifying Themes – create finite-state systems from infinite-state systems via predicates – use finite-state algorithms to guide test generation

4. Specification-based Testing What does this API do? –Executable spec prescribes potential behavior Where do concrete tests come from? –Explore behavior of spec, generate test strategies Does a test execution succeed or fail? –Use spec as oracle for runtime verification How do we know when we’re done testing? –Cover spec (and implementation) What do we know when we’re done testing? –Model and implementation agree!

5. 4 Steps to Testing Heaven –Modeling define infinite transition system –Exploration reduce to finite test graph –Gaming generate test strategies –Monitoring verify conformance

6. Example: Alternating Bit Protocol This protocol works on the producer-consumer model. Producer wants to send messages in a reliable manner to a consumer through unreliable channel

7. 1 st Step: Model the Infinite Transition System Describe (transition system in Spec# of) all possible runs of ABP Types –Msg, Ack State –Bitstatus, … Controllable action –Send Observable events: –Receive –Lose Msg –Lose Ack s0s0 s1s1 s0s0 s2s2 s3s3 s4s4 s5s5 s3s3 s2s2 s6s6 Send ?LoseMsg ?Receive ?LoseAck Send ?LoseMsg ?Receive ?Timeout Send

8. ABP: Data Structures class DataMsg {// Message readonly Data data; readonly bool bit; } class Ack {// Acknowledgement readonly bool bit; }

9. ABP: State bool SenderBit = true; int SenderRecordNr = 0; Ack SenderInbox = null; DataMsg ReceiverInbox = null; bool ReceiverBit = true; Seq ReceivedFile=Seq{}; Sends Message Returns Ack

10. ABP: Transitions void Send () { require AcknowledgmentArrived() || Timeout(); if(AcknowledgmentArrived()) { if(AcknowledgmentHasTheRightBit()) { ReceiverInbox = DataMsg(SenderRecordNr + 1, !SenderBit); SenderRecordNr += 1; SenderBit = !SenderBit; } SenderInbox = null; } else if(Timeout()) { ReceiverInbox = DataMsg(SenderRecordNr, SenderBit); }} Enabling Condition State change

11. ABP: Modeling Demo Show model in SpecExplorer Simulate as a console application

12. 2 nd Step: Finitize the Model Generate “all” possible behaviors (test graph): –At each state, execute any enabled method with any allowed argument values –Nondeterministic choice from enabled method explores possible interleavings Control the search, using –Finite unfolding –Stochastic means –User provided predicate abstractions [ISSTA 2002] This generates a test graph

13. User provided predicate abstraction Tree can be pruned, if an equivalent state has already been visited Example: Generate an FSM from a stack specification. Observation property: stack.IsEmpty() [0,0] Push(0) true false Push(1) Push(0) Top() Pop() [] Push(0) [0] Pop() Top() Pop() Push(1) [1,0] Top()

14. Test graph A test graph G is a directed graph s.t. there are two kinds of vertices in G: –states –choice points (CP) every edge e has a prob. p(e) s.t. for every CP u ∑{p(e): source of e = u}=1, there is a non-negative cost function c defined on edges state vertex p=0.3 c=1 p=0.7 c=2 CP vertex c=1

15. ABP: Modeling Demo Explorer options

16. 3 rd Step: Generate Test Strategies  Cover all edges of the test graph (edge coverage)  Reach certain goal states in the test graph (reachability game) [ISSTA 2004] 1.With maximal probability and minimal cost 2.With full certainty and minimal cost

17. Edge Coverage Strategy 1.Produce a tour of the edges in the graph (e.g. Chinese Postman tour) 2.Divide the tour into segments starting and ending at choice points 3.At a choice point IUT selects an edge e and TT chooses randomly any segment s starting from e and follows s until the end (s ends in a choice point).

18. Edge Coverage Example The tour is [e8,e2,e1,e5,e7, e9,e4,e3,e6,e7] The segments are: [e8,e2,e1,e5,e7] [e9,e4,e3,e6,e7] s1 e1 e2 e4 e3 e9 e8 e7 e5 e6

19. 4 th Step: Execute Tests and Verify Results Define conformance notion: Probes Rewrite the IUT to insert call backs into model Specify upper bounds on number of repetitions

20. Probes Given an abstract domain X A probe P m is a function from model states to X A probe P i is a function from implementation states to X Model state A and impl state B conform wrt the pair of probes (P m, P i ) if P m (A) = P i (B) Different probes may be enabled at different states A1A1 A2A2 B1B1 B2B2 B4B4 B3B3 B5B5 A3A3 P1mP1m P1iP1i X P2mP2m P2iP2i

21. IUT Model Event Buffering During conformance testing all events are buffered in an unbounded queue All enabled probes are checked at every state e1 Events e2 a1 Action Calls s0 s4 s1 s2 s3 a1 e1e2 a2 t0 t2 t1 a1’ (P1m,P1i)(P1m,P1i) e1’ (P2m,P2i)(P2m,P2i)

22. ABP Demo Test sequence generation Conformance testing

23. Experience Modeling for Testing –Models are created by testers, not designers –Models are based on informal specs, not impl. –Models are not comprehensive, only issues Several projects –Web services, Passport, Mediaplayer, Distributed File replication system –Models up to 100 pages More than 50 people are using it on a daily basis, steep uptake

24. More Info Public release of Spec# this Summer http://research/microsoft.com/foundations Contact schulte@microsoft.com if you would like to know more or are interested in a summer internship in 2005!schulte@microsoft.com Consider submitting something for ICFEM 2004 http://research/microsoft.com/chttp://research/microsoft.com/conferences/ICFE M2004

25. MSIL Unit Test Tool a hybrid helper Goal capture developer knowledge ASAP via a strong set of unit tests to form a specification of the code’s behavior How –generate tests based on analysis of MSIL –symbolic execution + constraint satisfaction –runtime analysis to check complicated invariants Facets –complements specification-based test generation –positive feedback cycle with programmer

26. What criteria should guide unit test generation?

27. Predicate-complete Testing Predicates –relational expression such as (x<0) –the expression (x 0) has two predicates –predicates come from program and safe runtime semantics Consider a program with m statements and n predicates –predicates partition input domain –m x 2 n possible observable states S Goal of Predicate-complete Testing: –cover all reachable observable states R  S

28. PCT Coverage L2: if (A || B) S else T L3: if (C || D) U else V PCT requires covering all logical combinations over {A,B,C,D} at –L2 and L3 –S, T, U and V Some combinations may not be reachable

29. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

30. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

31. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

32. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

33. L1: if (p) L2: if (q) L3: x=0; L4: y=p+q; Path Coverage does not imply PCT Coverage

34. L1: if (p) L2: if (q) L3: x=0; L4: y=p+q; Path Coverage does not imply PCT Coverage

35. Denominator Problem Coverage metrics require a denominator –e.g. statements executed / total statements Easy to define for observable states –executed observable states / (m x 2 n ) But (m x 2 n ) is not a very good denominator! –most observable states will not be reachable –R <<< S

36. Upper and Lower Bounds m x 2 n possible states S Upper bound U Reachable states R Lower bound L Bound reachable observable states – modal transition systems and predicate abstraction – |L| / |U| defines “goodness” of abstraction Test generation using lower bound L Refinement to increase |L| / |U| ratio

37. a a’ may MCMC MAMA   a a’ total MCMC MAMA   a a’ total & onto   a a’ onto   Abstraction Construction

38. Upper Bound: May-Reachability a b c may a b c

39. Upper Bound: May-Reachability a b c may a b c

40. c d total a b onto Pessimistic Lower Bound may

41. c d a b Pessimistic Lower Bound may onto total

42. c d a b Pessimistic Lower Bound may onto total

43. void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); } Example void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }

44. Observation Vector [ lo pivot ] lo<hi  lo<=hi  lo pivot)  (  a[lo] pivot) Only 10/16 observations possible

45. 13 labels x 10 observations = 130 observable states But, program constrains reachable observable states greatly. void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ; }

46. void partition() { decl lt, le, al, ah; enforce ( (lt=>le) & ((!lt&le)=>(al&!ah)|(!al&ah)) ); lt,le,al,ah := T,T,*,*; L0: while (le) { L1: ; L2: while (al) { L3: lt,le,al := (!lt ? F:*), lt, *; L4: ;} L5: while (ah) { L6: lt,le,ah := (!lt ? F:*), lt, *; L7: ;} L8: if (lt) { L9: al,ah := !ah,!al; LA: ;} LB: ;} LC: ; } Boolean Program

47. State Space of Boolean Program Upper Bound = 49 states [ lo pivot ]

48. plaintext

49. Test Generation DFS of L p generates covering set of paths Symbolically execute paths to generate tests Run program on tests to find errors and compute coverage of observable states

50. Array bounds violations Generated Inputs (L0:TTTT,L4:FTFT) { 0,-8,1 } (L0:TTTT,L4:TTFT) { 0,-8,2,1 } (L0:TTTT,L4:TTTT) { 0,-8,-8,1 } (L0:TTTF,L4:TTFF) { 1,-7,3,0 } (L0:TTTF,L4:FTTF) { 0,-7,-8 } (L0:TTTF,L4:TTTF) { 1,-7,-7,0 } (L0:TTFT,L7:TTFF) { 0,2,-8,1 } (L0:TTFT,L7:FTFT) { 0,1,2 } (L0:TTFT,L7:TTFT){ 0,3,1,2 } (L0:TTFF,L0:TTTT) { 1,2,-1,0 } void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ; }

51. Results Buggy partition function –U=49, L=43, Tested=42 Fixed partition function –U=56, L=37, Tested=43 What about the remaining 13 states?

52. Refinement

53. New Observation Vector [ lo<hi, lo<=hi, lo=hi+1, a[lo] pivot, a[lo-1] pivot ] Only 48/128 observations possible For this set of predicates, L p = U

54. Conclusions PCT coverage –new form of state-based coverage –similar to path coverage but finite Upper and lower bounds –computed using predicate abstraction and modal transitions –use lower bound to guide test generation –refine bounds