1. Test Input Generation for Java Containers using State Matching Willem Visser Corina Pasareanu and Radek Pelanek Automated Software Engineering Group NASA Ames Research Center

2. Background ISSTA 2004 –White-box Test Input Generation for Red-Black Trees Symbolic Execution of structural invariants –Korat-style Symbolic Execution of actual code –Verdict Got branch coverage for structures of < 7 nodes –Could not go much higher either Satisfactory results, but is this how one would really test containers?

3. Background (2) More natural to test using Black-box techniques –Test cases based on API calls and parameters Additional inspiration from Paolo Tonella –Email exchange on comparing with his evolutionary testing framework that works at the API level But there are many ways to do test generation at the API level…which way should we go –As many ways as we can! How to compare these? –Measure the coverage we get

4. General Idea SUT ENV (m,n) m is the seq. length of API calls & n is the number of values used in the parameters of the calls API … put(v) del(v) Evaluate different techniques for selecting test-cases from ENV(m,n) to obtain maximum coverage

5. SUT – Container Classes Binary Tree –154 LOC –add(x), remove(x) Red-black Trees (java.util.TreeMap) –580 LOC –put(x), remove(x) Fibonacci Heap –286 LOC –Insert(x), delete(x), removeMin() Binomial Heap –355 LOC –insert(x), delete(x), extractMin(), decreaseKey(x,y)

6. Coverage Measures Basic Block, i.e. Statement, Coverage –Simplest form of coverage –Aligned with “simple” bugs Predicate Coverage –Cover all combinations of a given set of predicates at each branch in the code –Much harder coverage to obtain –Aligned with “deeper” behavioral bugs Question: why not seed bugs? –Only automatic way of doing this is through mutation which is more likely to give easy bugs to find

7. Predicate Coverage Cover all combinations of a given set of predicates at each branch in the code Red-Black Tree Predicates root = null, e.left = null, e.right = null, e.parent = null, e.color = BLACK

8. public void add(int x) { Node current = root; if (root == null) { gen(0, current, x, null, null); root = new Node(x); return; } while (current.value != x) { if (x < current.value) { if (current.left == null) { gen(1, current, x, null, null); current.left = new Node(x); } else { gen(2, current, x, null, null); current = current.left; } } else { if (current.right == null) { gen(3, current, x, null, null); current.right = new Node(x); } else { gen(4, current, x, null, null); current = current.right; } } } }

9. public static int gen_native(MJIEnv env, int objRef, int br, int n0, int x, int n1, int n2) { String res = br + ","; int temp; if (n0 == -1) { res += "-"; } else { temp = env.getIntField(n0, null, "left"); res += (temp == -1) ? "L-" : "L+"; temp = env.getIntField(n0, null, "right"); res += (temp == -1) ? "R-" : "R+"; } res += (n1 == -1) ? "P-" : "P+"; if (n2 == -1) { res += "B-"; } else { temp = env.getIntField(n2, null, "left"); res += (temp == -1) ? "BL-" : "BL+"; temp = env.getIntField(n2, null, "right"); res += (temp == -1) ? "BR-" : "BR+"; } if (!tests.contains(res)) { tests.add(res); System.out.println("Test case number " + tests.size() + " for '" + res + "': "); return tests.size(); } return 0; }

10. Let’s Go Outside the Box Rather than over-approximate and refine, we under-approximate and refine –Clearly complements existing techniques If we restrict ourselves only to feasible behaviors when under-approximating then all safety property violations will be preserved Build on top of classic explicit-state model checking infrastructure

11. Classic Explicit-State Search PROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF s' NOT IN VisitedStates THEN Enter s' into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; } INIT { Enter s0 into VisitedStates; Push s0 onto Stack; dfs(); }

12. Explicit-State (1-step) αSearch PROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF α(s‘) NOT IN VisitedStates THEN Enter α(s‘) into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; } INIT { Enter α(s0) into VisitedStates; Push s0 onto Stack; dfs(); }

13. αSearch 1: x = 2; 2: while (x>0) 3: x = x - 1; 4: assert false; Abstraction Mapping p = (x>0) Map concrete states to abstract states for state storing 1,p 2,p 3,p Always traverse only feasible paths Under-approximation of the behaviors

14. Symbolic Execution and αSearch Current implementation is for a simple input language –oCaml using Simplify as a decision procedure We would like to integrate the technique in Java Pathfinder (JPF) that supports symbolic execution (using the Omega Library) –To allow application to programs with complex data structures (objects) Idea –Execute symbolically but along concrete path –Whenever the symbolic analysis can follow alternatives, add predicates for the alternative path

15. End of Part One Showed under-approximation based search with refinement –Backward weakest precondition based –Suggested forward symbolic execution based Part Two –Rather than automated refinement we use user-provided abstractions –Motivation is to generate test-cases to achieve high behavioral coverage for Java container classes

16. Explicit-State (1-step) αSearch PROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF α(s‘) NOT IN VisitedStates THEN Enter α(s‘) into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; } INIT { Enter α(s0) into VisitedStates; Push s0 onto Stack; dfs(); }

17. Techniques Considered Random selection Classic model checking –State matching on complete state Abstraction search –State matching on abstract (partial) state Symbolic Execution –Complete matching using subsumption checks –Abstract matching

18. Framework SUT with minor instrumentation ENV TestListener Abstraction Mapping + State Storage Coverage Manager JPF

19. Sample Output Test case number 77 for '15,L+R+P-REDroot': put(0);put(4);put(5);put(1);put(2);put(3);remove(4); Unique ID for the test Branch NumberPredicate Values Test-case to achieve above coverage Test case number 7 for '32,L-R-P+RED': X2(0) == X1(0) && X2(0) < X0(1) && X1(0) < X0(1) put(X0);put(X1);remove(X2); Test case number 7 for '32,L-R-P+RED': put(1);put(0);remove(0); Concrete Symbolic Path Condition with solutions Symbolic TC

20. Environment Skeleton M : sequence length N : parameter values A : abstraction used for (int i = 0; i < M; i++) { int x = Verify.random(N - 1); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break; } Verify.ignoreIf(checkAbstractState(A));

21. Symbolic Environment Skeleton M : sequence length A : abstraction used for (int i = 0; i < M; i++) { SymbolicInteger x = new SymbolicInteger(“X“+i); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break; } Verify.ignoreIf(checkAbstractState(A));

22. Abstraction Search Map state to an abstract version and backtrack if the abstract state was seen before, i.e. discard test-case Mapping can be lossy or not Abstraction mappings can be created by the user/tester Default abstraction mappings are provided

23. Default Mappings Structure of the heap of the program –e.g. structure of the containers Structure augmented with non-data fields Structure augmented with symbolic constraints on the data in the structure –This requires checking constraint subsumption

24. Linearization Comparing Structures 1 2 34 5 1 2 34 5 1 2 34 5 1 2 34 5 1 2 3 -1 -1 4 -1 -1 5 -1 -1 1 2 3 -1 -1 4 -1 5 -1 -1 -1

25. 4 3 2 1 Linearization + Mapping 1 2 3 4 5 1b 2b 3r -1 -1 4r -1 -1 5b -1 -1 5 1b 2r 3r -1 -1 4r -1 -1 5r -1 -1 Linearization takes a mapping object as parameter to indicate how each node in the heap should be linearized. In the example above each node gets, besides the unique identifier, a mapping of “r” if the original structure had a red node and “b” if the original structure had a black node in that position. If we also added the key values for each node the linearization might have looked something like: 1b6 2b4 3r3 -1 -1 4r5 -1 -1 5b7 -1 -1

26. Symbolic Execution Symbolic State x1 x2 x3x4 x5 + x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1 Shape Symbolic Constraints

27. Subsumption Checking x1 x2 x3x4 x5 + x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1 x1 x2 x3x4 x5 + x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1 If only it was this simple!

28. Getting Ready for Checking Existential Elimination x1 x2 x3x4 x5 PC s1 s3 & s4 < s1 & s4 < s5 & s7 s1 s1 s4 s2 s3s5 +  s1,s2,s3,s4,s5 such that x1 = s1 & x2 = s4 & x3 = s3 & x4 = s5 & x5 = s2 & PC x1 > x2 & x2 > x3 & x2 x1

29. Checking Subsumption new state = C1 & C2 & ….. old state = C1’ & C2’ & ….. Check new => old !(new => old) is unsatisfiable = !( !new \/ old ) = new & !old = C1 & C2 … & (!C1’ \/ !C2’ \/…) = (C1 & C2 & !C1’) \/ (C1 & C2 & !C2’) \/ … Oops: Negation and Or doesn’t work in our version of the Omega Lib.

30. Bidirectional Subsumption Checking If new => old –backtrack If old => new –new is more general than old –replace old with new to increase chances of getting a match in the future –Continue on path from new, i.e. don’t backtrack Ultimately for each shape we want to use disjunction of constraints –Small technicality prevents us – bug in omega lib

31. Evaluation Red-Black Trees Out of Memory runs are not reported Breadth-first Search unless stated Sequence Length = Values for the non-symbolic searches First compare under Branch Coverage

32. Exhaustive Techniques Branch Coverage SeqCovLenTimeMem Full MC7394.3536584 S+C+V7394.310.63517.47 Sym – S+Sub7394.314.20116.95 Optimal Branch Coverage is 39

33. Under-Approximation Techniques Branch Coverage SeqCovLenTimeMem S21396.157.35372.07 S+C18395.832.57721.16 Sym - S7394.310.05415.43 Sym – S+C7394.311.99810.76 Random939740.4293.06 Optimal Branch Coverage is 39

34. Exhaustive Techniques Predicate Coverage SeqCovLenTimeMem Full MC7795.2543309 S+C+V10955.7350228 Sym – S+Sub111026.1222117 Optimal Predicate Coverage is 106

35. Under-Approximation Techniques Predicate Coverage SeqCovLenTimeMem S25d10621.79013.31 S+C301068.3354100 Sym - S121006.1230123.27 Sym – S+C121046.2356138 Random6010630.161.4597.74 Optimal Predicate Coverage is 106

36. Observations For a simple coverage such as branch coverage, all the techniques work well, including the exhaustive ones But making the coverage more “behavioral”, even by a small increment, kills off the exhaustive techniques

37. Observations Full Blown Model Checking doesn’t work here Its close cousin, that only looks at the relevant state at the relevant time, scales much better Branch - full coverage after: –MC: 536s & 584Mb –Complete: 10s & 17Mb Predicate – best coverage after: –MC: 79 covered with 543s & 309Mb –Complete: 95 covered with 350s & 228Mb

38. Observations Symbolic techniques have a slight edge over concrete ones for exhaustive analysis –Comparing for Predicate Coverage (10) Full Concrete(95): 350s & 228Mb Full Symbolic(95): 123s & 62Mb Current results indicate symbolic under- approximation based search is less efficient than concrete Further experimentation required

39. Observations Random Search? –Seems to work rather well here It will always have an edge on memory, since it uses almost none It will most likely have an edge on speed, since it needs to do little additional work – it will however redo work often It will in general do worse on test-case length, since it requires longer sequences to achieve more complex coverage

40. Observations Search Order Matters for the lossy techniques –BFS is inherently better than DFS –On occasion though it is the other way round

41. Conclusions & Future Work Showed how predicate abstraction can be used for an under-approximation based search with refinement Showed how a lightweight variant, where the abstraction mapping is given and no refinement is done, can be used for bug-finding and test- case generation Goal: Derive predicates for analyzing containers automatically through the use of symbolic execution during refinement –Can we derive shape predicates automatically?