1. EFFICIENT EXPLICIT-STATE MODEL CHECKING FOR PROGRAMS WITH DYNAMICALLY ALLOCATED DATA Marcelo d’Amorim Ph.D. thesis committee: Gul Agha Jennifer Hou Darko Marinov (advisor, chair) Mahesh Viswanathan August 06, 2007

2. 2 Testing is important Programmers develop software with lots of errors Testing is the dominant approach in industry to assure software quality But...testing is expensive [NIS02] test automation can assist the programmer in finding errors

3. 3 model checkers are tools that can assist in the automation of software testing goal: improve model checking for software testing

4. 4 Reminder about Prelim exam Work presented at Prelim Symclat (ASE’06 paper) Improved software testing without model checking Mixed Execution (ICFEM’06 paper) More efficient model checking Delta Execution Idea More efficient model checking Work done since Prelim Delta Execution (ISSTA’07 paper) focus of my thesis

5. 5 Model Checking and Testing model checker model property “interesting” behaviors Model describes behaviors of a system description includes states and transition The model checker performs state-space exploration on the input model … performs systematic testing

6. 6 Design and implementation models Traditionally, model checkers operate on design models FDR [Ros94], SPIN [Hol97], NuSMV [CCGR99], Alloy Analyzer [Jac00]… Recently, more common for model checkers to operate on programs JPF [JV01], CMC [MPC+02], Bogor [RDH03], SpecExplorer [VCST05]…

7. 7 Design and implementation models Traditionally, model checkers operate on design models FDR [Ros94], SPIN [Hol97], NuSMV [CCGR99], Alloy Analyzer [Jac00]… Recently, more common for model checkers to operate on programs JPF [JV01], CMC [MPC+02], Bogor [RDH03], SpecExplorer [VCST05]… focus

8. 8 Goal speed up state-space exploration for programs with dynamically allocated data (/OO programs)

9. 9 Example: test subject BinarySearchTree (BST) and Node classes: public class BinarySearchTree { private Node root; private int size; public void add(int info) {…} public boolean remove(int info) {…} } class Node { Node left, right; int info; Node(int info) { this.info = info } } operations state 1 2 2

10. 10 Example: test sequence sequences of method calls from the initial state BinarySearchTree bst = new BinarySearchTree(); bst.add(1); bst.add(2); BinarySearchTree bst = new BinarySearchTree(); bst.add(2); bst.remove(1); Examples of sequences

11. 11 Example scenario: Bounded-Exhaustive Exploration explore all sequences up to given bounds: length of sequence range of values BinarySearchTree bst = new BinarySearchTree(); bst.add(1); bst.add(2); BinarySearchTree bst = new BinarySearchTree(); bst.add(2); bst.remove(1); Examples of sequences

12. 12 Example scenario: Bounded-Exhaustive Exploration // L: length of sequence // N: number of input values // in most experiments L == N public static void main(int L, int N) { BST bst = new BST(); // empty tree for (int i = 0; i < L; i++) { int methNum = Verify.getInt(0, 1); int value = Verify.getInt(1, N); switch (methNum) { case 0: bst.add(value); break; case 1: bst.remove(value); break; } stopIfVisited(bst); } bounds

13. 13 0 Example: Bounded-Exhaustive Exploration  add(1) chooses method and value add(2)remove(1) remove(2) L = N = 2

14. 14 0 Example: Bounded-Exhaustive Exploration  add(1) chooses method and value executes transition add(2)remove(1) remove(2) 1 1 L = N = 2

15. 15 0 Example: Bounded-Exhaustive Exploration  add(1) chooses method and value executes transition explores paths (backtracking) add(2) remove(1) remove(2) 1 1 L = N = 2

16. 16 0 Example: Bounded-Exhaustive Exploration  add(1) chooses method and value executes transition explores paths (backtracking) add(2) remove(1) remove(2) 2 1 1 1 L = N = 2

17. 17 0 Example: Bounded-Exhaustive Exploration  add(1) chooses method and value executes transition explores paths (backtracking) prunes paths (compares state) add(2) remove(1) remove(2) 2 1 1 1 0 L = N = 2

18. 18 0 Example: Bounded-Exhaustive Exploration  add(1) chooses method and value executes transition explores paths (backtracking) prunes paths (compares state) add(2) remove(1) remove(2) 2 1 1 1 0 0 L = N = 2

19. 19 0 Example: Bounded-Exhaustive Exploration  add(1) chooses method and value executes transition explores paths (backtracking) prunes paths (compares state) add(2) remove(1) remove(2) 2 1 1 1 0 0  add(1) 1 1 … L = N = 2

20. 20 0 Example: Bounded-Exhaustive Exploration  add(1) chooses method and value executes transition explores paths (backtracking) prunes paths (compares state) add(2) remove(1) remove(2) 2 1 1 1 0 0  add(1) 1 1 1 2 2 add(2) … L = N = 2

21. 21 Operations Costly operations for an important class of model checkers: execution of transitions exploration of paths (backtracking) pruning of paths (state comparison) specific goal is to speed up these operations

22. 22 Our contributions Two techniques Mixed execution Delta execution (∆Execution) Overall time reduction execution (mixed and ∆) backtracking (∆) state comparison (∆)

23. 23 Outline Introduction & Goal ∆Execution Mixed execution (brief) Related Work and Conclusions

24. 24 Our Technique: ∆Execution Observation: Many of the execution paths in state-space exploration partially overlap Different states can hold same values ∆Execution exploits these overlaps across executions and states to speed up model checking

25. 25 Example 31 2 3 3 1 2 3 BST #2BST #1 Consider separately invoking the method call add(4) on the following two Binary Search Trees

26. 26 Example 31 2 3 1: public void add(int elem) 2: if (root == null) 3: root = new Node(elem) 4: else 5: for (Node temp = root; true; ) 6: if (temp.info < elem) 7: if (temp.right == null) { 8: temp.right = new Node(elem) 9: break; 10: } else temp = temp.right; 11: } else if (temp.info > elem) { 12: if (temp.left = null) { 13: temp.left = new Node(elem); 14: break; 15: } else temp = temp.left; 16: } else return; // no duplicates 17: size++; 18: }

27. 27 Example Execution Trace:, 2 1, 4, 5, 6, 7, 10 5, 6, 7, 8, 9, 17 1: public void add(int elem) 2: if (root == null) 3: root = new Node(elem) 4: else 5: for (Node temp = root; true; ) 6: if (temp.info < elem) 7: if (temp.right == null) { 8: temp.right = new Node(elem) 9: break; 10: } else temp = temp.right; 11: } else if (temp.info > elem) { 12: if (temp.left = null) { 13: temp.left = new Node(elem); 14: break; 15: } else temp = temp.left; 16: } else return; // no duplicates 17: size++; 18: } 31 2 3 4 31 2 4

28. 28 Example 31 2 3 4 31 2 4 3 2 1 3 3 2 1 4 4 Execution Trace:, 2 1, 4, 5, 6, 7, 10 5, 6, 7, 8, 9, 17 1: public void add(int elem) 2: if (root == null) 3: root = new Node(elem) 4: else 5: for (Node temp = root; true; ) 6: if (temp.info < elem) 7: if (temp.right == null) { 8: temp.right = new Node(elem) 9: break; 10: } else temp = temp.right; 11: } else if (temp.info > elem) { 12: if (temp.left = null) { 13: temp.left = new Node(elem); 14: break; 15: } else temp = temp.left; 16: } else return; // no duplicates 17: size++; 18: }

29. 29 Execution Trace: Example Execution, 2 1, 4, 5, 6, 7, 10 5, 6, 7, 8, 9, 17 4 31 2 4 3 2 1 4 4 overlap across executions overlap across states

30. 30 ∆Execution Technique: uses “set of states” to perform state- space exploration Exploits the overlapping on states and executions Reduce time: execution of transitions exploration of paths (backtracking) pruning of paths (state comparison)

31. 31 standard exploration 0  2 1 1 1  1 2 2  2 2 1 0 0 2 1 2 1 1 1 0 0 1 1

32. 32 0  1 1 2 1 00 2 1 1 1 2 2 1 1 1 1 2 2 2 1 2 1 1 1 0 0  1 2 2 2 2 1 0  2 1 1 1  1 2 2  2 2 1 standard exploration delta exploration 0 0 2 1 2 1 1 1 0 0 1 1

33. 33 Back to our example 31 2 3 3 2 1 3 4 31 2 4 3 2 1 4 4 add(4) only one execution only one update on field size

34. 34 ∆Execution: Set Operations Merge: “merges” sets of states into one ∆State Split: “splits” a set of states into two subsets

35. 35 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 Combined Execution & Splitting add(4)

36. 36 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 Combined Execution & Splitting split splits on root.right == null add(4)

37. 37 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 Combined Execution & Splitting split splits on root.right == null 4 2 3 1 1 3 2 4 44 add(4)

38. 38 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 1 2 3 2 1 3 44 44 Combined Execution & Splitting split splits on root.right.right == null splits on root.right == null 4 2 3 1 1 3 2 4 44 add(4)

39. 39 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 1 2 3 2 1 3 44 44 Combined Execution & Splitting split splits on root.right == null 4 2 3 1 1 3 2 4 44 add(4) 3 1 2 4 4 splits on root.right.right == null

40. 40 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 1 2 3 2 1 3 44 44 Combined Execution & Splitting split splits on root.right == null 4 2 3 1 1 3 2 4 44 add(4) 3 1 2 4 4 splits on root.right.right == null reduction in number of executions: standard has 5, delta has 3

41. 41 3 1 2 4 4 4 2 3 1 1 3 2 4 441 2 3 2 1 3 44 44 Merging merge merges 4 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 4444 4 4 44 4

42. 42 Split + Merging Splits during execution Merges sets of post- states from execution … … SiSi … … … iteration i S i+1 iteration i + 1 …… …

43. 43 Efficient ∆Execution Representation of set of concrete states Program instrumentation Optimized state comparison

44. 44 ∆Execution: State representation Represents a set of concrete states 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 conceptual representation

45. 45 33112 Representing state: the  State 3 33211 3333 3 ∆Objects

46. 46 21??1 33112 Representing state: the  State 3 2 3 1 3 1 211 3333 3 ∆Objects for references

47. 47 21??1 33112 Representing state: the  State 3 2 3 1 3 1 2 3 1 3 1 2 3333 ??233 3

48. 48 ?? 21??1 33112 Representing state: the  State 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 ?????? ??233 1223 3

49. 49 ?? 2 1 ??1 3 3 112 Representing state: the  State 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 ???? ?? ??233 1 2 23 3

50. 50 Constants constants created during merging becomes constant after split due to root.right == null 33112????????23321??1?? 1223 3

51. 51 Optimized State Comparison State comparison in model checkers linearization + hashing use the hash for comparison in a hashtable

52. 52 Optimized State Comparison 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3

53. 53 Optimized State Comparison 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3 100 1 101 2 101 3 -1 101 4 -1 - 1 -1 2 1 3 3

54. 54 Optimized State Comparison 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3 100 1 101 2 101 3 -1 101 4 -1 - 1 -1 2 1 3 3 100 1 101 2 101 3 -1 -1 101 4 -1 -1 2 1 3 3

55. 55 Optimized State Comparison 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3 100 1 101 2 101 3 -1 101 4 -1 - 1 -1 2 1 3 3 100 1 101 2 101 3 -1 -1 101 4 -1 -1 2 1 3 3 100 1 101 2 -1 101 3 101 4 -1 -1 -1 2 1 3 3

56. 56 Optimized State Comparison 3 2 3 1 1 3 2 1 2 3 2 1 3 3 1 2 3333 100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3 100 1 101 2 101 3 -1 101 4 -1 - 1 -1 2 1 3 3 100 1 101 2 101 3 -1 -1 101 4 -1 -1 2 1 3 3 100 1 101 2 -1 101 3 101 4 -1 -1 -1 2 1 3 3 100 1 101 2 -1 101 3 -1 101 4 -1 -1 2 1 3 3 Observation: significant sharing of data!

57. 57 Program modification Primitives are replaced with  Objects that represent “sets” of primitive values Object references are replaced with generated  Objects that represent “sets” of references for a particular type Operations (+, -, <, ==, etc.) perform on  Objects (i.e., “sets” of values)

58. 58 Example Modifications public class BST { private DeltaNode root = DeltaNode.NULL; private DeltaInt size = DeltaInt.ZERO; public void add(DeltaInt info) { // original: if (root == null) if (root.eq(DeltaNode.NULL)) … else … } public DeltaBoolean remove(DeltaInt info) {... } } class Node { DeltaNode left, right; DeltaInt info; Node(DeltaInt info) { this.info = info; } }

59. 59 Example Modifications public class BST { private DeltaNode root = DeltaNode.NULL; private DeltaInt size = DeltaInt.ZERO; public void add(DeltaInt info) { // original: if (root == null) if (root.eq(DeltaNode.NULL)) … else … } public DeltaBoolean remove(DeltaInt info) {... } } class Node { DeltaNode left, right; DeltaInt info; Node(DeltaInt info) { this.info = info; } } reference equality on a ∆Object

60. 60 Example: Equality of Reference eq on the following objects OBJ-X returns true

61. 61 Example: Equality of Reference eq on the following objects OBJ-X returns true; “disable” the state 3 split returns false; “disable” all but state 3

62. 62 Summary of time reduction Execution of transitions Reduction of redundant operations Path exploration (backtracking) Reduction in number of executions Path pruning (state comparison) Simultaneous comparison of several states

63. 63 Evaluation Conducted experiment on 10 data structures and one larger case study, AODV Implemented in two model checkers: Java PathFinder BOX (Bounded Object eXplorer) Performed bounded-exhaustive exploration using breadth-first search

64. 64 Subjects Ten basic subject implementations BinHeap [Visser et al. 2006] BinarySearchTree [Boyapati et al. 2002, Xie et al. 2005] Deque FibHeap [Visser et al. 2006] FileSystem [Daisy File System, Qadeer 2004] HeapArray [Boyapati et al. 2002, Xie et. al. 2005] Queue [Darga & Boyapati 2006] Stack [Darga & Boyapati 2006] TreeMap [Java Collections 1.4] UBStack [Csallner & Smaragdakis 2004, Pacheco & Ernst 2005] One case-study, a network routing protocol Ad-hoc On-Demand Distance Vector (AODV) [Perkins & Royer 1999, Musuvathi et al. 2002, Sobeih et al. 2005]

65. 65 Results for JPF Implementation 931991189099118946.17x32.541502.24ubstack-9 5269778910354059.63x9.4390.80treemap-11 56109805613725714.43x4.1459.70stack-7 60118396014799516.63x5.0884.42queue-7 3598048090804809126.80x21.492724.63heaparray-9 156819483213535.59x3.0817.18filesystem-4 209490193154465918.57x21.59400.84xfibheap-8 8101122354062353019.14x28.84552.11deque-9 22688 863  Exec 4127900 4001328 Std # Executions 214.06 458.81 Standard Exploration Time (sec) 10.79xGMEAN 2063957.11x30.13bst-10 25008338.50x11.91binheap-8 Std /  Exec # StatesSubject- Bound 11 additional cases using other bounds (21 cases total)

66. 66 Results for JPF Implementation 11 additional cases using other bounds (21 cases total)

67. 67 Results for JPF Implementation 11 additional cases using other bounds (21 cases total)

68. 68 Results for JPF Implementation 11 additional cases using other bounds (21 cases total)

69. 69 Time breakdown for JPF fibheap-8: due to constants due to reduction in number of executions due to optimized linearization 21.60 400.84 total exploration selected times (sec) 4.07 256.79 execution 4.49 8.02 backtrack 1.41 136.03 state comparison 11.63 - merging  Execution Standard

70. 70 Results for data structures JPF exploration time improvement: 10.79x JPF peak memory usage reduction: 33% BOX exploration time improvement: 2.07x BOX peak memory usage increase: -3%

71. 71 AODV Case Study AODV: routing protocol for wireless networks written in the J-Sim network simulator 43 classes; over 3500 lines of code Three variations examined; each containing an error leading to a property violation routes should be cycle free Requires at least 8 transitions to show error Exploration time improved up to1.53x

72. 72 Outline Introduction & Goal ∆Execution Mixed execution (brief) Related Work and Conclusions

73. 73 Mixed Execution Goal: Speed up execution operation in model checkers with special state representation Representation of program state the model checker defines so to directly read and write data Impact model checking operations execution of transitions exploration of paths (backtracking) pruning of paths (state comparison) Examples: JPF [JV01], BogorVM [RDH03], SpecExplorer [VCST05]

74. 74 Mixed Execution Technique: Executes code on native state representation where possible Translates between special and native state representations Works on deterministic blocks only model checker does not track state changes but…concurrent code contains atomic blocks Implementation: Java PathFinder (JPF)

75. 75 Java PathFinder (JPF) Implements a backtrackable JVM for explicit- state model checking of Java bytecodes Implemented in Java Executes on top of a regular, host JVM JPF JVM Java program (classfiles)

76. 76 JPF Special State Representation Represents the heap as array of int. arrays Each object encoded with integer id Primitive and reference fields encoded as integers (distinguished by types) 2 … 96: [2, 97] 97: [1, -1, 98] 98: [2, -1, -1] … size root info left right ( == null ) tree node (2) node (1) JPF statenative state 1 2

77. 77 Example: Tree … 96: [2, 97] 97: [1, -1, 98] 98: [2, -1, -1] … JPF state start from a tree object with integers 1 and 2; invoke add(0) 2 1 2 native state

78. 78 Example: Tree … 96: [2, 97] 97: [1, -1, 98] 98: [2, -1, -1] … JPF state … 96: [3, 97] 97: [1, 99, 98] 98: [2, -1, -1] 99: [0, -1, -1] … add(0) 3 1 0 2 native state 2 1 2 JPF execution mutates two integer arrays and creates another

79. 79 Example: Tree … 96: [2, 97] 97: [1, -1, 98] 98: [2, -1, -1] … JPF state … 96: [3, 97] 97: [1, 99, 98] 98: [2, -1, -1] 99: [0, -1, -1] … add(0) execution on JPF state representation is slow special-state interpretation

80. 80 Example: Tree 2 … 96: [2, 97] 97: [1, -1, 98] 98: [2, -1, -1] … JPF statenative state 1 2 … 96: [3, 97] 97: [1, 99, 98] 98: [2, -1, -1] 99: [0, -1, -1] … 3 1 0 2 mixed execution add(0)

81. 81 Evaluation Conducted experiments in 7 data structures and one larger case study, AODV Bounded-Exhaustive Exploration Results Improved execution time up to 67.2% Improved overall exploration time up to 42.2% Speedup depends on bound, scales well: the larger the bound, the higher the speedup

82. 82 Outline Introduction & Goal ∆Execution Mixed execution Related Work and Conclusions

83. 83 Related Work Reduction in state space size abstraction Predicate Abstraction, Program Slicing path pruning Heap Symmetry Reduction, Thread Symmetry Reduction, Partial-Order Reduction Recent for OO: Abstract Matching [VPP06], Glass-Box Model Checking [DB06]

84. 84 Related Work Time reduction execution symbolic execution path pruning Incremental Hashing [MD05]

85. 85 Next Steps Alternative ways to execute, backtrack, compare state

86. 86 Conclusions Techniques improve overall exploration time ∆Execution, on average 10x Mixed Execution, up to 42% Techniques address costly operations execution (mixed and ∆) backtracking (∆) state comparison (∆)

87. 87 Publications on thesis work Testing Delta Execution for Efficient State-Space Exploration of Object-Oriented Programs. Marcelo d'Amorim, Steven Lauterburg and Darko Marinov. (ISSTA'07) London, UK, July, 2007. (Nominated to the list of best papers) Invited to the IEEE TSE journal, in preparation Delta Execution for Software Reliability. Yuanyuan Zhou, Darko Marinov, William Sanders, Craig Zilles, Marcelo d'Amorim, Steven Lauterburg, Ryan M. Lefever and Joe Tucek (HotDep'07) Edinburgh, UK, June, 2007. Optimized Execution of Deterministic Blocks in Java PathFinder. Marcelo d'Amorim, Ahmed Sobeih and Darko Marinov. (ICFEM'06) Macau, SAR, November, 2006. An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing. Marcelo d'Amorim, Carlos Pacheco, Darko Marinov, Tao Xie, and Michael D. Ernst. (ASE'06) Tokyo, Japan, September, 2006. (Nominated to the list of best papers) Invited to the ASE journal, submitted

88. 88 Publications on related work Runtime Verification Efficient Monitoring of Omega-Languages. Marcelo d'Amorim and Grigore Rosu. (CAV'05) Edinburgh, Scotland, July 2005 Checking and Correcting Behaviors of Java Programs at Runtime with Java-MOP. Feng Chen, Marcelo d'Amorim and Grigore Rosu. In Proc. of the 5th Workshop on Runtime Verification (RV'05) Edinburgh, Scotland, July, 2005. Event-Based Runtime Verification of Java Programs. Marcelo d'Amorim and Klaus Havelund. (WODA'05) St. Louis, U.S., May, 2005. A Formal Monitoring-based Framework for Software Development and Analysis. Feng Chen, Marcelo d'Amorim, and Grigore Rosu. (ICFEM'04) Seattle, U.S., November 2004. Other in SE and PL An Equational Specification for the Scheme Language. Marcelo d'Amorim and Grigore Rosu. In Proc. of the 9th Brazilian Symposium on Programming Languages (SBLP'05) Recife, Brazil, June, 2005. Integrating Code Generation and Refactoring. M. d'Amorim, C. Nogueira, G. Santos, A. Souza, and P. Borba. In Proc. of the Workshop on Generative Programming (ECOOP'02) Malaga, Spain, June 2002.

89. 89 Areas of interest Productivity in SE Testing & Runtime Verification (Ph.D. studies) Effectiveness Efficiency (Ph.D. dissertation) Refactoring and Program Generation (pre Ph.D.)