Tao Xie North Carolina State University

2 Why benchmarks in testing research? Before 2000, a testing tool/technique paper without serious evaluation could appear in ICSE/FSE/ISSTA/ASE but NOT now! – a healthy trend Need benchmarks to justify the benefits of the proposed technique (often comparing with existing techniques)

3 Outline Survey of incomplete OO testing tool literature after 2000 for benchmarks Discussion on history Discussion on future

4 TestEra [Marinov and Khurshid ASE 2001] Data structures singly linked lists.mergeSort java.util.TreeMap.remove INS 3 methods Alloy-alpha 1 method Require Alloy models for class invariants Success criterion: # n-bounded exhaustive tests

5 Korat [Boyapati et al. ISSTA 2002] Data structures korat.examples.BinaryTree.remove korat.examples.HeapArray.extractMax java.util.LinkedList.reverse java.util.TreeMap.put java.util.HashSet.add ins.namespace.AVTree.lookup Require repOk for class invariants, finitization Success criterion: # n-bounded exhaustive tests

6 JCrasher [ Csallner and Smaragdakis S&P 2004]  Canvas 6 methods  P1 16 methods  P1 15 methods  P1 16 methods  P1 18 methods  P1 15 methods  BSTree 24 methods  UB-Stack 11 methods Success criterion: #real bugs among uncaught exceptions

7 Check ’n’ Crash [ Csallner and Smaragdakis ICSE 2005]  Canvas 6 methods  P1 16 methods  P1 15 methods  P1 16 methods  P1 18 methods  P1 15 methods  BSTree 24 methods  UB-Stack 11 methods  jaba 17.9 k LOC  jboss.jms 5.1 k LOC Success criterion: #real bugs among uncaught exceptions

8 DSD-Crasher [ Csallner and Smaragdakis ISSTA 2006]  jboss.jms 5K LOC  groovy 34 classes 2K LOC Success criterion: #real bugs among uncaught exceptions

9 eToc [ Tonella ISSTA 2004] Data structures  StringTokenizer 6 methods  BitSet 26 methods  HashMap 13 methods  LinkedList 23 methods  Stack 5 methods  TreeSet 15 methods Require a configuration file (so far manually prepared) Success criterion: branch coverage and % found seeded faults (5 seeded faults each class)

10 JPF [Visser et al. ISSTA 2004] Data structure  java.util.TreeMap 3 methods  deleteEntry  fixAfterDeletion  fixAfterInsertion Require a driver to close the environment Success criterion: branch coverage

11 JPF [Visser et al. ASE 2005] Data structure  java.util.TreeMap Require a driver to close the environment Success criterion: basic block coverage

12 JPF [Visser et al. ISSTA 2006] Data structures  BinTree 154 LOC  BinomialHeap 355 LOC  FibHeap 286 LOC  Partial java.util.TreeMap 580 LOC Require a driver to close the environment Success criterion: basic block coverage &predicate coverage

13 Rostra [Xie et al. ASE 2004]  Data structures  IntStack 4 methods  UBStack 10 methods  BSet 9 methods  BBag 8 methods  ShoppingCart 7 methods  BankAccount 6 methods  BinarySearchTree 10 methods  LinkedList 10 methods Require existing tests to provide method arguments Success criterion: branch coverage

14 Symstra [Xie et al. TACAS 2005]  Data structures  IntStack push,pop  UBStack push,pop  BinSearchTree insert,remove  BinomialHeap insert,extractMin, delete  LinkedList add,remove, removeLast  TreeMap put,remove  HeapArray insert,extractMax Require a driver to close the environment Success criterion: branch coverage

15 Symclat [d'Amorim et al. ASE 2006]  Data structures  UBStack8 11m, UBStack12 11m, UtilMDE 69m, BinarySearchTree 9m, StackAr 8m, StackLi 9m, IntegerSetAsHashSet 4m, Meter 3m, DLList 12m, OneWayList 10m, SLList 11m, OneWayList 12m, OneWayNode 10m, SLList 12m, TwoWayList 9m, RatPoly (46 versions) 17 Require initial tests Success criterion: #real bugs

16 Evacon [Inkumsah and Xie ASE 2008]  Data structures  BankAccount 6m, BinarySearchTree 16m, BinomialHeap 10m, BitSet 25m, DisjSet 6m, FibonacciHeapm, HashMap 10m, LinkedList 29m, ShoppingCart 6m, Stack 5m, StringTokenizer 5m, TreeMap 47m, TreeSet 13m Require a configuration file (so far manually prepared) Success criterion: branch coverage

17 Nighthawk [Andrews et al. ASE 2007] Data structures java.util.BitSet 16 methods java.util.HashMap 8 methods java.util.TreeMap9 methods BinTree, BHeap, FibHeap, TreeMap Java Collection and Map classes: ArrayList, EnumMap, HashMap, HashSet, Hashtable, LinkedList, Pqueue, Propeties, Stack, TreeMap, TreeSet, Vector Success criterion: block, line, condition coverage

18 Random Test Run Length and Effectiveness [ Andrews et al. ASE 2008] Other system test subjects Real buffer overflow bugs Data structures JUnit MoneyBag TreeMap Success criterion: a real bug in MoneyBag, a seeded bug in TreeMap

19 Delta Execution [d’Amorim et al. ISSTA 2007] 9 data structures with manually written drivers binheap, bst, deque, fibheap, heaparray, queue, stack, treemap, ubstack 4 classes in filesystem (the Daisy code) Require a driver to close the environment Success criterion: # n-bounded exhaustive tests

20 Incremental State-Space Exploration [Lauterburg et al. ICSE 2008] 9 data structures with manually written drivers binheap, bst, deque, fibheap, heaparray, queue, stack, treemap, ubstack 4 classes in filesystem (the Daisy code) Aodv: routing protocol for wireless ad hoc networks Require a driver to close the environment Success criterion: % time reduction over versions

21 ARTOO [Ciupa et al. ISSTA 2007] Eiffel Data structures  STRING 175 methods  PRIMES 75 methods  BOUNDED STACK 66 methods  HASH TABLE 135 methods  FRACTION144 methods  FRACTION2 45 methods  UTILS 32 methods  BANK ACCOUNT 35 methods Success criterion: #real bugs

22 ARTOO [Ciupa et al. ICSE 2008 ] Eiffel Data structures  ACTION SEQUENCE 156 methods  ARRAY 86 methods  ARRAYED LIST 39 methods  BOUNDED STACK 62 methods  FIXED TREE 125 methods  HASH TABLE 122 methods  LINKED LIST 106 methods  STRING 171 methods Success criterion: #real bugs

23 Randoop [ Pacheco et al. ICSE 2007 ] Data structures  BinTree, Bheap, FibHeap, TreeMap, BinTree, BHeap, FibHeap, TreeMap Java JDK 1.5  java.util 39KLOC 204 classes 1019 methods  javax.xml 14KLOC 68 classes 437 methods Jakarta Commons  chain 8KLOC 59 classes 226 methods  collections 61KLOC 402 classes 2412 methods  See next slide Success criterion: #real bugs

24 Randoop [ Pacheco et al. ICSE 2007 ] – cont. Jakarta Commons  …  jelly 14K 99c 724m, logging 4K 9c 1 40m, math 21K 111c 910m, primitives 6K 294c 1908m.NET libraries  ZedGraph 33K 125c 3096m .NET Framework: Mscorlib 185K 1439c 17763m, System.Data 196K 648c 11529m, System.Security 9K 128c 1175m, System.Xml 150K 686c 9914m, Web.Services 42K 304c 2527m

25 Randoop [ Pacheco et al. ISSTA 2008]  A core component of the.NET Framework  > 100KLOC Success criterion: #real bugs

26 Pex [ Tillmann and HalleuxTAP 2008]  A core component of the.NET Framework  > 10,000 public methods  Selected results being presented  9 classes (>100 blocks ~ >500 blocks) Success criterion: block, arc coverage, #real bugs

27 MSeqGen [Thummalapenta et al. ESEC/FSE2009]  QuickGraph  165 classes and interfaces with 5 KLOC  Facebook  285 classes and interfaces with 40 KLOC Success criterion: branch coverage

28 Dynamic Symbolic Execution Tools  How about DART, CUTE/jCUTE, Crest, EXE, EGT, KLEE, SAGE, Smart, Splat, Pex...  Non-OO vs. OO

29 Summary of Benchmarks  (Mostly) data-structure (DS) classes only  TestEra [ASE01], Korat [ISSTA02], JCrasher [S&E04], eToc [ISSTA04], JPF [ISSTA04, ASE05, ISSTA06], Rostra [ASE04], Symstra [TACAS05], Symclat [ASE06], Evacon [ASE08], Nighthawk [ASE07, ASE08], UI JPF extensions [ISSTA07, ICSE08], ARTOO [ISSTA07, ICSE08]  Non-DS classes  Check ’n’ Crash [ICSE05], DSD-Crasher[ISSTA06]  Randoop [ICSE07, ISSTA08]  Pex [TAP08]  MSeqGen [ESEC/FSE 09]

30 Open Questions – history  Why/how do authors select their benchmarks used in their evaluation? Or Why not select other benchmarks? (Your answers here!)  Reason 1:  Reason 2:  …

31 Open Questions – history (cont.)  Are data structures mandatory?  Like Siemens programs in fault localization as sanity check?  Are data structures sufficient?  How much would the results generalize to broader types of real-world applications?  How about libraries (in contrast to applications)?  High payoff in terms of testing efforts? More logics/challenging?

32 Open Questions – future  Shall we have categorized benchmarks? What categories (general, DS, GUI, DB, Web, network, embedded, string- intensive, pointer-intensive, state-dependent-intensive, ML, …,)?  What criteria shall we use to include/exclude benchmarks?  Where to contribute and share? (UNL SIR, a new repository?)  How to provide cross-language benchmarks?  How about test oracles (if we care more than coverage)?  How about evaluation criteria (structural coverage, seeded faults, uncaught exceptions, …)?

33 Open Questions – cont.  Caveats in benchmarking  Tools can be tuned to work best on well-accepted benchmarks but don’t work/generalize on other applications  …