Specification-Based Unit Test Data Selection: Integrating Daikon and Jtest Tao Xie and David Notkin Computer Science & Engineering, University of Washington Problems Which test inputs are more “valuable” ? Developers manually write unit tests, which are usually insufficient to achieve satisfactory quality assurance of the unit. (Test insufficiency problem) Unit test generation tools automatically produce a large number of test inputs to exercise the unit. But how to know these automatically generated tests produce correct results? (Test oracle problem) How do we select “valuable” test inputs from automatically generated ones, equip them with oracles, and augment manually created unit tests? (Test selection problem) Those new test inputs that exercise at least one new feature not being exercised by any test case in the existing test suite. Black-box features: functionalities, predicates/states in specifications, etc. White-box features: methods, statements, branches, def-use pairs, etc. In traditional black box testing, , either formal or informal specifications are usually needed to help identify features, but in practice programs lack of specifications most of the time. A specification-based approach without a priori specification Infer likely specifications from existing test executions They reflect both the program behavior and the test history Any new tests that violate the inferred specifications are used as the candidates for test data selections – These tests possibly exercise new program behavior Jtest (http://www.parasoft.com/) Daikon (http://pag.lcs.mit.edu/daikon/) A popular commercial unit test tool that can produce a large number of test inputs for a Java class An invariant detection tool that can infer likely specifications from program executions Bounded Stack Example: Manual basic test # Manual JAX test # 8 (67%) 16 (97%) Auto generated test # (length 1) Auto generated test # (length 2) Auto generated test # (length 3) 14 (63%) 96 (86%) 1745 (86%) The number of selected tests, violating tests, and violated specifications Precondition Guard Removal Iteration. Type Test call length 1 Test call length 2 Test call length 3 1. Basic Full 0/1: 2 0/3: 3 0/0: 0 1. Basic No Pre 5/7: 15 8/13: 15 1/2: 2 1. JAX Full 1/3: 3 1/1: 1 1. JAX No Pre 1/3: 6 10/24: 41 2. Basic Full 2. Basic No Pre 0/1: 1 2. JAX Full 2. JAX No Pre Iterations reaching fixed point This work was supported in part by the National Science Foundation under grant ITR 0086003.