JMLAutoTest and Its Double- phase Testing Way Guoqing Xu Com Sci., East China Normal Univ. Shanghai , PRC FATES 2003, Montreal, Canada, Oct.6th 2003

Introduction to the current JML testing framework The main idea in the JML and JUnit framework: specification as test oracle. It can generate codes from formal specifications for unit test by translating pre- and post-conditions of a method into test oracles. It has solved the problem that it is hard to write unit test code by hand.

Problems Writing codes for generating test cases, especially those with a linked data structure is still a labor-intensive job. Meaningless test cases can not be avoided in the test, thus much time might be used to explore meaningless test inputs.

Problems (more) Some formal approach based testing frameworks require programmer to write special predicates to check the validity of test cases (i.e. Korat). However they can not be used in a black-box test since testers may know little about the details in the program to be tested.

Goals of our new framework Generate test cases automatically including those with a linked data structure. Prevent meaningless test cases from being processed in the final test instead of only identifying them.

The test case generation Set Value Domain for Fields of the Input class. Make a finitization of all cases by combining each value in the domain specified by testers. Verify the validity of a test case by checking the run-time state of instance invariant specified in its class.

Double-phase testing Goals: 1. Prevent meaningless test cases being processed in the final test, therefore saving the time. 2. Do not require testers to know the details of the program to be tested, thus making the black-box test possible.

Key steps in double-phase testing

Operational Profile Operational Profile is a standard made by the tester to divide the generated test case space into several partitions. the validity of double-phase testing is based on the quality of the operational profile. it is a good idea to start out with several models and evaluate their predictive accuracy before settling on one.

The first phase Take a relatively small number (e.g. 10%) of test cases out from each partition according to the average distribution. Run these groups of cases respectively. Make statistics on the number of meaningless test cases appeared in each group.

The second phase Calculate and determine the probability of meaningless test cases existing in each partition. Reorganize the test cases according to the inverse proportion of meaningless cases in each partition. (e.g. take 80% of cases from the partition which produces 20% of meaningless ones in the first phase.) Run the final test.

Experimental results Target Method to be tested: public requires parentTree!=null && thisNode!=null (\exists Node n; n.ID== assignable ensures \result.root.ID == thisNode.ID BinaryTree findSubTree(BinaryTree parentTree, Node thisNode)

Test case generation The pre-condition of the method says that neither of its arguments can be null and there must be a node in parentTree whose ID equals the ID of thisNode. We generate the test case space of type BinaryTree with a few nodes. We also generate the case space of type Node which contains 12 nodes whose IDs are from 0 to 11.

Divide the test case space For the test case space of type BinaryTree, we do not divide it and leave it as the only partition. For the space of type Node, We divide it into two partitions. The first one contains nodes whose ID varies from 0 to 5 and the second one contains the rest.

Test Results Table 1. Performance of JMLAutoTest for testing the method findSubTree with arguments “-pre 0.25”. nodes in binary tree generated binary trees candidates considered meaningful test cases in the final test total test cases in the final test

Test Results Table 2. Performance comparison between the double-phase testing in JMLAutoTest and the conventional way in JMLUnit testing framework. nodes in binary tree meaningful /total in final test time in the first phase (s) total time of the test (s) meaningful /total in final test time in the first phase (s) total time of the test (s) 5410/ / / / / / / / Double-phase way Conventional way

Conclusions JMLAutoTest provides interfaces for testers to automatically generate test cases, even those with a linked data structure. a) Set value domain for fields b) Finitization c) Verify validity of a test case by checking the instance invariant specified in its class

Conclusions (more) Based on statistical principle, JMLAutoTest uses the double-phase testing way to filter out meaningless test cases before the final test is run. a) Make an Operational Profile b) Determine the proportion ofmeaningless test cases of each partition in the first phase. c) Reorganize the test cases according the distribution if meaningless cases and run the final test

Acknowledgement We would like to thank Professor Gary T. Leavens in Iowa State University who gave us help on both JML and the testing framework.