Random Test Generation of Unit Tests: Randoop Experience Wujie Zheng wjzheng@cse.cuhk.edu.hk

Outline Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements

Outline Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements

Automated Test Generation of Unit Tests Problem Given a method of a class, how to automatically generate test cases to execute most of its code, i.e., to achieve high code coverage.

Automated Test Generation of Unit Tests A simple example class under test: java.util.Stack method under test: public synchronized int search(Object o) { int i = lastIndexOf(o); if (i >= 0) { return size() - i; } return -1;

Automated Test Generation of Unit Tests What need to be generated? Test Inputs not only primitive values but also objects, including the receiver object Test Oracles Could use some basic contracts For not much test inputs, manually added by testers

Automated Test Generation of Unit Tests An example test case // a test case for Stack.search(Object) // existing (previous generated) sequences that create parameters Stack var0 = new Stack(); String var1 = "hi!"; var0.push((Object)var1); // the method under test int var2 = var0.search(var1); // test oracles assertTrue(var0.equals(var0)); assertTrue(var2==1); // Regression assertion

Automated Test Generation of Unit Tests Main solutions Random Approaches Randomly create and mutate objects For a method under test, select objects created in existing sequences as its parameters Simple, scalable, but may not efficient Systematic Approaches Explore the possible paths systematically, for each path, collect the path condition and solve it to generate test inputs Potential high coverage, but complex Systematic approaches Build a connection between inputs and program paths Rely on constraint solvers: good for primitive values, but may not easy for objects

Outline Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements

Randoop: Feedback-directed Random Test Generation The basic idea Undirected random test generation is not efficient because it does not prune redundant or error objects (generated by duplicate or different method sequences) Identify redundant or error objects early can prune large search space

Randoop: Feedback-directed Random Test Generation The basic idea Identify redundant objects Use the equals() method Identify error objects Exceptions. Exceptions frequently correspond to precondition violations for a method, and therefore there is little point extending them Null Null dereference exceptions caused by using null as an argument are often uninteresting Set the redundant or error objects as not extensible

Randoop: Feedback-directed Random Test Generation Sequence generation for a method To execute a method m(T1,…,Tk), we need to find existing sequences that generate the parameters For each Ti Primitive type, randomly select a value from a pool Reference type, select a extensible (non-redundant and legal) value v from existing sequences, or use null Generate a new sequence, including the method m and the required sequences to generate the parameters

Randoop: Feedback-directed Random Test Generation The Randoop algorithm Repeat until timeout Randomly select a method m(T1,…,Tk) Generate a new sequence for m Execute the new sequence Check simple oracles, set the extensible flags of the objects Output the generated sequences as unit tests

Outline Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements

Randoop Experience Subject programs Interested Metrics Java.util Berkeley DB Interested Metrics Number of tests Code coverage Error detection

Randoop Experience Experiments on Java.util Demo 7 classes in Java.util, used in other researchers’ experiments Run Randoop 60 seconds 4834 tests 50.8% instruction coverage 195 failed tests, many false positives! Demo

Randoop Experience Experiments on Berkeley DB Demo Java edition of Berkeley DB, com.sleepycat.je Run Randoop 60 seconds Around 20,000 tests 30% instruction coverage 300 failed tests, many false positives! Need to prepare some environments, e.g., files in the disk Can more tests achieve higher coverage? Run Randoop 120 seconds: out of memory Demo

Outline Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements

Potential Improvements The problems of Randoop Not enough code coverage Difficult to generate interesting sequences quickly before running out of memory Too large test suites Need to perform test selection (reduction)

Thank you!