1. 50.530: Software Engineering
Sun Jun
SUTD

2. Week 2: Automatic Testing

3. A Big View: Testing the initial state C A B the behaviors we wanted
the behaviors we have

4. A Big View: Testing a test which shows a bug the initial state C A
the behaviors we wanted
the behaviors we have

5. Testing
Methods: white-box testing, black-box testing, grey-box testing
Levels: unit testing, integration testing, system testing, etc.
Types: installation testing, compatibility testing, smoke and sanity testing, regression testing, acceptance testing, alpha testing, beta testing, function/non-functional testing, combinatorial testing, performance testing, security testing, etc.

6. Research Question Isn’t jUnit good enough?
How do we automatically generate test cases so as to reveal bugs?

7. A Big View: Systematic Testing
the initial state C A B
the behaviors we wanted
the behaviors we have

8. A Big View: Random Testing
a test which shows a bug
the initial state C A
the behaviors we wanted
the behaviors we have

9. Korat: Automated Testing Based on Java Predicates
Boyapati et al., ISSTA 2002, ACM SIGSOFT Distinguished Paper Award
Korat: Automated Testing Based on Java Predicates

10. Motivation
It is important to be able to generate test cases automatically.
It is important to generate test cases which are representative.
Korat is merely a sample approach for systematic test case generation, however, it is similar in spirit to many systematic testing techniques (e.g., combinatorial testing, parameterized testing).

11. Example public class BinaryTree { public static class Node {
Node left; Node right; }
private Node root;
private int size;
public void remove (Node n) {
//some code …
How do we test remove(node n)?

12. Example How do we test remove(Node n)?
We need a valid BinaryTree object bt.
We need a valid Node object nd.
We need to know what is expected after executing bt.remove(nd)
public class BinaryTree {
public static class Node {
Node left; Node right; }
private Node root;
private int size;
public void remove (Node n) {
//some code …

13. Vocabulary Class invariant:
an invariant used to define what are valid objects of the class
e.g., size == 0 if root == null and size equals to the number of nodes in the tree
public class BinaryTree {
public static class Node {
Node left; Node right; }
private Node root;
private int size;
public void remove (Node n) {
//some code …

14. Vocabulary Pre-condition (of a method)
a condition which must be true prior to the execution of the method
e.g., n must not be null.
The class invariant is always part of the pre-condition.
public class BinaryTree {
public static class Node {
Node left; Node right; }
private Node root;
private int size;
public void remove (Node n) {
//some code …

15. Vocabulary Post-condition (of a method)
a condition which must be true after the execution of the method
e.g., after remove, size is decremented by 1.
The class invariant is always part of the post-condition.
public class BinaryTree {
public static class Node {
Node left; Node right; }
private Node root;
private int size;
public void remove (Node n) {
//some code …

16. Karat: Assumption
A class invariant is encoded as a method repOk(), which return true if and only if the object is in a state which satisfies the class invariant.
public boolean repOK() {
if (root == null)
return size == 0;
Set<Node> visited = new HashSet<Node>();
visited.add(root);
LinkedList<Node> workList = new LinkedList<Node>();
workList.add(root);
while (!workList.isEmpty()) {
Node current = (Node) workList.removeFirst();
if (current.left != null) {
if (!visited.add(current.left))
return false;
workList.add(current.left); }
if (current.right != null) {
if (!visited.add(current.right))
workList.add(current.right);
return (visited.size() == size);

17. Korat: Assumption
Pre-condition and post-condition are encoded in Java Modeling Language
public invariant repOk(); // class invariant
// for BinaryTree
public normal_behavior // specification for remove
@ requires has(n); // precondition
@ ensures !has(n); // postcondition
@*/
public void remove(Node n) {
// ... method body }
This is probably too harsh a pre-condition?

18. Generate a BinaryTree bt and a Node n
Karat: Approach
Generate a BinaryTree bt and a Node n
if repOk() and pre-condition is true
otherwise
Execute bt.remove(n)
if post-condition is true
otherwise

19. Finitization There are infinitely many candidates for bt and n.
For each variable
in the class, define its domain
all possible bt
interesting bt

20. Finitization public static Finitization finBinaryTree(int NUM_Node) {
Finitization f = new Finitization (BinaryTree.class);
ObjSet nodes = f.createObjSet(“Node”, NUM_Node);
nodes.add(null);
f.set("root", nodes);
f.set("Node.left", nodes);
f.set("Node.right", nodes);
return f; }
public class BinaryTree {
public static class Node {
Node left; Node right; }
private Node root;
private int size; …

21. Finitization public static Finitization finBinaryTree(int NUM_Node) {
Finitization f = new Finitization (BinaryTree.class);
ObjSet nodes = f.createObjSet(“Node”, NUM_Node);
nodes.add(null);
f.set("root", nodes);
f.set("Node.left", nodes);
f.set("Node.right", nodes);
return f; }
translation
nodes = {null, N0, N1, N2}
BinaryTree.root is a member of nodes
Node.left is a member of nodes
Node.right is a member of nodes

22. Example Trees
With finBinaryTree(3), there are 4 objects: one BinaryTree object, three Node objects, which could be set up as follows.

23. Finitization: the Space
How many bt are there with finBinaryTree(3), assume that bt.size is always set to the right value?
4^7
How many bt are there with finBinaryTree(n)?
(n+1)^(2n+1)
all possible bt
interesting bt

24. Filtering 1
For each candidate bt and n, check the pre-condition of remove. If the pre-condition is not satisfied, ignore that tree.
all possible bt
interesting bt
invalid bt

25. Is the following bt valid?
public boolean repOK() {
if (root == null)
return size == 0;
Set<Node> visited = new HashSet<Node>();
visited.add(root);
LinkedList<Node> workList = new LinkedList<Node>();
workList.add(root);
while (!workList.isEmpty()) {
Node current = (Node) workList.removeFirst();
if (current.left != null) {
if (!visited.add(current.left))
return false;
workList.add(current.left); }
if (current.right != null) {
if (!visited.add(current.right))
workList.add(current.right);
return (visited.size() == size);
Is the following bt valid?

26. Korat: Search Algorithm
Order all the elements in every class domain and every field domain
Node class ordering: <null, N0, N1, N2>
Assume domain of size: <3>
Generate a candidate as a vector of field domain indices, e.g., [1,0,2,2,0,0,0,0]

27. Korat: Search Algorithm
Invoke repOk() to check if the candidate is valid, e.g., [1,0,2,2,0,0,0,0] is invalid
Backtrack to generate the next candidate in line, e.g., [1,0,2,2,0,0,0,1]

28. Optimization 1
During the execution of repOk, Korat monitors the fields that repOk accesses.
e.g., [0, 2, 3] for the following example
If repOk() results in false, backtrack until the accessed fields are different
e.g., try [1,0,2,3,0,0,0,0] after [1,0,2,2,0,0,0,0]
Is this justified?

29. Theory For non-deterministic repOk methods,
All candidates for which repOk() always returns true are generated
Candidates for which repOk() always returns false are never generated;
Candidates for which repOk() sometimes returns true and sometimes false may or may not be generated.

30. Optimization 2
If we generated the above, we may not want to generate [1, 0, 3, 2, 0, 0, 0, 0].
Is this justified? N2 N1

31. Vocabulary object graph:
Isomorphic: two object graphs C and C’ are isomorphic iff there is a permutation per such that per(C) = C’ and per(C’) = C
e.g., per = {N1->N2, N2->N1} N2 N1

32. Optimization 2 interesting bt representative
all candidates in the same region are isomorphic

33. Representative Given the two graphs below
[1, 0, 2, 3, 0, 0, 0, 0] and [1, 0, 3, 2, 0, 0, 0, 0], Korat takes the latter as a representative, as it is “bigger”. N2 N1

34. Implementation: Op 2 When backtracking from [a, b, c, …,k, …],
Korat tries [a, b, c, …,k+1, …] if k+1 is smaller than or equal to any number in the vector which has the same associated type.
Korat tries [a, b, c, …, j+1, …] otherwise.

35. Example When backtrack from [1, 0, 2, 2, 0, 0, 0, 0],
Korat skips [1, 0, 2, 3, 0, 0, 0, 0] (since there is a “bigger” representative [1,0,3,2,0,0,0,0]), and continues with [1,0,3,0,0,0,0,0]

36. Result
Only 5 bt are generated – assuming size is set to 3 always.

37. Evaluation
Is this biased?

38. Experiment I

39. Experiment II

40. Experiment III

41. Conclusion
Korat generates test cases from a specified domain and correctness specification.
Korat reduces test cases based on
pre-condition
a simple learning
symmetry reduction

42. Exercise 1
Apply Korat to java.util.Stack by answering the following questions.
What is the repOk()?
What is the pre-condition and post-condition of method push and pop?
How would you track which fields are accessed in repOk()?
When are two stack objects isomorphic?

43. Discussion
Any thought on Korat?

44. Feedback-directed Random Test Generation
Pacheco et. al. ICSE 2007, cited 440+
Feedback-directed Random Test Generation

45. Random Testing Easy to implement Yields lots of test cases
Finds errors
1990: Unix utilities
1998: OS services
2000: GUI applications
2000: functional programs
2005: object-oriented programs
2007: flash memory, file systems
Perhaps simply got lucky?

46. Research Question
Which one is better: systematic testing or random testing?

47. Random vs Systematic
Theoretical work suggests that random testing is as effective as more systematic input generation techniques
Duran et al and Hamlet et al. 1990
Some empirical studies suggest systematic is more effective than random
Ferguson et al. 1996: vs. chaining
Marinov et al. 2003: vs. bounded exhaustive
Visser et al. 2006: vs. model checking and symbolic execution
small benchmarks; no measurement on error revealing effectiveness

48. Contributions Propose feedback-directed random test generation
Randomized creation of new test inputs is guided by feedback about the execution of previous inputs
Goal is to avoid redundant and illegal inputs
Empirical evaluation
Evaluate coverage and error-detection ability on a large number of widely-used, well-tested libraries (780KLOC)
Compare against systematic input generation
Compare against undirected random input generation

49. Sample Test Case public static void test1 () {
LinkedList l1 = new LinkedList();
Object o1 = new Object();
l1.addFirst(o1);
TreeSet t1 = new TreeSet(l1);
Set s1 = Collections.unmodifiableSet(t1);
Assert.assertTrue(s1.equals(s1)); }

50. Randoop Input: a class with multiple public methods.
Output: a set of test cases (sequences of method calls)
Main idea:
Build test inputs incrementally: New test inputs extend previous ones
As soon as a test input is created, execute it
Use execution results to guide generation

51. The Oracle Problem
If we are to do automatic testing, we must know what are the correct results, but how?

52. Specification
How to get a better specification in general?

53. Algorithm

54. “There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.”
1980, C.A.R.Hoare

55. assertTrue(s.equals(s));
Randoop Example
Date s = new Date(2006, 2, 14);
Assert specification
assertTrue(s.equals(s));
How do we randomly generate construct parameter values like 2006, 2, 14?

56. assertTrue(s.equals(s));
Randoop Example
HashSet s = new HashSet();
Randomly pick a public method
s.add(“”);
Assert specification
assertTrue(s.equals(s));
The default value for String “” is used since there is no other String in the system.

57. assertTrue(s.equals(s));
Randoop Example
HashSet s = new HashSet();
Randomly pick a public method
s.add(“”);
Randomly pick a public method
s.isEmpty();
Assert specification
assertTrue(s.equals(s));
A method is probably an observer method if it has no parameters; it is public and non-static; it returns primitive values; and its name is size, count, length, toString, or begins with get or is.

58. Randoop Example Date d = new Date(2006, 2, 14);
Randomly pick a public method
d.setMonth(-1); // pre: argument >= 0
A sequence of method calls result in an exception is added to errSeqs.

59. assertTrue(s.equals(s));
Randoop Example
Date d = new Date(2006, 2, 14);
Randomly pick a public method
d.setMonth(-1); // pre: argument >= 0
d.setDay(5);
Assert specification
assertTrue(s.equals(s));

60. Classifying a sequence
contract
violated?
execute and
check
contracts
minimize
sequence
yes
start no
components
sequence
redundant? no
contract-
violating
test case
yes
discard
sequence

61. Redundancy Checking Randoop maintains a set of objects for each type.
A sequence (of method calls) is redundant if the objects created during its execution are members of the above set.
Use equals() to compare
Or user-defined more sophisticated checking

62. Some Randoop options Avoid use of null Biased random selection
Favor smaller sequences
Favor methods that have been less covered
Use constants mined from source code
statically…
…and dynamically
Object o = new Object();
LinkedList l = new LinkedList();
l.add(null);
Object o = returnNull();
LinkedList l = new LinkedList();
l.add(o);

63. Research Question How effective would Randoop be?
How do we judge whether one set of random test cases are better than another set?

64. Coverage
Code block coverage: a set of random test cases are better if it covers more code blocks.
For instance, consider each branch as a block
Predicate coverage: given a set of predicates, a set of random test cases are better if it covers more valuations of the predicates.
For instance, consider the predicates to be the propositions in the program.

65. Coverage Achieved by Randoop
data structure
time (s)
branch cov.
Bounded stack (30 LOC) 1
100%
Unbounded stack (59 LOC)
BS Tree (91 LOC)
96%
Binomial heap (309 LOC)
84%
Linked list (253 LOC)
Tree map (370 LOC)
81%
Heap array (71 LOC)
Is this representative?

66. Predicate Coverage
feedback-directed
best systematic
feedback-directed
best systematic
undirected random
undirected random
On binary tree and fibonacci heap, randoop achieves higher coverage than both systematic and undirected generation. On binomial heap and tree map, randoop achieves the same coverage as systematic generation, but does so faster.
feedback-directed
feedback-directed
best systematic
best systematic
undirected random
undirected random

67. Bug Detection LOC Classes JDK (2 libraries) 53K 272
(java.util, javax.xml)
53K
272
Apache commons (5 libraries)
(logging, primitives, chain jelly, math, collections)
114K
974
.Net framework (5 libraries)
582K
3330 A C
How would Korat perform on these examples?

68. Methodology Ran Randoop on each library Contracts:
Used default time limit (2 minutes)
Contracts:
o.equals(o)==true
o.equals(o) throws no exception
o.hashCode() throws no exception
o.toString() throw no exception
No null inputs and:
Java: No NullPointerEexceptions
.NET: No NPEs, out-of-bounds, of illegal state exceptions

69. Results JDK 32 29 8 Apache commons 187 6 .Net framework 192 Total 411
test cases output
error-revealing tests cases
distinct errors
JDK 32 29 8
Apache commons
187 6
.Net framework
192
Total
411
250
206

70. Errors found: examples
JDK Collections classes have 4 methods that create objects violating o.equals(o) contract
Javax.xml creates objects that cause hashCode and toString to crash, even though objects are well-formed XML constructs
Apache libraries have constructors that leave fields unset, leading to NPE on calls of equals, hashCode and toString (this only counts as one bug)
Many Apache classes require a call of an init() method before object is legal—led to many false positives
.Net framework has at least 175 methods that throw an exception forbidden by the library specification (NPE, out-of-bounds, of illegal state exception)
.Net framework has 8 methods that violate o.equals(o)
.Net framework loops forever on a legal but unexpected input

71. Regression testing Randoop can create regression oracles
Generated test cases using JDK 1.5
Randoop generated 41K regression test cases
Ran resulting test cases on
JDK 1.6 Beta
25 test cases failed
Sun’s implementation of the JDK
73 test cases failed
Failing test cases pointed to 12 distinct errors
These errors were not found by the extensive compliance test suite that Sun provides to JDK developers
Object o = new Object();
LinkedList l = new LinkedList();
l.addFirst(o);
l.add(o);
assertEquals(2, l.size()); // expected to pass
assertEquals(false, l.isEmpty()); // expected to pass

72. Evaluation: summary Feedback-directed random test generation:
Is effective at finding errors
Discovered several errors in real code (e.g. JDK, .NET framework core libraries)
Can outperform systematic input generation
On previous benchmarks and metrics (coverage), and
On a new, larger corpus of subjects, measuring error detection
Can outperform undirected random test generation

73. Conclusion Feedback-directed random test generation Randoop:
Finds errors in widely-used, well-tested libraries
Can outperform systematic test generation
Can outperform undirected test generation
Randoop:
Easy to use—just point at a set of classes
Has real clients: used by product groups at Microsoft
A mid-point in the systematic-random space of input generation techniques

74. Exercise 2 Apply Randoop, manually, to OrderSet.java
To create 2 valid tests, one redundant test and one illegal sequence.
Create a test case to expose the bug.

75. How do we improve Korat or Randoop?
Research Question
How do we improve Korat or Randoop?