Systematic Software Testing Using Test Abstractions Darko Marinov RIO Summer School Rio Cuarto, Argentina February 2011

Examples of Structurally Complex Data T1 A1 T2 A2 XML document web sites s1 s2 s4 s3 s red-black tree Java program class A { int f; } class B extends A { void m() { super.f = 0; }

Running Example: Inheritance Graphs Java program interface I { void m(); } interface J { void m(); } class A implements I { void m() {} } class B extends A implements I,J { void m() {} }. IJ A B Inheritance graph Properties 1. DAG 2. ValidJava

Testing Setup Examples of code under test  Compiler or refactoring engines, input: programs  Abstract data type, input/output: data structure  Web traversal code, input: web topology  XML processing code, input/output: XML document code test generation test oracle pass fail inputsoutputs

Test Generation tool code inputs Fully automaticManual inputs tool Semi automated input set description inputs

Test Abstractions Each test abstraction describes a set of (structurally complex) test inputs  Example: inheritance graphs with up to N nodes Workflow  Tester manually writes test abstractions  Tool automatically generates test inputs Benefits  No need to manually write large number of test inputs  Useful for test generation, maintenance, and reuse

Key Questions for Test Abstractions tool test abstraction inputs 1. Which test inputs to generate? 2. What language to use for test abstractions? 3. How to generate test inputs from abstractions? 4. How to check test outputs? 5. How to map failures to faults?

Which Inputs to Generate? Bounded-exhaustive testing  Generate all inputs within given (small) bounds  Rationale: small-scope hypothesis [Jackson & Damon ISSTA’96] Found bugs in both academia and industry  Java compilers, model checkers… [Gligoric et al. ICSE’10]  Refactoring engines in Eclipse/NetBeans [Daniel et al. FSE’07]  Web traversal code from Google [Misailovic et al. FSE’07]  XPath compiler at Microsoft [Stobie ENTCS’05]  Fault-tree analyzer for NASA [Sullivan et al. ISSTA’04]  Constraint solver, network protocol [Khurshid & Marinov J-ASE’04]

How to Describe/Generate Inputs? Filtering test abstractions [SOFTMC’01, ASE’01, FME’02, ISSTA’02, OOPSLA’02, SAT’03, MIT-TR’03, J-ASE’04, SAT’05, LDTA’06, ALLOY’06, ICSE-DEMO’07, STEP’07, FSE’07, ISSTA’08, ICST’09, CSTVA’10] search input s filters bounds  Testers describe what properties test inputs satisfy  Tool: search Generating test abstractions [FSE’07, FASE’09] execute input s generators bounds  Testers describe how to generate test inputs  Tool: execution

Previously Explored Separately filtersgenerators Inheritance graph DAGX  ValidJava  X Other Java properties property 3  X ……… property NX  Some properties easier/harder as filters/generators Key challenge for combining: search vs. execution

UDITA: Combined Both [ICSE’10] filtersgenerators Inheritance graph DAGX  ValidJava  X Other Java properties property 3  X ……… property NX  Extends Java with non-deterministic choices Found bugs in Eclipse/NetBeans/javac/JPF/UDITA

Outline Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Example: Inheritance Graphs class IG { Node[ ] nodes; int size; static class Node { Node[ ] supertypes; boolean isClass; } Properties DAG  Nodes in the graph should have no direct cycle ValidJava  Each class has at most one superclass  All supertypes of interfaces are interfaces

Example Valid Inheritance Graphs IJ A B G2 C G1 G3 A CB G4 C D J IJ ABC G5

Example Invalid Inheritance Graphs I A B G2 C G1 G3 A CB G4 C J IJ ABC G5 J cycle D class implements class class extends two classes interface extends class class extends interface

(In)valid Inputs Valid inputs = desired inputs  Inputs on which tester wants to test code  Code may produce a regular output or an exception  E.g. for Java compiler Interesting (il)legal Java programs No precondition, input can be any text  E.g. for abstract data types Encapsulated data structure representations Inputs need to satisfy invariants Invalid inputs = undesired inputs

Outline Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Filtering Test Abstractions Tool requires:  Filters: encode what properties inputs satisfy  Bounds Tool provides:  All inputs within bounds that satisfy the properties search input s filters bounds

Language for Filters Each filter takes an input that can be valid or invalid  Returns a boolean indicating validity Experience from academia and industry showed that using a new language makes adoption hard Write filters in standard implementation language (Java, C#, C++…)  Advantages Familiar language Existing development tools Filters may be already present in code  Challenge: generate inputs from imperative filters

Example Filter: DAG Property boolean isDAG(IG ig) { Set visited = new HashSet (); Set path = new HashSet (); if (ig.nodes == null || ig.size != ig.nodes.length) return false; for (Node n : ig.nodes) if (!visited.contains(n)) if (!isAcyclic(n, path, visited)) return false; return true; } boolean isAcyclic(Node node, Set path, Set visited) { if (path.contains(node)) return false; path.add(node); visited.add(node); for (int i = 0; i < supertypes.length; i++) { Node s = supertypes[i]; for (int j = 0; j < i; j++) if (s == supertypes[j]) return false; if (!isAcyclic(s, path, visited)) return false; } path.remove(node); return true; }

Input Space All possible object graphs with an IG root (obeying type declarations)  Natural input spaces relatively easy to enumerate  Sparse: # valid test inputs << # all object graphs

Bounded-Exhaustive Generation Finite (small) bounds for input space  Number of objects  Values of fields Generate all valid inputs within given bounds  Eliminates systematic bias  Finds all bugs detectable within bounds Avoid equivalent (isomorphic) inputs  Reduces the number of inputs  Preserves capability to find all bugs

Example Bounds Specify number of objects for each class  1 object for IG: { IG 0 }  3 objects for Node: { N 0, N 1, N 2 } Specify set of values for each field  For size: { 0, 1, 2, 3 }  For supertypes[i]: { null, N 0, N 1, N 2 }  For isClass: { false, true } Previously written with special library In UDITA: non-deterministic calls for initialization

Bounds Encoded in UDITA IG initialize(int N) { IG ig = new IG(); ig.size = N; ObjectPool pool = new ObjectPool (N); ig.nodes = new Node[N]; for (int i = 0; i < N; i++) ig.nodes[i] = pool.getNew(); for (Node n : nodes) { int num = getInt(0, N − 1); n.supertypes = new Node[num]; for (int j = 0; j < num; j++) n.supertypes[j] = pool.getAny(); n.isClass = getBoolean(); } return ig; } Java with a few extensions for non-determinism static void mainFilt(int N) { IG ig = initialize(N); assume(isDAG(ig)); assume(isValidJava(ig)); println(ig); }

IG 0 N0N0 N1N1 N2N2 sizes[0]s[1]isCs[1]s[0]s[1]isCs[0]isC N1N1 N1N1 null 3231 Example Input Space 1 IG object, 3 Node objects: total 14 fields (“nodes” and array length not shown) null N0N0 N1N1 N2N2 N0N0 N1N1 N2N2 false true null N0N0 N1N1 N2N2 N0N0 N1N1 N2N2 false true null N0N0 N1N1 N2N2 N0N0 N1N1 N2N2 false true 4 * (4 * 4 * 2 * 3) 3 > 2 19 inputs, but < 2 5 valid IG 0 N0N0 N1N1 N2N2 sizes[0]s[1]isCs[1]s[0]s[1]isCs[0]isC

Input Generation: Search Naïve “solution”  Enumerate entire (finite) input space, run filter on each input, and generate input if filter returns true  Infeasible for sparse input spaces (#valid << #total)  Must reason about behavior of filters Our previous work proposed a solution  Dynamically monitors execution of filters  Prunes large parts of input space for each execution  Avoids isomorphic inputs

Outline Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Generating Test Abstractions Tool provides:  Generators: encode how to generate inputs  Bounds Tool requires:  All inputs within bounds (that satisfy the properties) execute input s generators bounds

Example Generator: DAG Property void mainGen(int N) { IG ig = initialize(N); generateDAG(ig); generateValidJava(ig); println(ig); } void generateDAG(IG ig) { for (int i = 0; i < ig.nodes.length; i++) { int num = getInt(0, i); ig.nodes[i].supertypes = new Node[num]; for (int j = 0, k = −1; j < num; j++) { k = getInt(k + 1, i − (num − j)); ig.nodes[i].supertypes[j] = ig.nodes[k]; } N0 N1 N2

Outline Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Filtering vs. Generating: DAG Property boolean isDAG(IG ig) { Set visited = new HashSet (); Set path = new HashSet (); if (ig.nodes == null || ig.size != ig.nodes.length) return false; for (Node n : ig.nodes) if (!visited.contains(n)) if (!isAcyclic(n, path, visited)) return false; return true; } boolean isAcyclic(Node node, Set path, Set visited) { if (path.contains(node)) return false; path.add(node); visited.add(node); for (int i = 0; i < supertypes.length; i++) { Node s = supertypes[i]; for (int j = 0; j < i; j++) if (s == supertypes[j]) return false; if (!isAcyclic(s, path, visited)) return false; } path.remove(node); return true; } void generateDAG(IG ig) { for (int i = 0; i < ig.nodes.length; i++) { int num = getInt(0, i); ig.nodes[i].supertypes = new Node[num]; for (int j = 0, k = −1; j < num; j++) { k = getInt(k + 1, i − (num − j)); ig.nodes[i].supertypes[j] = ig.nodes[k]; } Filtering arguably harder than generating for DAG  20+ vs. 10 LOC However, the opposite for ValidJava property

UDITA Requirement: Allow Mixing filtersgenerators DAGX  ValidJava  X UDITA input s generators bounds filters

Other Requirements for UDITA Language for test abstractions  Ease of use: naturally encode properties  Expressiveness: encode a wide range of properties  Compositionality: from small to large test abstractions  Familiarity: build on a popular language Algorithms and tools for efficient generation of concrete tests from test abstractions Key challenge: search (filtering) and execution (generating) are different paradigms

UDITA Solution Based on a popular language (Java) extended with non-deterministic choices  Base language allows writing filters for search/filtering and generators for execution/generating  Non-deterministic choices for bounds and enumeration  Non-determinism for primitive values: getInt/Boolean  New language abstraction for objects: ObjectPool class ObjectPool { public ObjectPool (int size, boolean includeNull) {... } public T getAny() {... } public T getNew() {... } }

UDITA Implementation Implemented in Java PathFinder (JPF)  Explicit state model checker from NASA  JVM with backtracking capabilities  Has non-deterministic call: Verify.getInt(int lo, int hi) Default/eager generation too slow Efficient generation by delayed choice execution  Publicly available JPF extension Documentation on UDITA web page

Delayed Choice Execution Postpone choice of values until needed Lightweight symbolic execution  Avoids problems with full blown symbolic execution  Even combined symbolic and concrete execution has problems, especially for complex object graphs Eager int x = getInt(0, N); … y = x; … … = y; // non-copy Delayed int x = Susp(0, N); … y = x; … force(y); … = y; // non-copy non-determinism

Object Pools Eager implementation of getNew/getAny by reduction to (eager) getInt  Simply making getInt delayed does not work because getNew is stateful We developed a novel algorithm for delayed execution of object pools  Gives equivalent results as eager implementation  Polynomial-time algorithm to check satisfiability of getNew/getAny constraints (disequality from a set) Previous work on symbolic execution typically encodes into constraints whose satisfiability is NP-hard (disjunctions)

Outline Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Evaluation UDITA  Compared Delayed and Eager execution  Compared with a previous Generating approach  Compared with Pex (white-box) Case studies on testing with test abstractions  Some results with filtering test abstractions  Some results with generating test abstractions  Tested refactoring engines in Eclipse and NetBeans, Java compilers, JPF, and UDITA

Eager vs. Delayed.

Generating vs. UDITA: LOC

Generating vs. UDITA: Time

UDITA vs. Pex Compared UDITA with Pex  State-of-the-art symbolic execution engine from MSR  Uses a state-of-the-art constraint solver Comparison on a set of data structures  White-box testing: tool monitors code under test  Finding seeded bugs Result: object pools help Pex to find bugs  Summer internship to include object size in Pex

Some Case Studies with Filtering Filtering test abstractions used at Microsoft Enabled finding numerous bugs in tested apps  XML tools XPath compiler (10 code bugs, test-suite augmentation) Serialization (3 code bugs, changing spec)  Web-service protocols WS-Policy (13 code bugs, 6 problems in informal spec) WS-Routing (1 code bug, 20 problems in informal spec)  Others SSLStream MSN Authentication 

Some Case Studies with Generating Generating test abstractions used to test Eclipse and NetBeans refactoring engines [FSE’07] Eight refactorings: target field, method, or class Wrote about 50 generators Reported 47 bugs  21 in Eclipse: 20 confirmed by developers  26 in NetBeans: 17 confirmed, 3 fixed, 5 duplicates, 1 won't fix  Found more but did not report duplicate or fixed Parts of that work included in NetBeans

Typical Testing Scenario Create a model of test inputs (e.g., Java ASTs) Write filters/generators for valid inputs UDITA generates valid inputs Translate from model to actual inputs (pretty-print) Run on two code bases, compare outputs pass fail =? one code another code T1 A1 T2 A2 T3 A3 T1 T2 UDITA bounds filters/generators pretty printer

Some More Bugs Found Eclipse vs. NetBeans refactoring engines  2 bugs in Eclipse and 2 bugs in NetBeans Sun javac compiler vs. Eclipse Java compiler  2 reports (still unconfirmed) for Sun javac Java PathFinder vs. JVM  6 bugs in JPF (plus 5 more in an older version) UDITA Delayed vs. UDITA Eager  Applying tool on (parts of) itself  1 bug in UDITA (patched since then)

Example Bug Generated program import java.util.List; class A implements B, D { public List m() { List l = null; A a = null; l.add(a.m()); return l; } } interface D { public List m(); } interface B extends C { public List m(); } interface c { public List m(); } Incorrect refactoring import java.util.List; class A implements B, D { public List m() { List > l = null; A a = null; l.add(a.m()); return l; } } interface D { public List m(); } interface B extends C { public List m(); } interface c { public List m(); }

Outline Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Summary Testing is important for software reliability  Necessary step before (even after) deployment Structurally complex data  Increasingly used in modern systems  Important challenge for software testing Test abstractions  Proposed model for complex test data  Adopted in industry  Used to find bugs in several real-world applications

Benefits & Costs of Test Abstractions Benefits  Test abstractions can find bugs in real code (e.g., at Microsoft or for Eclipse/NetBeans/javac/JPF/UDITA)  Arguably better than manual testing Costs 1. Human time for writing test abstractions (filters or generators) [UDITA, ICSE’10] 2. Machine time for generating/executing/checking tests (human time waiting for the tool) [FASE’09] 3. Human time for inspecting failing tests [FASE’09]

Collaborators from U. of Illinois and other schools  Brett Daniel  Danny Dig  Kely Garcia  Vilas Jagannath  Yun Young Lee Funding  NSF CCF , CNS , CNS  Microsoft gift Credits for Generating TA and UDITA  Milos Gligoric (Belgrade->Illinois)  Tihomir Gvero (Belgrade->EPFL)  Sarfraz Khurshid (UT Austin)  Viktor Kuncak (EPFL)

Test Abstractions Test abstractions  Proposed model for structurally complex test data  Adopted in industry  Used to find bugs in several real-world applications (e.g., at Microsoft, Eclipse/NetBeans/javac/JPF/UDITA) UDITA: latest approach for test abstractions  Easier to write test abstractions  Novel algorithm for faster generation  Publicly available JPF extension