DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel

Motivation ► ► Software Testing emerges as one of the most important aspects in Software Engineering ► ► Unit Testing ► ► Limitations in Unit Testing ► ► DART a novel approach to counter limitations in Unit Testing

Unit Testing Testing small portions of programs, such as methods, groups of methods, or classes. ► ► a sequence of calls to the units (for e.g. methods) ► ► BEFORE the calls some code to generate parameters for the methods ► ► AFTER the calls some code checking whether the method has performed correctly.

Goals of Unit Testing ► ► Detect Errors in the components logic ► ► Check ALL corner cases ► ► Provide 100% code coverage

Question What are some limitations of Unit Testing?

Limitations of Unit Testing ► Requires test driver and harness code which is representative of ALL external environment ► Expensive and hard to perform manually (rarely done properly)

DART To The Rescue Automates unit testing removing need for writing test drivers Automates unit testing removing need for writing test drivers ► Automated extraction of the program interface by static code parsing ► Automatic generation of test driver for this interface that performs random testing ► Dynamic Analysis of the program behavior to generate specific test inputs for the direction of program execution through alternate program paths

Execution Model For DART ► Concrete Execution: based on a memory model M that maps addresses to values ► Symbolic Execution: based on symbolic memory S that maps addresses to expressions

Directed Testing by Constraint Building ► Progressively constraints are built as branches are encountered ► Linear constraints are solved to explore various execution paths ► In case of getting stuck or non linear constraints symbolic execution falls back to concrete execution

Consider An Example …. int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}

Random Testing int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2);}

Random Testing int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2);} Evident Problem: Probability of Reaching Abort is very low

DART Approach main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Full example taken from Presentation by Koushik at

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=36t1=m

DART Approach Concrete Execution concrete state symbolic stateconstraints t1=36, t2=-7t1=m, t2=nmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=36, t2=-7t1=m, t2=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7x=m, y=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7, z=72x=m, y=n, z=2m

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7, z=72x=m, y=n, z=2m 2m != n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m != n x=36, y=-7, z=72x=m, y=n, z=2m

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution solve: 2m = n m=1, n=2 solve: 2m = n m=1, n=2 2m != n x=36, y=-7, z=72x=m, y=n, z=2m

Question Which portion of code identifies a new constraint ?

The Example Again main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}

The Example Again main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1t1=m

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1, t2=2t1=m, t2=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1, t2=2t1=m, t2=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n 2m = n m != n+10

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m = n m != n+10 x=1, y=2, z=2x=m, y=n, z=2m

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m = n m != n+10 x=1, y=2, z=2x=m, y=n, z=2m

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution solve: 2m = n m=1, n=2 solve: 2m = n and m=n+10 m= -10, n= -20 2m = n x=36, y=-7, z=72x=m, y=n, z=2m m != n+10

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10t1=m

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10, t2=-20t1=m, t2=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10, t2=-20t1=m, t2=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20x=m, y=n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20x=m, y=n, z=2m

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20 x=m, y=n, z=2m2m = n

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20 x=m, y=n, z=2m 2m = n m = n+10

DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution Program Error x=-10, y=-20, z=-20 x=m, y=n, z=2m 2m = n m = n+10

Properties of DART ► Sound w.r.t errors found (no false positives) ► Complete (in a limited sense) If DART terminates without reporting a bug, no bug exists If DART terminates without reporting a bug, no bug exists

Advantages: ► Dynamic Data Analysis ► Sound Errors

Dynamic Data Analysis struct foo { int i; char c; } bar (struct foo *a) { if (a->c == 0) { *((char *)a + sizeof(int)) = 1; if (a->c != 0) abort(); }

Dynamic Data Analysis ► Static Analyzers would not be able to detect the change in the value of a->c and declare the program to be safe ► DART locates the error by means of satisfying the constraint a->c==0

Sound Errors foobar(int x, int y){ if (x*x*x > 0){ if (x>0 && y==10) abort(); } else { if (x>0 && y==20) abort(); }

Sound Errors ► Static analysis using predicate abstraction state that both aborts may be reachable ► Test-generation using symbolic execution get stuck at first conditional ► DART detects first abort with high probability

DART For C ► Interface Extraction done by light weight static parsing of source code ► Test Driver Generation involves initialization of external variables with random values, stubs for external functions ► Directed Search comprises of implementation of constraint satisfaction, parsing and analysis of C code

Discussion