1. Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph Csallner December 12 th, 2012 The University of Texas at Arlington PhD Research Proposal

2. Outline Problem Description Motivation Limitation of Current Approaches (with example) Thesis Statement Background Solution Approach Experimental Results Plan of Action 2

3. Problem Description Current state of the art tools are not very good at generating test cases when the code under test: – requires additional pieces of code that are not yet part of the program. – uses multiple interfaces, annotations or reflection. – imposes complex (type) constraints. – uses native code. 3

4. Motivation Automatic test case generation is important At the initial stage of any development the implementation of some code may not be available Existing techniques (such as Pex and Moles*) are not very good at generating test cases when the code under test uses interfaces/annotations, multiple inheritance and native code. Significant number of Java programs use native code 4 * http://research.microsoft.com/en-us/projects/pex/http://research.microsoft.com/en-us/projects/pex/

5. 5 Limitation of Current Techniques: Motivating Example 1 public @interface A { /* … */ } public interface I { public int m1(); /* … */ } public interface J { public int m2(); /* … */ } public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } if ( i.getClass().isAnnotationPresent(A.class) ) { //.. } To reach this block of code ‘i’ must be an instance of I, as well as an instance of J To reach this block of code ‘i’ must be an instance of I, as well as annotated with A

6. 6 Motivating Example 2 (Dependency on Native Code) public class C { public native boolean isDivisible(int x, int y); public static void NativeTest(int a, int b) { boolean divisible = new C().isDivisible(a, b); if (divisible) { // … } bool isDivisible(int x, int y) { if ( y*(x/y) == x ) return true; return false; } Java codeC++ code To reach this block of code ‘a’ must be divisible by ‘b’

7. Motivation: Patterns in Real World Program 7

8. Pattern 1 8 R m(…, T t, …) { // … if ( … (t instanceof X) ) // … } ‘T’ an ‘X’ are non-compatible ‘m’ is user-defined At least one of {T, X} is an interface None of {T, X} is final At least one of {T, X} is a user-type

9. Thesis Statement We can generate automatic test inputs and systematically increase code coverage compared to existing techniques specially when the code under test: – requires additional pieces of code that are not yet part of the program. – uses multiple interfaces, annotations or reflection. – imposes complex (type) constraints. – uses native code. 9

10. Background: Symbolic Execution 10 Systematically explore all feasible execution paths Initialize the input with symbolic values and execute the program over symbolic values At conditional statements check if either of the branches can be taken For each path get an accumulated path condition C` = C ⋀ SC` = C ⋀ ⌝ S S truefalse C If (S) then … else …

11. Dynamic Symbolic Execution by Example taken with permission from Nikolai Tillmann (Microsoft Research) 11 public static void TestMe(int a[]) { if (a == null) return; if (a.length > 0) if (a[0] == 123) throw new Exception(“Error”); } a == null a[0] == 123 a.length > 0 FT FT FT Constraints to Solve Input (a) Observed Constraints null a == null a != null {} a != null && !(a.length > 0) a != null && a.length > 0 {0} a != null && a.length > 0 && a[0] != 123 a != null && a.length > 0&& a[0] == 123 {123} a != null && a.length > 0 && a[0] == 123 SolveExecute Choose Next Path Done: No Path Left

12. Java defines a binary sub-type relation If type B implements/extends type A then, – A is a direct super-type of B – B is a direct sub-type of A Reflexive: A is also a subtype of itself Transitive: if B is a subtype of A and C is a subtype of B then C is also a subtype of A 12 Sub-/Supertype relation in Java

13. A class has one direct class super type and arbitrarily many interface super types. Exceptions: type object – has no direct super type type null – has no direct sub type 13 Sub-/Supertype relation in Java

14. Introduce Dynamic Symbolic Mock Classes Technique implemented on:  Dsc 1 – Dynamic Symbolic Execution Engine for Java  Z3 2 – SMT solver (from Microsoft Research) 14 Our Solution Approach 1 - http://ranger.uta.edu/~csallner/dsc/index.htmlhttp://ranger.uta.edu/~csallner/dsc/index.html 2 - http://z3.codeplex.com/http://z3.codeplex.com/

15. 15 Solution Workflow Default Input values (0, null, …) New Test Cases (and corresponding mock classes ) Invoke DSE on given input values and collect path constraints Map the constraint solver model to new test cases (and mock classes) Invert one of the collected path constraints (Map each reference type to a constraint literal and encode their properties to build a constraint system) More Paths ? Add mock classes and map each of them to a constraint literal Encode properties (e.g., subtype relation) of mock classes in the constraint system Stop Constraint System Satisfiable ? No Yes Constraint System Satisfiable ? Yes No

16. 16 Subtype Constraints public @interface A { /* … */ } public interface I { public int m1(); /* … */ } public interface J { public int m2(); /* … */ } public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } //.. } A desired solution with new Type: M, to reach the code block Object IJC null Initial Types in the system A Annotation M Constraints: type(i) subtypeof I type(i) != null type type(i) subtypeof J

17. 17 Subtype Constraints public @interface A { /* … */ } public interface I { public int m1(); /* … */ } public interface J { public int m2(); /* … */ } public class C { public static void foo(I i) { int x = i.m1(); //.. if ( i.getClass(). isAnnotationPresent(A.class) ) { //.. } A desired solution with new Type: M1, to reach the code block Object IJC null Initial Types in the system A Annotation M1 Constraints: type(i) subtypeof I type(i) subtypeof A type(i) != null type

18. 18 Subtype Relation Matrix public @interface A { /* … */ } public interface I { public int m1(); /* … */ } public interface J { public int m2(); /* … */ } public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } //.. } nullObjectAnIJCAM 0nullxxxxxxxx 1Objectx 2Anxx 3Ixx 4Jxx 5Cxx 6Axxx 7MxmAnmImJmCmAx Solution: mI = true mJ = true mC = false mA = mAn = false nullObjectAnIJCA 0nullxxxxxxx 1Objectx 2Anxx 3Ixx 4Jxx 5Cxx 6Axxx nullObjectAn 0nullxxx 1Objectx 2Anxx

19. Experimental Results (1) 19 Experiments are done on simplified version of real world code C# codes are translated manually

20. Experimental Results (2) 20 Experiments are done on original code Randoop has several side effects

21. 21 Workflow: to handle Native Code Default Input values (0, null, …) New Test Cases Invoke DSE on given input values and collect path constraints Map the constraint solver model to new test cases Invert one of the collected path constraints and build constraint system More Paths ? Native Code Invoked? Collect the constraints from the native code for current input values Convert the native code constraints to Java constraints Stop Constraint System Satisfiable ? No Yes

22. Technique to handle Native Code 22 In each iteration of the dynamic symbolic execution of a Java program: Check if a native code call is invoked. If yes, start executing the native code on current input values Use any existing tool (such as Klee* for C++ code) to collect the path constraints of the native code Convert the constraints collected from the native code and merge them with the constraints previously collected from the Java program Solve the whole constraint system using a Constraint solver and generate test case. http://klee.llvm.org/

23. Initial Experiments 23 In JDK 1.6: Total # types: 23799 Total # of types (have at least one native method): 381 Total # of native methods: 2019 http://klee.llvm.org/

24. Plan of Actions TimeAction Plan (to do) Spring 2013Implement and evaluate the technique to handle native code. Submit a short paper Summer 2013Internship (currently interviewing) Join back to UTA in Fall 2013 Fall 2013Detail experiments of the technique to handle native code against the existing techniques. Submit the experiment results to a journal paper Spring 2014Write up and submit dissertation. Ph.D. defense 24

25. List of publications 1.Mainul Islam and Christoph Csallner. Generating Test Cases for Programs that are Coded Against Interfaces and Annotations ( submitted ) 2.Mainul Islam and Christoph Csallner. Dsc+Mock: A test case + mock class generator in support of coding against interfaces (In Proc. 8 th International Workshop on Dynamic Analysis (WODA), co-located with International Symposium on Software Testing and Analysis (ISSTA), 2010 ) 25

26. Thank You! 26

27. Backup Slides 27

28. Pattern 2 28 class P {T t, …} class Q { M m(…, P p, …) { // … if ( … (p.t instanceof X) ) // … }

29. Pattern 3 29 M m(…) { // … if ( … (..).foo() instanceof X ) // … }

30. Related work 30 Mock classes inferred from Programmer-Written specification: EasyMock, jMock, Mockito NMock Google Mock SynthiaMock

31. Related work: example with EasyMock 31 public interface K extends I, J { /* … */ } public class Test { public void testFooIsInstanceOfJ() { K mock = createMock(K.class); expect(mock.m1()).andReturn(0); expect(mock.m2()).andReturn(0); replay(mock); C.foo(mock); verify(mock); }

32. 32 0.5 u1u1 0.2 u2u2 0.6 u3u3 0.4 u4u4 0.8 u5u5 0.3 0.4 0.2 0.9 0.5 Visual similarity Geo similarity