A Test Case + Mock Class Generator for Coding Against Interfaces Mainul Islam, Christoph Csallner Software Engineering Research Center (SERC) Computer Science and Engineering Department University of Texas at Arlington, USA February 07, 2011. Published at: 8th International Workshop on Dynamic Analysis (WODA), 2010

Mock Class Generator for Coding Against Interfaces Talk Outline Test case generation Dynamic Symbolic Execution Mock Class Motivating examples to generate Mock Class Algorithm to generate Mock Class Results Demonstration Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Test Case Generation What is the goal behind testing ? public void Test(int x) { if (x > 0) { if (x == 1234567890) { throw new Exception(“Bug”); } Probability to reach the bug (with a random test input) is: ½ * 1/2^31 What can be a good test case to test this method ? Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Symbolic Execution Explore all feasible execution paths Initialize the input with symbolic values and execute the program over symbolic values At conditional statements check if either of the branches can be taken (using a constraint solver) For each path we get an accumulated path condition If (s) then … else … C s true false C` = C ⋀ s C` = C ⋀⌝s Dynamic symbolic execution: Combine Concrete and Symbolic execution Mock Class Generator for Coding Against Interfaces

Dynamic Symbolic Execution by Example Choose Next Path void TestMe (int a[]) { if (a == null) return; if (a.length > 0) if (a[0] == 123) throw new Exception(“Error”); } Solve Execute Constraints to Solve Input Observed Constraints null a == null a == null a != null {} a != null && !(a.length > 0) a == null T F a != null && a.length > 0 {0} a != null && a.length > 0 && a[0] != 123 a.length > 0 T a != null && a.length > 0 && a[0] == 123 {123} a != null && a.length > 0 && a[0] == 123 F a[0] == 123 T F Done: No path left This example is taken from the slides of Nikolai Tillman (with permission)

Introduction to Mock Class public interface A { // … } public class C { public void foo (A a, int x) { Mock Class Generator for Coding Against Interfaces

When we might need to generate mock-classes ? At the initial stage of any development Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Why is this important ? Test case generation is important Interfaces are very important Existing techniques are not very good at this Pex - can generate an instance of a class that implements an interface that is used in the code * * http://research.microsoft.com/en-us/projects/pex/ Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Goal: Support interfaces in test case generation for object-oriented programs Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Motivating Example 1 public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); } public interface A { int m1(int x); } public interface B { int m2(int x); To reach this block of code, ‘a’ must be an instance of A, as well as an instance of B Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Motivating Example 2 public class K implements A, B { // … } public class C { public void bar (A a, int x) { if (a == null) return; if ( !(a instanceof B) ) { B b = (B) a; int z = b.m2(x); public interface A { int m1 (int x); } public interface B { int m2 (int x); K implements all of the interfaces referred by the bar method, but fails reach the code block Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Framework DSC – Dynamic Symbolic Execution Engine (for Java) Z3 – Constraint Solver http://research.microsoft.com/en-us/um/redmod/projects/z3/ Mock Class Generator for Coding Against Interfaces

Our approach to generate Mock Class Determining the type of the Mock Class Generate meaningful method body if needed Mock Class Generator for Coding Against Interfaces

Sub-/Supertype relation in Java Java defines a binary subtype relation If type B implements/extends type A then, - A is a direct super type of B - B is a direct sub type of A Reflexive: A is also a subtype of itself Transitive: if B is a subtype of A and C is a subtype of B then C is also a subtype of A Mock Class Generator for Coding Against Interfaces

Sub-/Supertype relation in Java (Cont.) A class has one direct class super type and arbitrarily many interface super types. Exceptions: type object – has no direct super type type null – has no direct sub type Mock Class Generator for Coding Against Interfaces

Subtype constraints Object public interface A { int m1 (int x); } public interface B { int m2(int x); public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); Object A B C A B C M null Initial types in the system null A desired solution: with new type M Constraints: type(a) subtypeof A type(a) != null type type(a) subtypeof B Mock Class Generator for Coding Against Interfaces

Subtype Relation Matrix public interface A { int m1 (int x); } public interface B { int m2(int x); public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); Null Object A B C x 1 2 3 4 Null Object A B C M x 1 2 3 4 5 Null Object A B C M x 1 2 3 4 5 MA MB MC Null Object x 1 Solution: MA = true MB = true MC = false Mock Class Generator for Coding Against Interfaces

Algorithm: Generating Mock Class Program execution Initialization: NO_MC = 0 MAX_MC = 2 Build Subtype Relation matrix (Generate the constraints) Yes Found Solution? No Solution not found NO_MC < MAX_MC No Yes So, here is how our algorithm works. Once in a while the normal execution of the program is suspended and the correctness of the data structure is checked. If this check by calling repOk() method fails, our By the way, in this slide I-repOk means Instrumented repOk Yes Introduce a new Mock Class, N0_MC++ Encode the Mock Class in the type system Generate Test Case Dynamic Symbolic Data Structure Repair

Generating Method Body: for Example 1 public interface A { int m1 (int x); } public interface B { int m2(int x); public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); if (z >= 10) { //… public class M implements A, B { int m1 (int x){ return 0; } int m2(int x){ Outline of the source code for Mock Class M return 10; Mock Class Generator for Coding Against Interfaces

Results We have run our solution on JMS (Java messaging system) These are all hand-written test cases Interfaces used from JMS Number of goals Number of goals reached w/o mock-classes Number of goals reached with mock-classes Message 6 ByteMessage 5 MapMessage 4 StreamMessage TextMessage ObjectMessage Mock Class Generator for Coding Against Interfaces

Comparison with Pex: Motivating Example 1 public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); } public interface A { int m1(int x); } public interface B { int m2(int x); Goal 1 Goal 2 PEX reaches 1/2 goal(s) DSC reaches 2/2 goal(s) Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Future Work Extend the solution to support more types, e.g. ‘Array’ Determine a upper bound to the number of mock classes needed to solve a system Evaluation Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Future Work (Cont.) Extend method body generation for complex types public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); if (z >= 10) { //… } public class M implements A, B { int m1 (int x){ return 0; } int m2(int x){ Outline of the source code for Mock Class M return 10; Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Demo Mock Class Generator for Coding Against Interfaces

Mock Class Generator for Coding Against Interfaces Thank you! Mock Class Generator for Coding Against Interfaces