Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph.

Slides:



Advertisements
Similar presentations
Program Verification Using the Spec# Programming System ETAPS Tutorial K. Rustan M. Leino, Microsoft Research, Redmond Rosemary Monahan, NUIM Maynooth.
Advertisements

Leonardo de Moura Microsoft Research. Z3 is a new solver developed at Microsoft Research. Development/Research driven by internal customers. Free for.
Semantics Static semantics Dynamic semantics attribute grammars
Satisfiability Modulo Theories (An introduction)
Identity and Equality Based on material by Michael Ernst, University of Washington.
Abstraction and Modular Reasoning for the Verification of Software Corina Pasareanu NASA Ames Research Center.
Parallel Symbolic Execution for Structural Test Generation Matt Staats Corina Pasareanu ISSTA 2010.
1 Symbolic Execution for Model Checking and Testing Corina Păsăreanu (Kestrel) Joint work with Sarfraz Khurshid (MIT) and Willem Visser (RIACS)
1/20 Generalized Symbolic Execution for Model Checking and Testing Charngki PSWLAB Generalized Symbolic Execution for Model Checking and Testing.
Chapter 1: Computer Systems
Symbolic execution © Marcelo d’Amorim 2010.
Hybrid Concolic Testing Rupak Majumdar Koushik Sen UC Los Angeles UC Berkeley.
CSE503: SOFTWARE ENGINEERING SYMBOLIC TESTING, AUTOMATED TEST GENERATION … AND MORE! David Notkin Spring 2011.
Pexxxx White Box Test Generation for
DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel.
Synthesis of Interface Specifications for Java Classes Rajeev Alur University of Pennsylvania Joint work with P. Cerny, G. Gupta, P. Madhusudan, W. Nam,
COMP 14: Intro. to Intro. to Programming May 23, 2000 Nick Vallidis.
CHAPTER 10 Recursion. 2 Recursive Thinking Recursion is a programming technique in which a method can call itself to solve a problem A recursive definition.
Chapter 2: Algorithm Discovery and Design
Software Testing and QA Theory and Practice (Chapter 4: Control Flow Testing) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory and Practice.
272: Software Engineering Fall 2012 Instructor: Tevfik Bultan Lecture 4: SMT-based Bounded Model Checking of Concurrent Software.
Copyright © 2009 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Java Software Solutions Foundations of Program Design Sixth Edition by Lewis.
Java Software Solutions Lewis and Loftus Chapter 2 1 Copyright 1997 by John Lewis and William Loftus. All rights reserved. Software Concepts -- Introduction.
1 Identifiers  Identifiers are the words a programmer uses in a program  An identifier can be made up of letters, digits, the underscore character (
DySy: Dynamic Symbolic Execution for Invariant Inference.
OOPs Object oriented programming. Based on ADT principles  Representation of type and operations in a single unit  Available for other units to create.
Software Engineering Prof. Dr. Bertrand Meyer March 2007 – June 2007 Chair of Software Engineering Static program checking and verification Slides: Based.
Java: Chapter 1 Computer Systems Computer Programming II.
Introduction to Java Prepared by: Ahmed Hefny. Outline Classes Access Levels Member Initialization Inheritance and Polymorphism Interfaces Inner Classes.
Advanced Programming Collage of Information Technology University of Palestine, Gaza Prepared by: Mahmoud Rafeek Alfarra Lecture 2: Major Concepts of Programming.
Introduction to Programming David Goldschmidt, Ph.D. Computer Science The College of Saint Rose Java Fundamentals (Comments, Variables, etc.)
Hello.java Program Output 1 public class Hello { 2 public static void main( String [] args ) 3 { 4 System.out.println( “Hello!" ); 5 } // end method main.
Parameterized Unit Tests By Nikolai Tillmann and Wolfram Schulte Proc. of ESEC/FSE 2005 Presented by Yunho Kim Provable Software Lab, KAIST TexPoint fonts.
Boolean expressions, part 2: Logical operators. Previously discussed Recall that there are 2 types of operators that return a boolean result (true or.
Java The Java programming language was created by Sun Microsystems, Inc. It was introduced in 1995 and it's popularity has grown quickly since A programming.
CMP-MX21: Lecture 4 Selections Steve Hordley. Overview 1. The if-else selection in JAVA 2. More useful JAVA operators 4. Other selection constructs in.
Documentation Dr. Andrew Wallace PhD BEng(hons) EurIng
Java Basics Hussein Suleman March 2007 UCT Department of Computer Science Computer Science 1015F.
An Undergraduate Course on Software Bug Detection Tools and Techniques Eric Larson Seattle University March 3, 2006.
A Test Case + Mock Class Generator for Coding Against Interfaces Mainul Islam, Christoph Csallner Software Engineering Research Center (SERC) Computer.
 In the java programming language, a keyword is one of 50 reserved words which have a predefined meaning in the language; because of this,
PROGRAMMING TESTING B MODULE 2: SOFTWARE SYSTEMS 22 NOVEMBER 2013.
David Evans CS201j: Engineering Software University of Virginia Computer Science Lecture 10: Programming Exceptionally.
Symbolic Execution in Software Engineering By Xusheng Xiao Xi Ge Dayoung Lee Towards Partial fulfillment for Course 707.
© Dr. A. Williams, Fall Present Software Quality Assurance – Clover Lab 1 Tutorial / lab 2: Code instrumentation Goals of this session: 1.Create.
CS223: Software Engineering Lecture 26: Software Testing.
1 Problem Solving  The purpose of writing a program is to solve a problem  The general steps in problem solving are: Understand the problem Dissect the.
Modern Programming Tools And Techniques-I
JAVA MULTIPLE CHOICE QUESTION.
Working with Java.
Chapter 4 Assignment Statement
Dynamic Symbolic Execution
Chapter 3 Assignment Statement
A Test Case + Mock Class Generator for Coding Against Interfaces
RDE: Replay DEbugging for Diagnosing Production Site Failures
Java Software Structures: John Lewis & Joseph Chase
Microsoft Visual Basic 2005 BASICS
White-Box Testing Using Pex
New Ideas Track: Testing MapReduce-Style Programs
New Ideas Track: Testing MapReduce-Style Programs Christoph Csallner, Leonidas Fegaras, Chengkai Li Computer.
Dynamic Symbolic Data Structure Repair
Starting JavaProgramming
null, true, and false are also reserved.
Chapter 1: Computer Systems
Focus of the Course Object-Oriented Software Development
Diagnostic Evaluation
Introduction to Programming
Chap 2. Identifiers, Keywords, and Types
Outline Software Development Activities
Review for Midterm 3.
Presentation transcript:

Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph Csallner December 12 th, 2012 The University of Texas at Arlington PhD Research Proposal

Outline Problem Description Motivation Limitation of Current Approaches (with example) Thesis Statement Background Solution Approach Experimental Results Plan of Action 2

Problem Description Current state of the art tools are not very good at generating test cases when the code under test: – requires additional pieces of code that are not yet part of the program. – uses multiple interfaces, annotations or reflection. – imposes complex (type) constraints. – uses native code. 3

Motivation Automatic test case generation is important At the initial stage of any development the implementation of some code may not be available Existing techniques (such as Pex and Moles*) are not very good at generating test cases when the code under test uses interfaces/annotations, multiple inheritance and native code. Significant number of Java programs use native code 4 *

5 Limitation of Current Techniques: Motivating Example 1 A { /* … */ } public interface I { public int m1(); /* … */ } public interface J { public int m2(); /* … */ } public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } if ( i.getClass().isAnnotationPresent(A.class) ) { //.. } To reach this block of code ‘i’ must be an instance of I, as well as an instance of J To reach this block of code ‘i’ must be an instance of I, as well as annotated with A

6 Motivating Example 2 (Dependency on Native Code) public class C { public native boolean isDivisible(int x, int y); public static void NativeTest(int a, int b) { boolean divisible = new C().isDivisible(a, b); if (divisible) { // … } bool isDivisible(int x, int y) { if ( y*(x/y) == x ) return true; return false; } Java codeC++ code To reach this block of code ‘a’ must be divisible by ‘b’

Motivation: Patterns in Real World Program 7

Pattern 1 8 R m(…, T t, …) { // … if ( … (t instanceof X) ) // … } ‘T’ an ‘X’ are non-compatible ‘m’ is user-defined At least one of {T, X} is an interface None of {T, X} is final At least one of {T, X} is a user-type

Thesis Statement We can generate automatic test inputs and systematically increase code coverage compared to existing techniques specially when the code under test: – requires additional pieces of code that are not yet part of the program. – uses multiple interfaces, annotations or reflection. – imposes complex (type) constraints. – uses native code. 9

Background: Symbolic Execution 10 Systematically explore all feasible execution paths Initialize the input with symbolic values and execute the program over symbolic values At conditional statements check if either of the branches can be taken For each path get an accumulated path condition C` = C ⋀ SC` = C ⋀ ⌝ S S truefalse C If (S) then … else …

Dynamic Symbolic Execution by Example taken with permission from Nikolai Tillmann (Microsoft Research) 11 public static void TestMe(int a[]) { if (a == null) return; if (a.length > 0) if (a[0] == 123) throw new Exception(“Error”); } a == null a[0] == 123 a.length > 0 FT FT FT Constraints to Solve Input (a) Observed Constraints null a == null a != null {} a != null && !(a.length > 0) a != null && a.length > 0 {0} a != null && a.length > 0 && a[0] != 123 a != null && a.length > 0&& a[0] == 123 {123} a != null && a.length > 0 && a[0] == 123 SolveExecute Choose Next Path Done: No Path Left

Java defines a binary sub-type relation If type B implements/extends type A then, – A is a direct super-type of B – B is a direct sub-type of A Reflexive: A is also a subtype of itself Transitive: if B is a subtype of A and C is a subtype of B then C is also a subtype of A 12 Sub-/Supertype relation in Java

A class has one direct class super type and arbitrarily many interface super types. Exceptions: type object – has no direct super type type null – has no direct sub type 13 Sub-/Supertype relation in Java

Introduce Dynamic Symbolic Mock Classes Technique implemented on:  Dsc 1 – Dynamic Symbolic Execution Engine for Java  Z3 2 – SMT solver (from Microsoft Research) 14 Our Solution Approach

15 Solution Workflow Default Input values (0, null, …) New Test Cases (and corresponding mock classes ) Invoke DSE on given input values and collect path constraints Map the constraint solver model to new test cases (and mock classes) Invert one of the collected path constraints (Map each reference type to a constraint literal and encode their properties to build a constraint system) More Paths ? Add mock classes and map each of them to a constraint literal Encode properties (e.g., subtype relation) of mock classes in the constraint system Stop Constraint System Satisfiable ? No Yes Constraint System Satisfiable ? Yes No

16 Subtype Constraints A { /* … */ } public interface I { public int m1(); /* … */ } public interface J { public int m2(); /* … */ } public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } //.. } A desired solution with new Type: M, to reach the code block Object IJC null Initial Types in the system A Annotation M Constraints: type(i) subtypeof I type(i) != null type type(i) subtypeof J

17 Subtype Constraints A { /* … */ } public interface I { public int m1(); /* … */ } public interface J { public int m2(); /* … */ } public class C { public static void foo(I i) { int x = i.m1(); //.. if ( i.getClass(). isAnnotationPresent(A.class) ) { //.. } A desired solution with new Type: M1, to reach the code block Object IJC null Initial Types in the system A Annotation M1 Constraints: type(i) subtypeof I type(i) subtypeof A type(i) != null type

18 Subtype Relation Matrix A { /* … */ } public interface I { public int m1(); /* … */ } public interface J { public int m2(); /* … */ } public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } //.. } nullObjectAnIJCAM 0nullxxxxxxxx 1Objectx 2Anxx 3Ixx 4Jxx 5Cxx 6Axxx 7MxmAnmImJmCmAx Solution: mI = true mJ = true mC = false mA = mAn = false nullObjectAnIJCA 0nullxxxxxxx 1Objectx 2Anxx 3Ixx 4Jxx 5Cxx 6Axxx nullObjectAn 0nullxxx 1Objectx 2Anxx

Experimental Results (1) 19 Experiments are done on simplified version of real world code C# codes are translated manually

Experimental Results (2) 20 Experiments are done on original code Randoop has several side effects

21 Workflow: to handle Native Code Default Input values (0, null, …) New Test Cases Invoke DSE on given input values and collect path constraints Map the constraint solver model to new test cases Invert one of the collected path constraints and build constraint system More Paths ? Native Code Invoked? Collect the constraints from the native code for current input values Convert the native code constraints to Java constraints Stop Constraint System Satisfiable ? No Yes

Technique to handle Native Code 22 In each iteration of the dynamic symbolic execution of a Java program: Check if a native code call is invoked. If yes, start executing the native code on current input values Use any existing tool (such as Klee* for C++ code) to collect the path constraints of the native code Convert the constraints collected from the native code and merge them with the constraints previously collected from the Java program Solve the whole constraint system using a Constraint solver and generate test case.

Initial Experiments 23 In JDK 1.6: Total # types: Total # of types (have at least one native method): 381 Total # of native methods:

Plan of Actions TimeAction Plan (to do) Spring 2013Implement and evaluate the technique to handle native code. Submit a short paper Summer 2013Internship (currently interviewing) Join back to UTA in Fall 2013 Fall 2013Detail experiments of the technique to handle native code against the existing techniques. Submit the experiment results to a journal paper Spring 2014Write up and submit dissertation. Ph.D. defense 24

List of publications 1.Mainul Islam and Christoph Csallner. Generating Test Cases for Programs that are Coded Against Interfaces and Annotations ( submitted ) 2.Mainul Islam and Christoph Csallner. Dsc+Mock: A test case + mock class generator in support of coding against interfaces (In Proc. 8 th International Workshop on Dynamic Analysis (WODA), co-located with International Symposium on Software Testing and Analysis (ISSTA), 2010 ) 25

Thank You! 26

Backup Slides 27

Pattern 2 28 class P {T t, …} class Q { M m(…, P p, …) { // … if ( … (p.t instanceof X) ) // … }

Pattern 3 29 M m(…) { // … if ( … (..).foo() instanceof X ) // … }

Related work 30 Mock classes inferred from Programmer-Written specification: EasyMock, jMock, Mockito NMock Google Mock SynthiaMock

Related work: example with EasyMock 31 public interface K extends I, J { /* … */ } public class Test { public void testFooIsInstanceOfJ() { K mock = createMock(K.class); expect(mock.m1()).andReturn(0); expect(mock.m2()).andReturn(0); replay(mock); C.foo(mock); verify(mock); }

u1u1 0.2 u2u2 0.6 u3u3 0.4 u4u4 0.8 u5u Visual similarity Geo similarity