Parameterized Unit Tests By Nikolai Tillmann and Wolfram Schulte Proc. of ESEC/FSE 2005 Presented by Yunho Kim Provable Software Lab, KAIST TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A AA A A A A

Contents Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 2/27 Introduction Parameterized unit tests Framework Conclusion

Unit testing is a method of testing that verifies the individual units of source code are working properly From A unit test consists of a sequence of method calls with assertions Unit tests are often served as simple specifications Introduction What Is Unit Testing? Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST3/27 [TestMethod] void TestAdd(){ ArrayList a = new ArrayList(0); // Make an array list object o = new object(); a.Add(o); // Push a value into it Assert.IsTrue(a[0] == o); // Check that value was added }

Unit tests tools do not provide any guidance for – which tests should be written for high code coverage – how to come up with a minimal number of test cases – what guarantees the test cases provide What about other array lists and other objects? Introduction What Is the Problem? Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST4/27 [TestMethod] void TestAdd(){ ArrayList a = new ArrayList(0); // Make an array list object o = new object(); a.Add(o); // Push a value into it Assert.IsTrue(a[0] == o); // Check that value was added }

Parameterized unit tests are generalized unit tests by allowing parameters – Parameterized test methods are specifications of the behavior of the methods-under-tests – PUTs describe a set of traditional unit tests which can be obtained by instantiating the parameterized test methods Introduction Parameterized Unit Tests(PUTs) Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 5/27 [PexMethod] void TestAdd(ArrayList a, object o){ // Given arbitrary array // lists and objects Assume.IsTrue(a!=null); // Assume a  null int i = a.Count; a.Add(o); // Push a value into it Assert.IsTrue(a[i] == o); // Check that value was added }

Contents Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 6/27 Introduction Parameterized unit tests Framework Conclusion

Pex generates Unit Tests from hand-written Parameterized Unit Tests through Automated Exploratory Testing based on Dynami c Symbolic Execution. Parameterized Unit Tests Pex: a PUTs framework Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST7/27 From

Symbolic execution is used to automatically and systematically produce the minimal set of actual parameters(test inputs) Parameterized Unit Tests Test Case Generation Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST8/27 Algorithms 1. For each formal parameter a symbolic variable is introduced 2. For each code path, a path condition is built over symbolic variables 1. Loops and recursions are approximated up to a specified number of unfoldings 3. After all constraints are collected, find solutions for the constraints 4. Finally, deduce what inputs cause a code path to be taken

The number of possible paths to consider can grow exponentially with the number of control flow decisions Parameterized Unit Tests Reusing Parameterized Unit Tests Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST9/27 1 class Bag{ 2 Hashtable h = new Hashtable(); 3 4 void Add(object o){ 5 if (h.ContainsKey(o)) h[o] = (int)h[o] + 1; 6 else h[o] = 1; 7 } 8 int multiplicity(object o){ 9 if (h.ContainsKey(o)) return (int)h[o]; 10 else return 0; 11 } 12 } 1 [PexMethod] 2 Void AddMultiplicityTest(Bag b, objectx){ 3 Assume.IsTrue(b!=null & x!=nul); 4 int m = b.multiplicity(x); 5 b.Add(x); 6 Assert.IsTrue(m+1 == b.Multiplicity(x)); During symbolic execution, ContainsKey method is symbolically run twice

If we have axioms summarizing the hash table’s operations, we can avoid execution of the hash table Parameterized Unit Tests Reusing Parameterized Unit Tests Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST10/27 [TestClass, UsingAxioms(typeof(HashtableTests))] Class BagTests{ [PexMethod] void AddMultiplicityTest(  ) {  }  } [TestClass, ProvidingAxioms(typeof(Hashtable))] Class HashtableTests{ [TestAxiom] void NothingContainedInitially(object key){  } [TestAxiom] void SetImpliesContainsAndGet(  ){  }  } Provided axioms are re- written as constraints

Contents Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 11/27 Introduction Parameterized unit tests Framework – Symbolic state – Constraints – Symbolic Evaluation – Axioms Conclusion

Symbolic states are like concrete states except that symbolic states can contain symbolic expressions Symbolic expressions E is described by the following grammar ObjectId is an infinite set of potential object identifiers VarId, TypeId and FuncId are a set of variable, type and function identifiers respectively Framework Symbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST12/27 E = o object ids 2 ObjectId | v variables 2 VarId | t types 2 TypeId | f ( ) function application 2 FuncId | 8. E universal quantification

There are two types of function symbols Predefined function symbols have a fixed interpretation in the theorem provers used –add ( x, y ), equals ( x, y ), subtype ( x, y ) – null, void, 0, 1, 2, … (constants are considered as nullary functions) – Storage function symbols operating on maps update ( m, x, y ) denotes the update of map m at index x to the new value y select ( m, x ) selects the value of map m at index x Uninterpreted function symbols represent properties of objects and method calls appearing in axioms Framework Symbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST13/27

Uninterpreted function symbols represent properties of objects and method calls appearing in axioms – For example, type ( x ) denotes the type of object x, field f ( x ) denotes the address of field f of object x For each method m of the program with n parameters, up to three uninterpreted function symbols are introduced, m s, m x, m r – They are used to summarize different dynamic aspects of m Let h be an expression denoting the state of the heap in which m ( ) is called –m s ( h, ) denotes the resulting state of the heap –m r ( h, ) denotes the return value of the call –m x ( h, ) represents the type of an exception Framework Symbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST14/27

There are two kinds of heaps, ‘extensional’ and ‘intensional’ heap The extensional heap is represented by nested applications of the map update function For example, the execution of the code fragment turns a given extensional heap H e into H ’ e assuming the locations p, q hold the expressions t, u Extensional heap operations have fixed interpretations in the theorem provers Framework Symbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST15/27 p.f=1; q.g=2 H ’ e = update ( update ( H e, field f ( t ), 1), field g ( u ), 2)

The intensional heap is described by a history of method invocations m s ( H i, ) represents the sequence of method calls followed by a call to m ( ) Consider for example the execution of the following code fragment assuming that t is the expression held in location o Intensional heap operations are uninterpreted functions Framework Symbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST16/27 ArrayList a = new ArrayList(); a.Add(o); H ’ i = Add s ( ArrayList s ( H i, a ), a, t )

A symbolic state is a 5-tuple S = ( O, A, H e, H i, X ) –O ½ ObjectId –A is a stack of activation records associated with a method, a program counter, as well as a store for the formal parameters and local variables –H e and H i are expression denoting the extensional heap and the intensional heap respectively –X is an object expression denoting the current exception S +1 is like S except that the program counter has been incremented The set of all symbolic states is called State Framework Symbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST17/27

A constraint on a symbolic state is a pair C = ( BG, PC ) BG is the static background only depending on the program declarations PC is the dynamic path condition built up during symbolic execution The set of all constraints is called Constraints A constrained state is a pair ( S, C ) Framework Constraints Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST18/27

The one-step relation describes the effect of the current instruction from a given constrained state ( S, C ) Most instructions are handled in the standard way – A method call instruction pushes a new activation record onto the program stack The interesting cases are object creation, conditional branch, member access, assume and assert Framework Symbolic Evaluation Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST19/27 ! µ ( State £ Constraints ) £ ( State £ Constraints )

New object creation of type T – Let o 2 ObjectId – O ( S ) be a fresh object identifier – ( S, C ) ! ( S ’, C Æ type ( o ) = T ) where S ’ is like S +1 except that H e ( S ) is replaced with – Where f 1, , f n are the fields of T and v 1, , v n default values Conditional branch with a condition c and label l – If ( S, C Æ c ) is feasible, then ( S, C ) ! ( S ’, C Æ c ) where the program counter in S ’ is set to l – If ( S, C Æ : c) is feasible, then ( S, C ) ! ( S +1, C Æ : c ) Framework Symbolic Evaluation Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST20/27 update (  update ( h, field f1 ( o ), v 1 ) , field fn ( o ), v n )

Member access with target expression t and result location r – If ( S, C Æ t  null ) is feasible, then ( S, C ) ! ( S ’, C Æ t  null ) where S ’ is like S +1 except that the location r holds ( H e ( S ), field f ( t )) – If ( S, C Æ t = null ) is feasible, then ( S, C ) ! ( S’ ’, C Æ type ( e ) = NullReferenceException Æ t = null ) where S ’’ is like S except that the current exception X = e Assume.IsTrue with condition c – If ( S, C Æ c ) is feasible, then ( S, C ) ! ( S +1, C Æ c ) Assert.IsTrue with condition c is semantically equivalent to Framework Symbolic Evaluation Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST21/27 If (!c) throw new AssertFailedException();

PUT can be seen as a summary, an axiom of external behavior of method-under-test To summarize the set of methods M, we refine the behavior of ! relation for method calls to M Call to a method m 2 M – Let H ’ i = m s ( H i ( S ), ). – If ( S, C Æ m x ( H i ( S ), ) = void ) is feasible, ( S, C ) ! ( S ’, C Æ m x ( h i ( S ), ) = void) where S ’ is S +1 except that H i is replaced with H ’ i – IF ( S, C Æ m x ( H i ( S ), )  void ) is feasible, ( S, C ) ! ( S ’’, C Æ m x ( H i ( S ), )  void ) where S ’’ is S +1 except that H i is replaced with H ’ i and X = m x ( h i ( S ), ) Framework Axioms Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST22/27

PUTs generate axiom formulas for by exploring a test axiom method like we do for test case generation PUTs explore the method with a modified one-step relation ! ‘ ! ‘ is like except that Assert.IsTrue ( c ) in a state ( S, C ) is treated like Assume.IsTrue ( c ) and a new axiom formula 8 h,. PC ( C ) ! c is generated and conjoined BG ( C ) Parameterized Unit Tests Axioms Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST23/27 [PexMethod] void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4 }

For example Parameterized Unit Tests Axioms Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST24/27 [PexMethod] void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4 }

For example Exploring TestAdd generates the following universally quantified formula as an axiom Parameterized Unit Tests Axioms Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST25/27 [PexMethod] void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4 } 8 h,a,o. (a = null Ç subtype(type(a), ArrayList)) Æ a  null ! getItem r ( Add s (getCount s (h,a),a,o), a, getCount r (h,a)) = o

Conclusion Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 26/27 The authors presented the concept of parameterized unit tests, a generalization of unit tests PUTs can be turned into axioms, which summarize three different aspects of the methods’ behavior – The axioms can be reused during symbolic execution

Reference Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 27/27 Parameterized Unit Tests by Parameterized Unit Tests in Proc. of ESEC/FSE 2005