Technology of Test Case Generation Levi Lúcio University of Geneva Marko Samer Vienna University of Technology

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Introduction In this presentation three technologies for test case generation are introduced; –Symbolic Execution; –Theorem Proving; –Model Checking (presentation not available); All the models used by these methods can be reduced to state machines; Test case generation can be seen as state space exploration for traces obeying a criteria.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Symbolic Execution Talk Overview The technique –How symbolic execution works; –Issues related to the approach; –Symbolic execution of abstract models; Test case generation –A classification of test case generation frameworks using symbolic execution; –Abstract model based test case generation; –Code based test case generation.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Symbolic Execution Technique invented in the 1970s for verifying the consistency of code; The main principle consists of executing the code with symbolic inputs… … in which a symbolic input consists of replacing a « real input » (number, string, structure, object) with a symbol; For example, if the program asks a number to the user, «  » would be passed to the program as a representation of all possible numbers.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation The technique Allows exploring the possible control paths in an program. Calculates the path condition for each control path, composed of symbolic equations. Int foo (int a, int b) { 1a++; 2if (a>b) 3 a=a-b; 4else 5 a=b-a; 6if (a<=-1) 7 a=-a; 8return a } Control paths for foo 1,2,3,6,7,8 1,2,3,6,8 1,2,5,6,7,8 1,2,5,6,8

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation The technique (cont) a:  b:  PC: True a:  +1  b:  PC: True (1) a:  b:  PC:  (3) (8) a:  b:  PC:  a:  b:  PC:  a:  b:  PC:  (2 -true)(2 - false) … a:  b:  PC:  a:  b:  PC:  (6 - true)(6 - false) No solution for PC equations: Infeasible Control Path (search backtracks) Int foo (int a, int b) { 1a++; 2if (a>b) 3 a=a-b; 4else 5 a=b-a; 6if (a<=-1) 7 a=-a; 8return a }

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation The technique (cont) From the symbolic execution of foo, one can retrieve the following information: Control PathPath Condition 1,2,3,6,7,8  Not feasible 1,2,3,6,8    1,2,5,6,7,8  Not feasible 1,2,5,6,8    Each path condition fully documents the decisions taken during the execution of the code.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test case generation It is possible to generate test cases by finding values that satisfy the path condition equations: Control PathPath ConditionExample test case 1,2,5,6,8  Test case (a=1, b=3) would force the execution of foo to follow the control path 1,2,5,6,8; Test cases are thus created as by-products of the activity of symbolic execution; Symbolic execution uncovered that in the case of foo instruction 7 is never reached...

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Issues related to the approach Dealing with loops –Loops generate infinite control paths (unless the number of iterations is known). Solution? Prompt the user at each loop iteration; Establish an upper bound for the number of iterations; Try to automatically find a fixed point to the loop (costly). Solving path condition equations –Necessary for: understanding when a path is infeasible; generating test cases. –Algorithms heavy on computing resources.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Symbolic execution of abstract models Up until now we have considered symbolic execution only as a “white box” verification technique… However, nothing prevents from applying the same technique to an abstract model (state machine) Code Abstract Model (state machine) Symbolic Execution Test Cases State space of a model can be symbolically searched for interesting control paths; Symbolic execution reduces state space explosion by associating classes of inputs.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Symbolic execution for test case generation In the text several frameworks for test case generation from different models of the application are studied; We classify the frameworks according to two axis: Application model Synergies with other verification techniques Model Checking Theorem proving Abstract (B, AUTOFOCUS, CO-OPN), Code (Java)

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Abstract model based test case generation In the text three frameworks are described that have their starting point on different abstract models: –B(framework from Legeard, Peureux et al) –AUTOFOCUS(framework from Pretschner et al) –CO-OPN(framework from Buchs et al) The models can be considered as state machines and the technique for test case generation is similar: Abstract Model (state machine) Logic Programming Model Test Cases Translation step Symbolic state space search

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Abstract model based test case generation (cont) The step of translation into a logic programming language (theorem prover) is necessary to: –Animate the (static) abstract specification; –Search the state space; –Perform symbolic execution. Two kinds of logic programming languages may be used: –Pure Prolog; –Constraint Logic Programming (CLP) Languages: enabled to deal with numeric constraints at the unification level.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test generation from an Abstract Model Example The specification language chosen is B - supports development of C code from an abstract model; In B an application can be seen as a state machine: –States: defined by state variables; –Transitions: parameterized operations. Framework from Legeard et al divides the activity of test case generation into three steps: –Translating B into CLPS-B (custom CLP language); –Calculate boundary states for the application; –Generate the test case traces through the state space.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test generation from an Abstract Model Example (cont) In a boundary state one or more state variables assume a boundary value; Calculation of boundary values is done by partitioning the domain of state variables (by symbolic execution): Successive operations pose the following constraints on a given state variable x with enumerated domain [1..10]: x  {1,2,3,4,5} x ≠ 3 P-Domain {1,2,3} U {3} U {3,4,5} U {5,…,10} Boundary Values (extremes of the P-Domain’s subsets) {1,3,5,10}

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test generation from an Abstract Model Example (cont) Traces that represent interesting test cases are then generated as follows : –Calculate the preamble trace to the boundary state; –Calculate the “body invocation” step; –Calculate the identification traces. Initial State Boundary State 1 Boundary State 2 Boundary State 3 Final state of body 1.1 Final state of identification Final state of body 1.2 Final state of body 1.3 Final state of identification Final state of identification Preamble traces Body invocation traces Identification traces An instantiation phase of the concatenated traces is necessary to create the “real” tests; Until then only the path condition exists.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test generation from an Abstract Model Example 2 – AUTOFOCUS AUTOFOCUS is a CASE tool inspired from UML-RT (real time) and directed towards embedded systems; In this paradigm a system is defined as a hierarchic network of components, represented by state machines: a.1 a.2 a.3 a b.1 b.2 b c System Highest level component Lowest level component

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test generation from an Abstract Model Example 2 – AUTOFOCUS (cont) Framework from Pretschner et al considers three possible test coverage classes: –Functional: covers the specification’s execution scenarios; –Structural: e.g. test critical components of the system; –Stochastic: generate random state space traces. The coverage heuristics are annotated in the specification;

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test generation from an Abstract Model Example 2 – AUTOFOCUS (cont) AUTOFOCUS specification is translated into a CLP language for animation. Each transition of a bottom level component K is modeled into a formula of type: Interesting traces (according to coverage heuristics) are generated by symbolic execution over system inputs. Instantiation of symbolic traces is the last step… step K (  src  dst )  guard( ,  src ) and assgmt(o,  dst ) A transition of a non-bottom level component is a composition of the transitions of it’s sub-components.

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Code based test case generation Why take a detour from model based test case generation (title of the section)? In the text we present a framework from Khurshid et al that generates test cases from Java code. It uses: –A model checker (Java PathFinder) to overcome some difficulties in symbolic execution; –The symbolic execution technique to overcome some difficulties in model checking. We think an insight on the framework is enriching for the discussion…

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test generation from code Example Symbolic execution helps dealing with state space explosion by dealing with inputs as classes of values; Model checking deals automatically with loops, recursion or method invocation… Model Checker Logic Formulas State Space Witness Trace Counter-example Trace Formulas holdFormulas don’t hold

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Test generation from code Example (cont) The code is instrumented to deal with symbolic values and accumulate path conditions (via Java libraries): Java Code Instrumented Java code Test case traces Java Symbolic Execution libraries Model checking The algorithm that performs symbolic execution knows how to deal with complex dynamic structures (objects)! A phase of test case instantiation is required…

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions