Towards Automatic Generation of Parameterized Test Cases from Abstractions Jens R. Calamé Natalia Ioustinova Jaco van de Pol Centrum voor Wiskunde en Informatica, Amsterdam

IPA Lentedagen Agenda 1.Testing Theory 2.Data Abstraction 3.Test Case Parameterization 4.Some Lemmas 5.Conclusion

IPA Lentedagen Testing Theory Conformance Testing: –Implementation Imp conforms specification Spec, iff for all traces t in Spec holds: all input should be accepted after t, but at most the specified output should be generated by Imp. –Based on Tretmans‘ ioco theory Test Generation by the tool TGV (Test Generation with Verification Techniques) –Based on state enumeration –Limitation: Data often leads to state-space explosion –Solution: Data abstraction mitigates this limitation

IPA Lentedagen Our Test Generation Process Abstract System Specification System Specification (formal) Test Purpose TGV System Specification (e.g. UML) Data Abstraction Abstract Test Case Test Case Generation Parameterizable Test Case (TTCN-3) Rule System for Data Selection TTCN-3 Generation Constraint Generation

IPA Lentedagen Data Abstraction Motivation: Input and output data from large (infinite) domains leads to state explosion  Application of state-based test generation tools impossible Solution (from model-checking): data abstraction –Introduction of a chaotic value ╥ for each datatype D –Lifting functions for original values from D –Replacement of input variables by ╥ and propagation through the system –Introduction of may functions for guards over three- value-logic (semantics: may( ╥ ) = true) –Consequence: Introduction of extra traces by non-determinism

IPA Lentedagen Data Abstraction – Example ?getPin(x) ?getBalance !Balance(b) 6 7 ?getAmount(y) ? (y) 8 01 ?initPin(p)?initBalance(b) !pinIncorrect  )(px  !pinCorrect  )(px  !Money(y); b:=b-y  )(by  !LowBalance(b)  )(by 

IPA Lentedagen Data Abstraction – Test Case by TGV

IPA Lentedagen Test Case Parameterization To be solved after data abstraction: –Pruning of traces introduced by overapproximation –Finding possible value ranges for test data Rule System (Prolog) –Represents SUT specification –Defines rules on data (addition, substraction; and, or etc.) –Defines rules for process behavior of Spec Query (Prolog) –Represents the abstract test case (one rule per trace) –All transitions in trace taken as query body –Alternative: one query for whole CTG

IPA Lentedagen Testing with Abstraction – Rule System … getPin(state(2,P,B,X,Y), state(3,P,B,X1,Y), param(X1)). pinIncorrect(state(3,P,B,X,Y), state(8,P,B,X,Y),param(_)) :- X ≠ P. pinCorrect(state(3,P,B,X,Y), state(4,P,B,X,Y), param(_)) :- X = P. … ?getPin(x) ?getBalance !Balance(b) 6 7 ?getAmount(y) ? (y) 8 01 ?initPin(p)?initBalance(b) !pinIncorrect  )(px  !pinCorrect  )(px  !Money(y); b:=b-y  )(by  !LowBalance(b)  )(by 

IPA Lentedagen Testing with Abstraction – Query oracle(P,B,X,Yin,Yout) :- initPin(state(0,0,0,0,0), G1, param(P)), initBalance(G1, G2, param(B)), getPin(G2, G3, param(X)), pinCorrect(G3, G4, _), getAmount(G4, G5, param(Yin)), money(G5, _, param(Yout)). P=P{-1.0inf..1.0inf} B=B{-1.0inf..1.0inf} X=P{-1.0inf..1.0inf} Yin=Yout{-1.0inf..1.0inf} Yout=Yout{-1.0inf..1.0inf} Yout{-1.0inf..1.0inf}-B{-1.0inf..1.0inf}=<0

IPA Lentedagen Test Execution 1.Pre-solve one trace to pass statically (skip internal steps) 2.Execute this trace until the SUT leaves it 3.Try to find a trace to pass  solve and execute it 4.If no trace to pass: Try to find a trace to inconc  solve it 5.If no trace to inconc: set test verdict fail PASS 01 INCONC !initPin(P)!initBalance(B) !getPin(X) ?pinCorrect ?pinInCorrect ?lowBalance !getAmount(Yin)!getBalance ?Balance(x50) !getAmount(x60) ?Money(Yout) ?eatCard

IPA Lentedagen Some Lemmas The original system Spec is simulated by Spec ╥ in all details (and some more). The control flow of the synchronous product Spec x TP is simulated by Spec ╥ x TP. The set of accepting/refusing traces of Spec x TP is a subset of those of Spec ╥ x TP. If the test oracle holds for a trace in the CTG under a certain valuation, this trace is valid in the original system under the given valuation. The test algorithm terminates with a sound verdict. For further details and proofs see the technical report.

IPA Lentedagen Case Study CEPS Common Electronic Purse Specifications: protocol for electronic payment using a multi-currency smart-card Input and output parameters of card actions mainly natural numbers  (in)finite domain Variables partially arrays (up to 16 elements in simplified µCRL realization) As µCRL specification: –54 summands –44 process variables (netto) –207process variables (brutto), due to arrays of structures

IPA Lentedagen Case Study CEPS (cont'd) Instantiation and reduction (abstracted specification): ca. 16 min. on five 2.2GHz Athlon 64bit single CPU computers (1 GB RAM each) Generation of two test cases: 594 and 109 states, resp. in less than one second (one 2.2GHz AMD Athlon XP 32 bit CPU and 1 GB RAM) Constraint solving produces results in negligable time

IPA Lentedagen Conclusion Data abstraction makes state-based test generation applicable to systems with large data domains Approach successfully evaluated on CEPS case study Ongoing and future work: –Redesign of tools (add "real" constraint-solving) –On-the-fly constraint-solving –Treatment of  -steps –(Generation and) execution of TTCN-3 test cases –Integration of UML as specification language

IPA Lentedagen Related Links TT-Medal Project: Test generator TGV: –www-verimag.imag.fr/~async/TGVwww-verimag.imag.fr/~async/TGV – Data abstraction tools: Technical report and papers: Jens R. Calamé Natalia Ioustinova Jaco van de Pol