LSR TOBIAS and some ASE challenges Y. Ledru P. Bontron, O. Maury, L. du Bousquet, M.L. Potet, C. Oriat, S. Beghdadi, H. Bouldjedri LSR/IMAG – Grenoble ASE Workshop at Irvine, June 2002

LSR Overview Two challenges of ASE TOBIAS How TOBIAS is confronted to these challenges

LSR Overview Two challenges of ASE TOBIAS How TOBIAS is confronted to these challenges

LSR Signature of ASE tools Formal or structured document Formal or structured document ASE tool Source code, tests, formal specifications, UML diagrams, XML documents

LSR 1st challenge: process documents Complexity of input/output documents Algorithmic complexity of processing: –Combinatorial explosion –Undecidable problems Formal or structured document Formal or structured document ASE tool Challenge 1 : Process documents Domain specific tools

LSR 2nd challenge: produce documents Software engineers only like to write code! [XP] Millions of amateur software engineers… … how can we learn them category theory? Formal or structured document Formal or structured document ASE tool Capture information Challenge 1 : Process documents Challenge 2 : Produce documents

LSR Elements of solutions Don’t underestimate software engineers: provide useful/efficient/scalable tools! Provide easier formalisms (graphical formalisms, GUI tools,…) Integrate your tools in the process of the company! Formal or structured document Formal or structured document ASE tool Capture information Challenge 1 : Process documents Challenge 2 : Produce documents costs vs benefits

LSR Overview Two challenges of ASE TOBIAS How TOBIAS is confronted to these challenges

LSR TOBIAS Generates a large number of tests… … from a generative pattern (test schema) Should be used in combination with tools that provide the test oracle (VDMTools, TGV). Test schemas Test data/ Test cases/ Test purposes TOBIAS Capture information Class signatures

LSR A simple example An integer set class with two operations « manual » tests: –Add(0) –Add(0),Add(1),Add(2) –Add(0),Add(0) –Add(0),Remove(0) –Add(0),Add(1),Remove(0) –Add(0),Remove(1) Add(v:int) Remove(v:int) IntegerSet

LSR Add(0) Add(1) Add(2) Add(3) Add(x)^1..3;Remove(y)^0..2 Where x : {0,1,2,3} y : {0,1,2} Add(0),Add(0) Add(2),Add(0) Add(0),Add(1) Add(2),Add(1) Add(0),Add(2) Add(2),Add(2) Add(0),Add(3) Add(2),Add(3) Add(1),Add(0) Add(3),Add(0) Add(1),Add(1) Add(3),Add(1) Add(1),Add(2) Add(3),Add(2) Add(1),Add(3) Add(3),Add(3) A TOBIAS test schema … Add(0),Add(1),Add(2) … Add(2),Add(1),Add(0) … Add(0),Remove(0) Add(0),Remove(1) … Add(0),Add(1),Remove(0) … Add(x1) Add(x1),Add(x2) Add(x1),Add(x2),Add(x3) Add(x1),Remove(y1) Add(x1),Add(x2),Remove(y1) Add(x1),Add(x2),Add(x3), Remove(y1) Add(x1),Remove(y1),Remove(y2) Add(x1),Add(x2),Remove(y1),Remove(y2) Add(x1),Add(x2),Add(x3), Remove(y1),Remove(y2) (4+4*4+4*4*4)*(1+3+3*3) = 1092 sequences

LSR Results of TOBIAS TOBIAS generates the manual tests + lots of similar tests for the same design cost A recent experiment: –45 manual tests –4320 TOBIAS tests –same test design effort –2 new errors detected  TOBIAS helps detecting more errors!

LSR Overview Two challenges of ASE TOBIAS How TOBIAS is confronted to these challenges

LSR 1st challenge complexity Intrinsic to the tool: –generates a large number of tests –because IUTs are complex Still too many tests are useless: –Require untractable resources –May not detect more errors  Keep the number of tests small enough: Avoid redundant test cases  Extend the test schemas (constraints, hypotheses) Remove non-conform test cases  Verify vs specification

LSR Test schemas TOBIAS and complexity Test data/ Test cases/ Test purposes TOBIAS Capture information Class signatures Solutions to the first challenge… … make the second one more complex! Extended Test schemas Specification (UML diagrams, Pre/post cond., …)

LSR Test schemas are simple and short ! TOBIAS provides a GUI for test schemas Specifications can be used in conjunction with generated tests (oracle, TGV,…) Specifications are not needed to generate the first set of tests! Class signatures can be extracted from existing documents Capturing knowledge for TOBIAS Test schemas Test data/ Test cases/ Test purposes TOBIAS Capture information Class signatures Specification (UML diagrams, Pre/post cond., …) UML class diag. Or Java Program extract

LSR Conclusion TOBIAS is confronted to both ASE challenges –complexity –information acquisition Complexity addressed by additional knowledge Information acquired progressively: –Provide first benefits at low cost –Motivate efforts for specification –Multiple use of specifications (oracle, TGV) –Fits into a classical process

LSR THANKS !

LSR Redundant test cases The implementation of the integer set may be independent from the actual values of integers Lots of tests generated by TOBIAS can be considered redundant This corresponds to the expression of test hypotheses. Add(0) Add(1) Add(2) Add(3) … Add(0),Add(1),Add(2) … Add(2),Add(1),Add(0) … Add(0),Add(0) Add(2),Add(0) Add(0),Add(1) Add(2),Add(1) Add(0),Add(2) Add(2),Add(2) Add(0),Add(3) Add(2),Add(3) Add(1),Add(0) Add(3),Add(0) Add(1),Add(1) Add(3),Add(1) Add(1),Add(2) Add(3),Add(2) Add(1),Add(3) Add(3),Add(3)

LSR Add(x)^1..3;Remove(y)^0..2 Where x : {0,1,2,3} y : {0,1,2} Forall x_i,x_j. i x_i<=x_j Add(0),Add(0) Add(2),Add(0) Add(0),Add(1) Add(2),Add(1) Add(0),Add(2) Add(2),Add(2) Add(0),Add(3) Add(2),Add(3) Add(1),Add(0) Add(3),Add(0) Add(1),Add(1) Add(3),Add(1) Add(1),Add(2) Add(3),Add(2) Add(1),Add(3) Add(3),Add(3) … Add(0),Add(1),Add(2) … Add(2),Add(1),Add(0) …

LSR Overview Context: the COTE project (COmponent TEsting) Why TOBIAS ? TOBIAS: description of the tool –Test purpose generation –Test case generation with VDM –Smarter generation

LSR Overview Context: the COTE project (COmponent TEsting) Why TOBIAS ? TOBIAS: description of the tool –Test purpose generation –Test case generation with VDM –Smarter generation

LSR The COTE project A RNTL Project Partners –SOFTEAM (tool objecteering) leader –IRISA (tools UMLAUT/TGV) –GEMPLUS (smart cards/software components) –FT R&D (OO technologies/components) –LSR-IMAG (Software engineering)

LSR The context of COTE Conformance testing of software components –The component is tested against its UML specification –Black box testing: the internal behaviour of the product under test is hidden

LSR Aim of the COTE project corbaejb.net UML Test cases Automatic generation of executable tests UML sequence diagrams provide an abstraction for tests cases provide tools for automatic executable test generation from UML test cases target three component technologies

LSR Aim of the COTE project Test synthesis Assist the user in the specification of tests UML specification of tests corbaejb.net UML Test cases Automatic generation of executable tests test purpose

LSR Test case / Test purpose An example : A coffee vending machine A test purpose a sequence of transitions to reach A test case a complete path in the behavioural specification

LSR General structure of COTE UML Test purposes Behaviour computation UMLAUT/TGV UML Test cases corbaejb.net UML Specification Test synthesis Automatic tests generation

LSR Overview Context: the COTE project (COmponent TEsting) Why TOBIAS ? TOBIAS: description of the tool –Test purpose generation –Test case generation with VDM –Smarter generation

LSR Why TOBIAS ? Behavioural testing requires the specification of many tests. –Test campaign => (potentially) thousands of test cases –UMLAUT/TGV: one test purpose and one execution = one test case –New values => creation of new test purposes Hugues MARTIN’s doctoral thesis (GEMPLUS): « Une méthodologie de génération automatique de suites de tests pour applets java-card. » A methodology for automatic generation of test suites for java-card applets.

LSR Generation of test purposes Idea : notion of high level test purpose (test pattern) => abstraction of test purposes on objects, methods or values… A test pattern => a set of test purposes => a set of test cases  Significantly reduces the repetitive tasks of test definitions

LSR TOBIAS UML Specification UML Test purposes Behaviour computation UMLAUT/TGV Test synthesis UML Test cases TOBIAS Test pattern

LSR Overview Context: the COTE project (COmponent TEsting) Why TOBIAS ? TOBIAS: description of the tool –Test purpose generation –Test case generation with VDM –Smarter generation

LSR Principles of TOBIAS TOBIAS provides: –help for test purposes design –test pattern descriptions Idea: –instantiation of test patterns according to different criteria (objects, parameters …)

LSR TOBIAS inputs test patterns UML specification class diagram deployment diagram Test purpose in UML TOBIAS

LSR A little example (1) UML class diagram Actor IAdministratorIUser HCI * VendingMachine AdditionnalDrink Cancel Button NumButton : integer PushButton() insertCoin(Coin p) GiveChange()

LSR A little example (2) UML deployment diagram (one for each test campaign) TestConfiguration: DrinkMachine:VendingMachine Testor:Actor Front:HCI Coffee:Drink Tea:Drink Cappucino:Drink Chocolate:Drink Reset:Cancel

LSR Some test purposes For: - 4 different drinks - 1 cancel touch (Reset.GiveChange) - 5 different coins = {0.1, 0.2, 0.5, 1, 2} The number of different test purposes is around: (5+5²) x (4 + 1) = 150 test purposes Coffee.ClickButton() Chocolate.ClickButton() Tea.ClickButton() Cappucino.ClickButton() Reset.GiveChange() insertCoin(0.2€); insertCoin(0.1€); … 1. insertCoin(2€); 2. insertCoin(0.5€); 3. insertCoin(1€); 4. insertCoin(0.1€); 5. insertCoin(0.2€); 5 test purposes :

LSR Coffee.ClickButton() Chocolate.ClickButton() Tea.ClickButton() Cappucino.ClickButton() Reset.GiveChange() insertCoin(0.2€); insertCoin(0.1€); … 1. insertCoin(2€); 2. insertCoin(0.5€); 3. insertCoin(1€); 4. insertCoin(0.1€); 5. insertCoin(0.2€); Principle of test patterns Provide the generation of test purposes according to different criteria: –the method parameters –the objects –the groups –the number of group or method calls

LSR Principle of test patterns Provide the generation of test purposes according to different criteria: –the method parameters –the objects –the groups –the number of group or method calls Coffee.ClickButton() Chocolate.ClickButton() Tea.ClickButton() Cappucino.ClickButton() Reset.GiveChange() insertCoin(0.2€); insertCoin(0.1€); … 1. insertCoin(2€); 2. insertCoin(0.5€); 3. insertCoin(1€); 4. insertCoin(0.1€); 5. insertCoin(0.2€);

LSR Coffee.ClickButton() Chocolate.ClickButton() Tea.ClickButton() Cappucino.ClickButton() Reset.GiveChange() insertCoin(0.2€); insertCoin(0.1€); … 1. insertCoin(2€); 2. insertCoin(0.5€); 3. insertCoin(1€); 4. insertCoin(0.1€); 5. insertCoin(0.2€); Principle of test patterns Provide the generation of test purposes according to different criteria: –the method parameters –the objects –the groups –the number of group or method calls

LSR Principle of test patterns Provide the generation of test purposes according to different criteria: –the method parameters –the objects –the groups –the number of group or method calls Coffee.ClickButton() Chocolate.ClickButton() Tea.ClickButton() Cappucino.ClickButton() Reset.GiveChange() insertCoin(0.2€); insertCoin(0.1€); … 1. insertCoin(2€); 2. insertCoin(0.5€); 3. insertCoin(1€); 4. insertCoin(0.1€); 5. insertCoin(0.2€);

LSR Coffee.ClickButton() Chocolate.ClickButton() Tea.ClickButton() Cappucino.ClickButton() Reset.GiveChange() insertCoin(0.2€); insertCoin(0.1€); … 1. insertCoin(2€); 2. insertCoin(0.5€); 3. insertCoin(1€); 4. insertCoin(0.1€); 5. insertCoin(0.2€); Principle of test patterns Provide the generation of test purposes according to different criteria: –the method parameters –the objects –the groups –the number of group or method calls

LSR An example GroupAction = { Drink.ClickButton(), Cancel.GiveChange()} Using the following test pattern: insertCoin(x) ^1..2 ; GroupAction TOBIAS generates the 150 test purposes previously showed

LSR Advantages of using TOBIAS TOBIAS is useful to generate many test purposes  generating many test cases using UMLAUT/TGV Generating test purposes for new values is easier TOBIAS can be used to generate test suites for VDM in combination with VDMTools

LSR Using TOBIAS with VDM VDMtools provide an environment to specify an application, to execute test suites and to evaluate the test results The output of TOBIAS is directly used as a test suite. Using TOBIAS for the automatic generation of VDM test cases - VDM Workshop 2002 at Floc02

LSR Using TOBIAS with VDM TOBIAS VDMTools UML specification Test pattern VDM test suites Results: fault detection code coverage VDM specification

LSR Using TOBIAS with VDM We did the following experiment: –specify an application: managing groups of students –manually specify a test suite (about 15 test cases) –generate a test suite using TOBIAS (about 4300 test cases)

LSR Using TOBIAS with VDM We obtained the following results: –the generated and the manually written test suite have the same code coverage –The development effort is similar –the generated test suite detect more errors one unknown error one known error (from reading the specification) new activations of a known error –the generated test suite takes significantly more time to be executed: TOBIAS generates non conforming test cases TOBIAS generates redundant test cases

LSR Smarter generation Preventing non conforming test purposes / test cases –using more elements from the specification relations between classes pre/post conditions state diagrams Reducing the test suite size –eliminating redundant test purposes / test cases –defining constraints on the schemas

LSR Using more elements TOBIAS extract classes from the UML class diagram. Class A Class BClass D Class C A!C  C!A  B!C  C!B  A!B  B!A  A!D  D!A  B!D  D!B  C!D  D!C  A!A  B!B  C!C  D!D  the class A calls the methods of the class C

LSR Using more elements Relations between classes A!C  C!A  B!C x C!B x A!B x B!A x Class A Class BClass D Class C A!D x D!A  B!D  D!B  C!D x D!C x A!A x B!B x C!C x D!D x

LSR Using more elements Deployment diagram that gives a fixed state of the application a:A b:Bd:D c:C A!C  a!c  C!A  c!a  B!C x C!B x A!B x B!A x A!D x D!A  d!a x B!D  b!d  D!B  d!b  C!D x D!C x A!A x B!B x C!C x D!D x Deployment diagram

LSR Using more elements Deployment diagram that gives the initial state of the application a:A b:Bd:D c:C A!C  a!c  C!A  c!a  B!C x C!B x A!B x B!A x A!D x D!A  d!a  B!D  b!d  D!B  d!b  C!D x D!C x A!A x B!B x C!C x D!D x

LSR Adding constraints The user can associate a constraint to each test pattern Constraints deal with parameters Constraints are used to filter test purposes / test cases

LSR Adding constraints A test pattern: insertCoin(x) ^1..2 ; GroupAction GroupAction = { Drink.ClickButton(), Cancel.GiveChange()} A constraint: Number of selected test purposes : (5+9)x(4+1) = 70 test purposes - insertCoin(x_1) ; GroupAction - insertCoin(x_1) ; insertCoin(x_2) ; GroupAction 

LSR Conclusion TOBIAS provides a new abstract level for test design: the test patterns A test pattern => a set of similar test cases To avoid combinatorial explosion, the user can associate constraints to the schemas TOBIAS can be used to generate test sequences TOBIAS will be used and validated on an industrial test case in the COTE project

LSR Prospects Refining the test generation –Deleting redundant –Using pre/post conditions on operation to eliminate non-conforming sequences Helping the decision to stop test campaign: –Specification coverage by test patterns  using more elements from the specification