LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble, France ASE2001, Nov , San Diego

LSR Conformance testing Given a black box And its specification Test cases compare the black box to its specification X = sqrt(X)*sqrt(X) and sqrt(X) >= 0 ?sqrt(4) !2 accept sqrt

LSR Complex test cases Unfortunately, complex applications often result in complex test cases! ?Money_Transfer( €) !OK accept e-Bank Money_transfer

LSR Specifying test cases It then makes sense to specify these complex test cases « I want a test case that will execute the Money_transfer procedure successfully! » It is the notion of test purpose! ?Money_Transfer( €) !OK accept

LSR The COTE project COTE: –conformance testing of components vs UML specifications –Softeam, France Telecom, Gemplus, IRISA, LSR/IMAG –Languages and tools to express and exploit test purposes and test cases –(compilers from test cases to target technologies) This work comes from a brainstorming activity around the notion of test purpose within the COTE project

LSR The notion of test purpose Test purpose : Description of a precise goal of the test case, in terms of exercising a particular execution path or verifying the compliance with a specific requirement [ISO IEC …conformance testing methodology and framework…] This notion is used: –As a structuring element in industrial test suites –In some tools based on formal specifications (SAMSTAG, TGV, TorX)

LSR Test purposes in TGV TGV is a tool developed jointly by IRISA (Rennes) and Vérimag (Grenoble) Test synthesis tool Given –A specification of the system under test –A test purpose It generates one corresponding test case.

LSR Example : coffee machine ?coin(1) ?coin(2) Pay 1 or 2 coins ?sugar Ask for sugar (optional) ?tea ?coffee Ask for tea or coffee !coffee !tea !sugar Get the beverage Loop

LSR TGV ?coin(1) ?coin(2) ?sugar ?tea ?coffee !coffee !tea !sugar Dynamic specification ?coin(1) ?coin(2) ?sugar ?tea ?coffee !coffee !tea !sugar A sample synthesis !coffee accept Test purpose ?coin(2) ?coffee !coffee Test case

LSR Test purposes as specification A test purpose (P) is an abstraction of test cases In TGV, a partial sequence of the test cases A test purpose specifies a set of test cases !coffee accept ?coin(2) ?coffee !coffee ?coin(1) ?sugar ?tea ?coffee !coffee ?coin(1) ?coin(2) ?sugar ?tea ?coffee !coffee !tea !sugar

LSR Benefits of abstraction Capture the essence of the test (here test the delivery of coffee) Shorter to state than the full test case (clerical completion left to the tool) Robust to evolutions of the dynamic specification (test cases are affected more often than test purposes, e.g. pay 3 instead of 2) !coffee accept

LSR Relations between test purposes (P), test cases (C) and specifications (S)

LSR Weakly_refines(C,P) A test case « refines » the test purpose (if it includes the partial sequence) But several irrelevant test cases refine the test purpose also. Þ Weak refinement! !coffee accept ?coin(2) ?coffee !coffee test case not conforming to the specification !coffee

LSR refines(C,P,S) The refinement takes place in the context of the specification !coffee accept ?coin(2) ?coffee !coffee ?coin(1) ?coin(2) ?sugar ?tea ?coffee !coffee !tea !sugar refines

LSR weakly_refines vs refines Refines is a ternary relation Weakly_refines is defined as weakly_refines(C,P) Û E S ¥ refines(C,P,S) refines weakly_refines Test case (C) Specification (S) Test Purpose (P)

LSR Semantics for refines(C,P,S) Depending on the languages used for C, P, and S, a semantics can be defined for refines(C,P,S) Semantics of refines in TGV: C is one of the paths of the synchronous product of P and S See also, in the paper: semantics of a test refinement relation in the context of the B method

LSR Semantics of weakly_refines As a consequence: weakly_refines(C,P) Û E S ¥ refines(C,P,S) gives a semantics to weakly_refines on top of the semantics of refines

LSR conforms(C,S) conforms(C,S) Û E P ¥ refines(C,P,S) Conforms checks that the test case corresponds to the specification refines Test case (C) Specification (S) Test Purpose (P) conforms

LSR consistent(P,S) consistent(P,S) Û E C ¥ refines(C,P,S) Consistent checks if the test purpose is compatible with the specification refines consistent Test case (C) Specification (S) Test Purpose (P)

LSR Summary If refines(C,P,S) has a semantics, it gives a semantics to the binary relations! In some cases, it is possible to systematically construct the third element of the relation refines weakly_refines consistent Test case (C) Specification (S) Test Purpose (P) conforms

LSR Test specification vs software specification

LSR refines SW Program (M) Software Specification (SS) refines Test case (C) Specification (S) Test Purpose (P)

LSR Automated Software Engineering Formal specs as a basis for software tools: –Verification of specification consistency –Program synthesis –Program verification

LSR Specification consistency often demonstrated by proving: E M ¥ refines SW (M,SS) (there exists a model which refines the specification) corresponds to consistent(P,S) Û E C ¥ refines(C,P,S) (a test purpose makes sense if there exists a test case) Tool Detection of inconsistent purposes (e.g. !sugar comes after !coffee) avoids useless searches !coffee accept reject ?coin(*) !sugar

LSR Synthesis Major theme at ASE synthesis SW (SS)  M corresponds to test synthesis e.g., TGV can be seen as a function synthesis TGV (P,S)  C Good news for tools The search domain is restricted by S (more decidable?, less complex?)

LSR Verification checks refines SW (M,SS) (program M is a valid refinement of specification SS) corresponds to 3 possible test case verifications –refines(C,P,S) C is a refinement of P in the context of S –weakly_refines(C,P) C is a refinement of P in some context –conforms(C,S) C is a test case for S

LSR Tools for test verification Synthesis involves searching a huge space of solutions If verification is cheaper than synthesis, it provides a natural complement. Two examples: –Evolution of specifications –Test case retrieval

LSR Evolution of specifications Evolution of S into S’ –Given a set of (Ci,Pi) for S –Verify refines(Ci,Pi,S’) for each (Ci,Pi) –When it fails, verify conforms(Ci,S’) –A second failure reveals regression of S’ Example: No need to recompute existing test cases if the coffee machine also serves hot chocolate ?tea ?coffee ?choc

LSR Test case retrieval Given a large set of test cases –Retrieve a test that weakly_refines a new test purpose –Minimize the set by identifying test cases that weakly_refine several test purposes !coffee accept ?coin(2) ?coffee !coffee weakly_refines ?coin(2) accept

LSR Related work and conclusion

LSR Test selection criteria 1st category: domain independent –Coverage criteria –Fault models (mutation testing) –Analysis rules, strategies –Random testing 2nd category: exploits domain knowledge –Test purposes (based on reqts) –Operational profiles Must be consistent with the reference Synthesis tool Test selection criterion (aka test requirement) Test case(s) Reference (specification, program)

LSR Conclusion Test purposes are a specification of test cases! 4 relations between C, P, and S –Not specific to a given tool or language –Not specific to an application domain (TGV: reactive systems, B: transf. systems) Tools: from ASE to Automated Test Eng. –Based on these 4 relations –Test verification complements test synthesis –Applications to specification evolution, test case retrieval,…

LSR Interactive Coffee Crafting