Presentation is loading. Please wait.

Presentation is loading. Please wait.

LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,

Similar presentations


Presentation on theme: "LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,"— Presentation transcript:

1 LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble, France ASE2001, Nov 27-29 2001, San Diego

2 LSR Conformance testing Given a black box And its specification Test cases compare the black box to its specification X = sqrt(X)*sqrt(X) and sqrt(X) >= 0 ?sqrt(4) !2 accept sqrt

3 LSR Complex test cases Unfortunately, complex applications often result in complex test cases! ?Money_Transfer(1000000€) !OK accept e-Bank Money_transfer

4 LSR Specifying test cases It then makes sense to specify these complex test cases « I want a test case that will execute the Money_transfer procedure successfully! » It is the notion of test purpose! ?Money_Transfer(1000000€) !OK accept

5 LSR The COTE project COTE: –conformance testing of components vs UML specifications –Softeam, France Telecom, Gemplus, IRISA, LSR/IMAG –Languages and tools to express and exploit test purposes and test cases –(compilers from test cases to target technologies) This work comes from a brainstorming activity around the notion of test purpose within the COTE project

6 LSR The notion of test purpose Test purpose : Description of a precise goal of the test case, in terms of exercising a particular execution path or verifying the compliance with a specific requirement [ISO IEC 9646-1 …conformance testing methodology and framework…] This notion is used: –As a structuring element in industrial test suites –In some tools based on formal specifications (SAMSTAG, TGV, TorX)

7 LSR Test purposes in TGV TGV is a tool developed jointly by IRISA (Rennes) and Vérimag (Grenoble) Test synthesis tool Given –A specification of the system under test –A test purpose It generates one corresponding test case.

8 LSR Example : coffee machine ?coin(1) ?coin(2) Pay 1 or 2 coins ?sugar Ask for sugar (optional) ?tea ?coffee Ask for tea or coffee !coffee !tea !sugar Get the beverage Loop

9 LSR TGV ?coin(1) ?coin(2) ?sugar ?tea ?coffee !coffee !tea !sugar Dynamic specification ?coin(1) ?coin(2) ?sugar ?tea ?coffee !coffee !tea !sugar A sample synthesis !coffee accept Test purpose ?coin(2) ?coffee !coffee Test case

10 LSR Test purposes as specification A test purpose (P) is an abstraction of test cases In TGV, a partial sequence of the test cases A test purpose specifies a set of test cases !coffee accept ?coin(2) ?coffee !coffee ?coin(1) ?sugar ?tea ?coffee !coffee ?coin(1) ?coin(2) ?sugar ?tea ?coffee !coffee !tea !sugar

11 LSR Benefits of abstraction Capture the essence of the test (here test the delivery of coffee) Shorter to state than the full test case (clerical completion left to the tool) Robust to evolutions of the dynamic specification (test cases are affected more often than test purposes, e.g. pay 3 instead of 2) !coffee accept

12 LSR Relations between test purposes (P), test cases (C) and specifications (S)

13 LSR Weakly_refines(C,P) A test case « refines » the test purpose (if it includes the partial sequence) But several irrelevant test cases refine the test purpose also. Þ Weak refinement! !coffee accept ?coin(2) ?coffee !coffee test case not conforming to the specification !coffee

14 LSR refines(C,P,S) The refinement takes place in the context of the specification !coffee accept ?coin(2) ?coffee !coffee ?coin(1) ?coin(2) ?sugar ?tea ?coffee !coffee !tea !sugar refines

15 LSR weakly_refines vs refines Refines is a ternary relation Weakly_refines is defined as weakly_refines(C,P) Û E S ¥ refines(C,P,S) refines weakly_refines Test case (C) Specification (S) Test Purpose (P)

16 LSR Semantics for refines(C,P,S) Depending on the languages used for C, P, and S, a semantics can be defined for refines(C,P,S) Semantics of refines in TGV: C is one of the paths of the synchronous product of P and S See also, in the paper: semantics of a test refinement relation in the context of the B method

17 LSR Semantics of weakly_refines As a consequence: weakly_refines(C,P) Û E S ¥ refines(C,P,S) gives a semantics to weakly_refines on top of the semantics of refines

18 LSR conforms(C,S) conforms(C,S) Û E P ¥ refines(C,P,S) Conforms checks that the test case corresponds to the specification refines Test case (C) Specification (S) Test Purpose (P) conforms

19 LSR consistent(P,S) consistent(P,S) Û E C ¥ refines(C,P,S) Consistent checks if the test purpose is compatible with the specification refines consistent Test case (C) Specification (S) Test Purpose (P)

20 LSR Summary If refines(C,P,S) has a semantics, it gives a semantics to the binary relations! In some cases, it is possible to systematically construct the third element of the relation refines weakly_refines consistent Test case (C) Specification (S) Test Purpose (P) conforms

21 LSR Test specification vs software specification

22 LSR refines SW Program (M) Software Specification (SS) refines Test case (C) Specification (S) Test Purpose (P)

23 LSR Automated Software Engineering Formal specs as a basis for software tools: –Verification of specification consistency –Program synthesis –Program verification

24 LSR Specification consistency often demonstrated by proving: E M ¥ refines SW (M,SS) (there exists a model which refines the specification) corresponds to consistent(P,S) Û E C ¥ refines(C,P,S) (a test purpose makes sense if there exists a test case) Tool Detection of inconsistent purposes (e.g. !sugar comes after !coffee) avoids useless searches !coffee accept reject ?coin(*) !sugar

25 LSR Synthesis Major theme at ASE synthesis SW (SS)  M corresponds to test synthesis e.g., TGV can be seen as a function synthesis TGV (P,S)  C Good news for tools The search domain is restricted by S (more decidable?, less complex?)

26 LSR Verification checks refines SW (M,SS) (program M is a valid refinement of specification SS) corresponds to 3 possible test case verifications –refines(C,P,S) C is a refinement of P in the context of S –weakly_refines(C,P) C is a refinement of P in some context –conforms(C,S) C is a test case for S

27 LSR Tools for test verification Synthesis involves searching a huge space of solutions If verification is cheaper than synthesis, it provides a natural complement. Two examples: –Evolution of specifications –Test case retrieval

28 LSR Evolution of specifications Evolution of S into S’ –Given a set of (Ci,Pi) for S –Verify refines(Ci,Pi,S’) for each (Ci,Pi) –When it fails, verify conforms(Ci,S’) –A second failure reveals regression of S’ Example: No need to recompute existing test cases if the coffee machine also serves hot chocolate ?tea ?coffee ?choc

29 LSR Test case retrieval Given a large set of test cases –Retrieve a test that weakly_refines a new test purpose –Minimize the set by identifying test cases that weakly_refine several test purposes !coffee accept ?coin(2) ?coffee !coffee weakly_refines ?coin(2) accept

30 LSR Related work and conclusion

31 LSR Test selection criteria 1st category: domain independent –Coverage criteria –Fault models (mutation testing) –Analysis rules, strategies –Random testing 2nd category: exploits domain knowledge –Test purposes (based on reqts) –Operational profiles Must be consistent with the reference Synthesis tool Test selection criterion (aka test requirement) Test case(s) Reference (specification, program)

32 LSR Conclusion Test purposes are a specification of test cases! 4 relations between C, P, and S –Not specific to a given tool or language –Not specific to an application domain (TGV: reactive systems, B: transf. systems) Tools: from ASE to Automated Test Eng. –Based on these 4 relations –Test verification complements test synthesis –Applications to specification evolution, test case retrieval,…

33 LSR Interactive Coffee Crafting


Download ppt "LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,"

Similar presentations


Ads by Google