1.  Yves Le Traon 2003 OO System Testing Behavioral test patterns Automatic test synthesis from UML models

2.  Yves Le Traon 2003 Outline  System testing  Behavioral test patterns  Generating behavioral test patterns

3.  Yves Le Traon 2003 Testing product lines  Benefiting from the PL specificities  Testing commonalities  Deriving tests according to the variants  Specific tests  Reusing tests Building test assets  Defining test independently from the products  Using generic scenarios  Deriving product-specific test cases from those generic scenarios

4.  Yves Le Traon 2003 Test système et UML  Meeting distribué

5.  Yves Le Traon 2003 The use case scenarios  High level, simple, incomplete  Wildcards for genericity  Example: Enter use case scenario (x is a scenario parameter) (b) :Server x:user enter(*, x) ok (a) Nominal case (b) Exceptional case :Server x:user enter(*, x) nok

6.  Yves Le Traon 2003 Test système et UML Cas d’utilisationScénarios Nominaux Scénarios Exc. Rares Scénarios Exc. Echecs A PlanifierN A1, N A2 E A1, E A2 B OuvrirN B1 E B1, E B2 I ClôturerN I1 R I1 C ConsulterN C1 E C1 D EntrerN C1 R D1 E D1, E D2 E Demander la Parole N E1 E E1 G ParlerN G1, N G2 R G1 E G1, E G2 H SortirN H1 E H1 F Donner la ParoleN F1 E F1, E F2

7.  Yves Le Traon 2003 Test système et UML  Critère minimum: Couvrir chaque scénario avec une donnée de test  Ici 27 cas de test  Critère de couverture des combinaisons de use-cases  Prérequis : un diagramme d’activité des use-cases

8.  Yves Le Traon 2003 Activity diagrams  A swimlane per actor  Visually significant  Within UML notations  Suitable to apply algorithms  Difficult to build  Hard or impossible to express certain behaviors  Not suitable for use cases shared by actors

9.  Yves Le Traon 2003 Test système et UML

10.  Yves Le Traon 2003 Test système et UML  Critère : tester chaque scénario de chaque cas d’utilisation dans chaque séquence nominale élémentaire (1 passage par boucle)

11.  Yves Le Traon 2003 Test système et UML  Données de tests à générer pour la séquence A.B.I.H => 4 + 2x3 + 2x1x2 + 2x1x2x2 = 22 cas de test doivent être générés via ce « pattern » Cible de Test A B I H Combinaison des Cas de Test             2 1 2 1 A A A A E E N N       2 1 A A N N            2 1 1 B B B E E N       2 1 A A N N   1 B N        1 1 I I R N       2 1 A A N N   1 B N        1 1 I I R N        1 1 H H E N

12.  Yves Le Traon 2003 Test système et UML  En évitant les redondances: ordonner les « cibles de test » => 10 cas de test doivent être générés via ce « pattern » Cible de Test H I B A Combinaison des Cas de Test       2 1 A A N N   1 B N  N I1        1 1 H H E N       2 1 A A N N   1 B N  R       2 1 A A N N   1 B E       2 1 A A E E

13.  Yves Le Traon 2003 Behavioral Test Patterns  Based on the use case scenarios  high level  generic (use of wildcards)  incomplete  nominal or exceptional  A selection from among the scenarios :  An accept scenario (  test objective)  Reject scenarios (optional)  Prefix scenarios (  initialisation, optional)

14.  Yves Le Traon 2003 Benefits from test patterns  Generation of product specific test cases from product independant test patterns  But tedious to build test patterns especially for « basis tests »  Idea : being able to build automatically significant sets of test patterns

15.  Yves Le Traon 2003 How to exploit use cases ordering ?  Generate pertinent paths of use cases  In order to reach a test criterion  Issues:  An algorithm to assemble the use cases taking into account the pre and post conditions  Defining pertinent test criterions

16.  Yves Le Traon 2003 Conclusion  From early modeling to test cases :  From reusable and generic test pattern  To concrete test cases, specific to each product  Two ways of selecting test patterns:  manually (qualitative approach)  driven by use cases sequential dependencies (quantitative approach)

17.  Yves Le Traon 2003 From system level test patterns to specific test cases : application to product-Line architectures

18.  Yves Le Traon 2003 Product Line architectures  A product line : a set of systems which share a common software architecture and a set of reusable components.  Building a product line aims at developing once the common core of a set of products, and to reuse it for all the products.  Defining a product family  Variants and commonalities  Reuse assets  For our purpose: specify behavioural test patterns, that become reusable “test assets” of the product-line

19.  Yves Le Traon 2003 Product Line architectures: a key challenge  Use case scenarios cannot be used directly for testing  Generic and incomplete.  Parameters are not known, nor object instances (scenarios concern roles).  Specify the general system functionality without knowing – at that stage - the exact sequence calls/answers.  Generating test cases from such test patterns for a given UML specification is thus one of the key challenges in software testing today.

20.  Yves Le Traon 2003 PL  Variants  optional, when a component can be present or not,  alternative, when at a variation point, one and only one component can be chosen among a set of components,  multiple, when at a variation point, several components can be chosen among a set of components.  All the variants must appear in the architecture but not all the possible combination of variants  Extracting a product from the global product line architecture : product instantiation

21.  Yves Le Traon 2003 Product Line architectures: example  Virtual Meeting Server PL offers simplified web conference services:  it aims at permitting several kinds of work meetings, on a distributed platform. ( general case of a ‘chat’ software).  When connected to the server, a client can enter or exit a meeting, speak, or plan new meetings.  Three types of meetings  standard meetings where the client who has the floor is designated by a moderator (nominated by the organizer of the meeting)  democratic meetings which are standard meetings where the moderator is a FIFO robot (the first client to ask for permission to speak is the first to speak)  private meetings which are standard meetings with access limited to a defined set of clients.

22.  Yves Le Traon 2003 The Virtual Meeting Example  Connection to the server  Planning of meetings  Participation in meetings  Moderation of meetings VirtualMtg enter plan open close consult leave hand over speak moderator manageruser connect Virtual meeting use case diagram

23.  Yves Le Traon 2003 Product Line architectures: example  Due to marketing constraints, the Virtual Meeting PL is derivable into three products  a demonstration edition: standard and limited  a personal edition: any type but limited  an enterprise edition: any type, no limitations  Two variants : type (multiple) and participants limitation (optional )  (also OS, languages, interfaces etc.)

24.  Yves Le Traon 2003 The Virtual Meeting Example  Two main variants:  the kinds of meetings available  the limitation of the number of participants  Three products:  Demonstration edition  Personal edition  Enterprise edition Virtual Meeting Variant 1 {multiple}: available meetings Variant 2 {optional}: meetings limitation Demonstration edition Standardtrue Personal edition Standard, private, democratic true Enterprise edition Standard, private, democratic false

25.  Yves Le Traon 2003 Testing product lines  Benefiting from the PL specificities  Testing commonalities  Deriving tests according to the variants  Specific tests  Reusing tests Building test assets  Defining test independently from the products  Using generic scenarios  Deriving product-specific test cases from those generic scenarios

26.  Yves Le Traon 2003 A contradiction  Test scenarios must be expressed at a very high level  to be reusable  to be independent from the variants and the products  Generic scenarios are too vague and incomplete  cannot be directly used on a specific product  Impossible to reuse generic test scenarios ?

27.  Yves Le Traon 2003 Behavioral Test Patterns  Based on the use case scenarios  high level  generic  product independent  nominal or exceptional  A selection from among the scenarios :  An accept scenario  Reject scenarios  Prefix scenarios

28.  Yves Le Traon 2003 Testing a PL  Behavioral Test Patterns (or Test Objective)  an accept scenario: it expresses the behavior that has to be tested, e.g. the successful exit (“leave” a meeting use case) of a participant from a meeting,  one or several (optional) reject scenarios: they express the behaviors that are not significant for the tester, e.g. the consult function of a meeting state does not interact with the entering into a meeting.  one or several (optional) preamble (or prefix) scenarios that must precede the accept scenario. For example, a meeting must be opened before any participant can enter the virtual meeting.

29.  Yves Le Traon 2003 An Example S- Prefix S+ :Server x:user enter(*, x) nok x:user :Server connect(x) ok plan(*, x) ok open(*, x) ok :Server y:user close(*, y) :Server leave(*, y) y:user

30.  Yves Le Traon 2003 The reject scenarios  Optional  Reduce the « noise »  Avoid calls irrelevant for the test  Exclude interfering calls

31.  Yves Le Traon 2003  Describes the preamble part of the test case  Guides the synthesis  A composition of use-case scenarios  Scenarios versus object diagram ? The prefix Prefix x:user :Server connect(x) ok plan(*, x) ok open(*, x) ok user2:user user3:user user4:user server:demoServer user1:user Object diagram

32.  Yves Le Traon 2003 Typical reject scenarios  Some scenarios can be added automatically  Use of a boolean dependency matrix PlanOpenCloseConsultEnterSpeakLeave PlanXX OpenXX CloseXX ConsultXX EnterXXXX SpeakXXX LeaveXX Scenarios independent from the enter use case : added as reject scenarios

33.  Yves Le Traon 2003 Typical reject / prefix scenarios  Use of the activity diagram  Accept scenario = the targeted scenario in a use cas  Prefix = the previous scenarios in the path  Reject = all the scenarios of the use cases that are not involved in the path.

34.  Yves Le Traon 2003 Generating test patterns Product instanciation Detailed Design General Design Main classes Interfaces… P1 P2 P3 TP1 TP2 Test cases synthesis Use cases UC1 UC2 nominalexceptional nominalexceptional Evolution Test patterns specification (test objective) Accept scenario Reject scenarios (optional) Prefix scenarios (optional) selection manual automated or

35.  Yves Le Traon 2003

36. Compiling the Test Pattern  Inputs venant d’UML:  Le diagramme de classes détaillé avec – autant que possible – un statechart par classe active du système  Un diagramme d’objets initial  Le pattern de test  Les aspects dynamiques sont fournis à TGV sous forme d’API  Output :  Un scénario détaillé UML décrivant tous les appels précis et les verdicts attendus à effectuer sur le système pour observer le comportement spécifié dans le pattern

37.  Yves Le Traon 2003 Compiling the Test Pattern  accept+ = sequential composition of the prefix and the accept scenario  Scenarios making up the test case =  accepted by accept+  rejected by none of the reject scenarios  accept+  LTS S +  reject scenarios {seq j - } j  J  LTS {S j - } j  J  Test pattern  LTS S +    j  J S j -

38.  Yves Le Traon 2003 Synthesis of the test case  Inputs of TGV:  Simulation API  LTS representing the Test Pattern  Which actions are internal ?  Which actions are inputs ? outputs ?  Output of TGV: IOLTS representing a test case  UML test case derivation

39.  Yves Le Traon 2003 Product Line architectures: example (a) non-limited meetings (b) limited meetings

40.  Yves Le Traon 2003 An Example S- Prefix S+ :Server x:user enter(*, x) nok x:user :Server connect(x) ok plan(*, x) ok open(*, x) ok :Server y:user close(*, y) :Server leave(*, y) y:user

41.  Yves Le Traon 2003 Test patterns and test cases user2:useruser3:useruser4:user server:demoServer user1:user enter(aMtg, user2) ok enter(aMtg, user3) ok enter(aMtg, user4) nok enter(aMtg, user1) ok connect(user1) ok plan(aMtg, user1) ok open(aMtg, user1) ok Preamble Test Objective

42.  Yves Le Traon 2003 Conclusion  From early modeling to test cases :  From reusable and generic test pattern  To concrete test cases, specific to each product  Methodology « not fully automated » …