1 Benoit Baudry – 2000 : Masters degree at the Univ. de Rennes 1 in – june 2003 : PhD thesis, « Testable assembly and component validation » with Yves Le Traon and Jean-Marc Jézéquel in the Triskell group at the Univ. de Rennes 1 – Next : Postdoc position at the CEA-Saclay on MDA

2 Test in the Triskell group Triskell : Model Driven Engineering for Component Based Software ( UML-based OO testing (Yves Le Traon) : – Test order, integration strategies Vu Le Hanh. -- test et modèle UML : stratégie, plan et synthèse de test. -- PhD thesis – Test generation from the requirements (Clémentine Nebut) – UML-based test generation Alain Le Guennec. -- Génie Logiciel et Méthodes Formelles avec UML Spécification, Validation et Génération de tests. -- PhD thesis – PhD starting on MDA and testing (Franck Fleurey)

Testable assembly and component validation Benoit Baudry Triskell group, IRISA, Rennes, France Jean-Marc Jézéquel Yves Le Traon

4 Complex software systems are built with components as the unit for reuse Trustable components Techniques to assemble components

5 Trustable component Specification Implementation V & V : Test cases set Trust based on consistency executables contracts

6 Components assembly Impact of design by contract Impact of coupling on a testability factor

7 Context Software components Object oriented design and analysis Specific structures in OO programs Need to adapt testing techniques R. V. Binder, "Testing Object-Oriented Systems: Models, Patterns and Tools". Addison-Wesley 1999.

8 Summary Related work – Software testing – Mutation analysis Automatic test cases generation Design by Contract and testing Component testability

9 Software testing Two types : – Structural based on implementation – Functional based on specification of functionalities Objectives – Examine or execute a program searching for errors – Possibly : robustness performances safety properties

10 Le test de logiciel Generation Test data Execution Test case Oracle Diagnostic Test criterion Stop correct ¬ correct verified ¬ verified

11 Mutation analysis R. DeMillo, R. Lipton and F. Sayward, "Hints on Test Data Selection : Help For The Practicing Programmer". IEEE Computer 11(4): April Technique that aims at validating the quality of a test cases set – Errors injected in the program under test – Compute the proportion of errors detected by the test cases

12 Mutation analysis Several error types: mutation operators – Operators defined from analysis of errors sets observed during development J. Offutt, A. Lee, G. Rothermel, R. H. Untch and C. Zapf, "An Experimental Determination of Sufficient Mutant Operators". ACM Transactions on Software Engineering and Methodology 5(2): April – Recent work [Ma’02, Alexander’02] propose OO specific operators

13 Mutation analysis Faulty program : mutant Test cases detect mutants – Test cases kill mutants Mutation score – Proportion of killed mutants  quality of test cases Two oracles – Traces difference – Executable contracts

14 Mutation analysis P Mutants generation Mutant 1 Mutant 2 Mutant 3 Mutant 4 Mutant 5 Mutant 6 TC Executio n Killed mutant Diagnosis Alive mutant Incomplete specification Automatique Manuel Optimiser Equivalent mutant Deleted from the set of mutants Add contracts Insufficient test cases

15 Automatic test cases generation and optimisation

16 Automatic test cases optimisation Average mutation score easy to reach – By hand or random generation Unit testing – Class testing in an OO context Component testing – Classes assembly with a main interface class

17 Genetic algorithm Build an initial population Compute the fitness value for each individual  Mutation score Reproduction Crossover Mutate one or several individuals Boucle génétique Several stop criteria: good mutation score, number of generations… test_set is do time.set_hour(10) time.set_minute(7) time.set_second(2) time.set(10,7,5) end test_hour_12 is do time.set_hour(0) end test_is_am_pm is do time.set_hour(12) time.set_hour(1) time.set_hour(13) time.set_hour(0) end test_hour is do time.set_hour(0) time.set_hour(1) end test_out is do time.set(02,02,02) end test3 is do time.set(0,0,0) end test_comparaison is local time1 : P_TIME do !!time1 time.set(15,59,59) time1.set(14,59,59) time1.set(16,00,00) end test_hour_12 is do time.set_hour(0) end test_out is do time.set(02,02,02) end test_set is do time.set_hour(10) time.set_minute(7) time.set_second(2) time.set(10,7,5) end test_hour_12 is do time.set_hour(0) end test_is_am_pm is do time.set_hour(12) time.set_hour(1) time.set_hour(13) time.set_hour(0) end test_hour is do time.set_hour(0) time.set_hour(1) end test_out is do time.set(02,02,02) end test3 is do time.set(0,0,0) end test_set is do time.set_hour(10) time.set_minute(7) time.set_second(2) time.set(10,7,5) end test_out is do time.set(02,02,02) end test3 is do time.set(0,0,0) end test_hour is do time.set_hour(0) time.set_hour(1) end test_hour_12 is do time.set_hour(0) end test_is_am_pm is do time.set_hour(12) time.set_hour(1) time.set_hour(13) time.set_hour(0) end test_hour is do time.set_hour(0) time.set_hour(1) end test_out is do time.set(02,02,02) end test3 is do time.set(0,0,0) end test_hour is do time.set_hour(0) time.set_hour(1) end test_out is do time.set(02,02,02) end test3 is do time.set(0,0,10) end

18 Genetic algorithm Tools development (Java, C#) – For mutation analysis: JMutator, NMutator – Framework for the genetic algorithm – Test driver Experiments – Several classes or components

19 C# case study

20 Genetic algorithm Fixed size for the test cases set Crossover not much useful Reproduction not efficient to keep memory

21 Bacteriologic algorithm Rosenzweig, “Species diversity in space and time”, CUP,1995 Delete : – The notion of individual – Crossover operation Introduc : – The notion of bacterium  a test case – A memory set of good bacteria

22 Bacteriologic algorithm Build an initial medium Compute the fitness value for each bacterium  Mutation score Memorise the best ones Mutation Several stop criteria: good mutation score, number of generations… Memory test_out is do time.set(15,59,59) end test_hour is do time.set_hour(0) time.set_hour(1) end test3 is do time.set(0,0,0) end test_set is do time.set_hour(10) time.set_minute(7) time.set_second(2) time.set(10,7,5) end Bacteriologic medium test_out is do time.set(15,59,59) end test_hour is do time.set_hour(0) time.set_hour(1) end test_out is do time.set(02,02,02) end test3 is do time.set(0,0,0) end test_out is do time.set(15,59,59) end test_out is do time.set(15,60,59) end

23 Bacteriologic algorithm Tools development (Java, C#) Experiments – Several case studies – Tuning / validating the model

24 C# case study

25 Results Original algorithm for automatic test cases generation Tools development Work in progress with new programs and new fitness functions

26 Design by contract for robustness and diagnosability

27 Design-by-contract™ A design method for OO components (B. Meyer) Boolean assertions: – pre et post conditions for each method – invariants for global properties A broken contract indicates the presence of a bug: – Precondition violated  a client has broken the contrat – Postcondition violated  an error in the method

28 Two quality criteria Robustness – Ability for a component to detect a faulty internal state Diagnosability – effort for the localization of a fault and the preciseness allowed by a test strategy on a given system, knowing there is a fault

29 Robustness A Local robustnessability of contracts to detect errors Combination is better than addition Global robustness B C A contracts Det(A,C)

30 Robustness Global robustness for a components assembly depends on: – Local robustness of components – The Det(i,j) probability a component i detects errors in j Mutation analysis test cases set with a 100% mutation score local robustness  mutation score of contracts Det(i,j) probability  mutation score for contracts in i with mutants of component j

31 Robustness

32 Diagnosis scope Classical softwareDiagnosability Disgnosis scope Software designed by contracts Exception handling

33 Diagnosability 00,20,40,60,81 Contracts/assertions density Diagnosability Contracts /assertions efficiency

34 Results Qualitative study of design-by-contract Summary Adding contracts, even if they are weak, improves the component’s quality Efficiency of contracts has more impact than the density

35 Testability anti-patterns in a UML class diagram

36 Testability of OO software Control is distributed – Numerous interactions between objects Ambiguities in the design can lead to hard points for testing Class diagram Test criterion Detecting / deleting testability anti- patterns

37 Example

38 Example

39 Example

40 Example

41 Example

42 Anti-patterns Two anti-patterns on the design : – Self-usage BuddyState Buddy 1 -currentState 1

43 Anti-patterns – Class interaction

44 Improve testability Add preciseness on the design to make it closer to implementation Refactoring for testability – Use interfaces when possible Use stereotypes – Specify the roles of associations : consult, create, modify

45 Improve testability

46 Improve testability

47 Improve testability

48 Improve testability «create»

49 Methodology for testability Class diagram Improve the design Refuse the design Accept and implement the design Test Testability analysis Check properties on implementatio n

50 Results Testability analysis on the design Methodology to improve the class diagram Catalogue for the testability of design patterns

51 Conclusion Work for the validation and design of software components – Algorithms for test generation – Models to measure the quality of components Test tools Publications (JSS, ISSRE, Metrics, ASE…)

52 Approach 0 1 Mutation analysis Evolutionist algorithm Design by contract Testability « Testing can prove the presence of bugs, but never their absence » Dijkstra

53 Future work Test generation for efficient diagnosis – Diagnosis algorithms – Test criteria and bacteriologic algorithm Mutation analysis for security Design by contract for test oracle Testability of design patterns – The addition of stereotypes is seen as a model tranformation