Download presentation
Presentation is loading. Please wait.
1
A Generic Framework for Model-Set Selection for the
Unification of Testing and Learning MDE Tasks Edouard Batot, Houari Sahraoui DIRO, Université de Montréal
2
Outline Motivation Framework Preliminary evaluation Conclusion
‘ Motivation Learning from (which) examples Model generation tools Framework Characterizing example sets Multi-objective genetic algorithm Preliminary evaluation Conclusion
3
Motivation Automation in Model-Driven Engineering
‘ Motivation Automation in Model-Driven Engineering Motivation Framework Evaluation In order to automate specific tasks MDE aims at rising software abstraction. But specific abstract knowledge is hard to explicit due to the remaining cognitive gap. Example People who actually use the models are not the ones able to write transformations. As an effective alternative, teams have started to use examples to derive the specific knowledge
4
Motivation Learning from example
‘ Motivation Learning from example Motivation Framework Evaluation Existing approaches Gather examples from the industry Use prototypical examples provided by experts Limitation Lack of representativeness Impact the quality of the knowledge learned Automated example set derivation Characterization and control of example set production Exploration of wider horizons than experts/modellers intuition Similar concerns for testing Representativeness and coverage of potential inputs
5
Motivation Related works
‘ Motivation Related works Motivation Framework Evaluation Goals Learning Testing Activities [12,26,27…] [2,6,45,18…] Transformations Activities Manipulate models (input) Goals Close concerns (representativeness) Well-formedness checking [5] Metamodelling [4,8] [14] Refactoring [7] In the paper we assess the effectiveness of our approach in a case study related to the testing of metamodels Metamodelling learning = deriving metamodel from example models Our objective Characterizing and generating example sets tailored to a specific task
6
Applications Proposal
Metamodel Example generation 1 Task Coverage definition ‘ Motivation Framework Evaluation Different input spaces Subsets of the metamodel Similar structure Metamodelling Coverage: Whole metamodel Example: (Source model, {Conform, Not conform}) Learning and testing transformations Coverage: Footprint on source metamodel Example: (Source model, Target model) Learning well-formedness rules Coverage: Footprint on source metamodel (invariant patterns) Example: (Source model, {Valid, Invalid}) Learning MT requires examples of input models to ensure the transformation actually provides the right target. Hence, an example is a couple source morel target model. Here, the literature shows that the footprint of the transformation can be used to ensure the representativeness of the source models. Now, in the activity of learning from examples, close to metamodeling, an example is a couple source model and its annotation as valid or invalid w.r.t. the application domain. Concerning the coverage we investigate the potential benefit of using footprint of what I will precise later the invariant pattern. Proposal Consider example generation with a unified view using a common framework
7
Thorough generation process
‘ Motivation Framework Evaluation Artifact under test Task to learn Metamodel Coverage Definition Oracle Precision Complete examples Current focus Generation of source models 1 Manual Examples’ Completion 2 Minimality Criteria Combination Partial Examples
8
Characterizing example sets A. Coverage definition
‘ Characterizing example sets A. Coverage definition Motivation Framework Evaluation Example Learning well-formedness rules With use of invariant patterns [1] Using patterns’ footprint to reduce the search-space What matters in the metamodel ? Invariant AcyclicReference MOF Structure: The pattern applies whenever there is a class containing a many reference which type is it self. OCL Expression template: context ClassA inv AcyclicRef : ->closure(it : ClassA | it.refA) ->excludes(self) ClassA refA 0..* Invariant ReferenceDifferentFromSelf MOF Structure: Occurs when a class contains a reference typed with itself. OCL Expression template: context ClassA inv ReferenceDifferentFromSelf: attA <> self ClassA attA 0..1 NamedElement ETypedElement lowerBound: int upperBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* NamedElement ETypedElement lowerBound: int upperBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* NamedElement ETypedElement lowerBound: int upperBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EEnum EENumLiteral +eType +eSubPackages +eSuperTypes 0..* 0..1 ERefere nce containement: boolean +eOpposite NamedElement ETypedElement lowerBound: int upperBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* Invariant patterns Refactoring patterns Transformation footprint We mean eliciting which constructs must be considered during the evaluation of coverage. Coverage definition specifies which constructs of the metamodel must be considered during the evaluation of coverage. Coverage definitions To tighten coverage notion to the specific input domain of the task [1] Juan José Cadavid Gómez. Assistance à la méta-modélisation précise. PhD Thesis, Informatique Rennes 1, 2012, supervised by Jézéquel, Jean-Marc and Baudry, Benoit
9
Characterizing example sets B. Structure of an example
‘ Characterizing example sets B. Structure of an example Motivation Framework Evaluation Source space Target space Automated production maximizing coverage Cannot be generated automatically Task Input metamodel describes the input space Target models depend on their corresponding source model -> no automation. No automation -> as few cases as possible ! Sets need to be kept as small as possible. Input domain Target model Example / Case Source model Target model True False Task Source model
10
Characterizing example sets (C.) Minimality
‘ Characterizing example sets (C.) Minimality Motivation Framework Evaluation Model set size must be limited Expert manual completion Computational costs A minimality criterion is required Dissimilarity between models ? (DIS) Number of models ? (MIN) Average number of objects per models ? (MIN-R) Complete examples Manual Examples’ Completion 2 TIME 10 minutes Method Consider model generation as a multi-objective space exploration problem
11
Unified framework Inputs Outputs Metamodel Coverage definition
‘ Motivation Framework Evaluation Inputs Metamodel Coverage definition Minimality criteria DIS MIN MIN-R Outputs Example sets tailored to the task
12
Multi-objective genetic algorithm
Unified framework ‘ Motivation Framework Evaluation Operating ETypedElement NamedElement upperBound: int lowerBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* Metamodel Model base ( models) AtlanMod Instantiator Potential solutions Task Multi-objective genetic algorithm Objectives ETypedElement NamedElement upperBound: int lowerBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* ETypedElement NamedElement upperBound: int lowerBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* Multi-objective optimization for modelling space exploration Coverage NSGA-II Minimality DIS/MIN/MIN-R
13
Near optimal example sets
Unified framework ‘ Motivation Framework Evaluation Operating ETypedElement NamedElement upperBound: int lowerBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* Metamodel Model base ( models) AtlanMod Instantiator Potential solutions Task Front 3 P0 Q0 Front 1 Front 2 Front 5 Front 4 Rejected Genetic operators Non-dominance Sorting Crowding distance sorting Looping generation P1 Objectives ETypedElement NamedElement upperBound: int lowerBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* ETypedElement NamedElement upperBound: int lowerBound: int EClassifier EPackage EStructuralFeature derived: boolean EClass EDataType EAttribute EReference containement: boolean EEnum EENumLiteral +eType +eSubPackages +eOpposite +eSuperTypes 0..1 0..* Multi-objective optimization for modelling space exploration Coverage Minimality DIS/MIN/MIN-R 13 Near optimal example sets (Pareto)
14
Preliminary evaluation
‘ Motivation Framework Evaluation COV DIS MIN COV&DIS COV&MIN MIN-R Different configurations Two measures Coverage Size 3 metamodels One of great size Probabilistic algorithm 30 runs per config. Statistical tests Mann-Witney Coverage Random Mono-objective 2-objectives 3-objectives Number of models Avg. size of models Griser les oints au fur et a mesure. With a low cost on coverage we can reduce the size of the solution COV DIS MIN COV&DIS COV&MIN MIN-R
15
Coverage Number of models Avg. size of models Results
‘ Motivation Framework Evaluation COV DIS MIN COV DIS MIN-R COV MIN MIN-R COV DIS COV MIN COV MIN-R DIS COV&DIS COV&MIN DIS MIN Coverage 1,00- ,90- ,80- ,70- ,60- ,50- ,40- 65 11 Random is not an option Mono-obj. #models explodes Coverage Random Mono-objective Mono-objective 2-objectives 3-objectives # models 50 60 58 55 59 121 186 -180 -140 -100 -60 -160 -120 -80 Average # of objects 150- 100- 50- 0- Best tradeoff DIS Number of models Avg. size of models With a low cost on coverage we can reduce the size of the solution (+ MIN) COV&DIS COV&MIN COV DIS COV MIN COV MIN-R COV DIS MIN COV DIS MIN-R COV MIN MIN-R DIS DIS MIN
16
Conclusion A unified model generation framework
‘ Motivation Framework Evaluation A unified model generation framework Generic, it provides input models for different types of activities and goals Flexible, it considers Specific notions of coverage Minimality criteria Encouraging preliminary results Metamodel testing
17
Discussion & Future work
‘ Motivation Framework Evaluation Limits Diversity of the model base Complexity of the input metamodel Transformations Well-formedness rules Metamodelling Refactoring Testing Learning [12,26,27] [4,8] [2,6,45,18…] [7] [14] [5] Objectives Tasks Future work Quantitative effect Specific Knowledge Abstraction 3 Does coverage impacts knowledge’s accuracy ? Regarding the learning accuracy /testing efficiency Qualitative feedback Regarding the expert’s concerns Manual Examples’ Completion 2 Of “reasonable” size ? Meaningfull ? Partial examples
18
A unified model generation framework
‘ A unified model generation framework Motivation Framework Evaluation Obj. and tasks that share the same concerns of coverage to be max. and size to be min.
Similar presentations
