Automatic Effective Model Discovery Sagar Sen Thesis Director: Jean-Marc Jézéquel Thesis Supervisor: Benoit Baudry Co-supervisor : Hans Vangheluwe 1 DOMINO

There are many mansions in the mind of a scientist 2 Nikola Tesla (Inventor)

3 Laws of Science Experiments Instruments Model of the Electric Motor design

4 Models : Are simplified abstract views of complex reality Effective Models: Models that serve a purpose or satisfy an objective. It is created based on knowledge from heterogeneous sources. The notion of effectiveness

DNA Molecule 5 Tesla Electric Motor An effective model to predict and invent An effective model discovered in nature

Effective Models in Software 6 Effective Model of Phone Behavior used by millions

An effective model is a product of mental tacit and manual work 7 Can we automate discovery of effective models? In the context of Software Systems

Outline Context and Problem The Solution Model Discovery Product Discovery Conclusion Future Work 8

Model-driven Engineering (MDE) of Software Systems 9 Modelling Domain Specification (A Blueprint : Concepts with Properties allowed in a set of models) Modelling Domain Phone Behavior Model Models are first-class artifacts in MDE

Set of all Finite State Machines 10 Example Modelling Domain Metamodel (a blueprint) of Finite State Machines

Automatic Effective Model Discovery The Problem 11

Modelling Domain specification 12

Source 1 Source 2 Source N … Heterogeneous sources of knowledge may guide discovery 13

Source 1 Source 2 Source N … Effective Modelling Domain Discovery Mechanism? Effective Models 14 ?? Reliable Sources?

Outline Context and Problem The Solution Model Discovery Product Discovery Conclusion Future Work 15

Conceptual Solution 16 Source 1 Source 2 Source N … Modelling Domain specification Transform Effective Modelling Domain specification defines Constraint Satisfaction Problem (CSP)

17 Source 1 Source 2 Source N … Modelling Domain specification Constraint Satisfaction Problem Solve Effective Modelling Domain Conceptual Solution

Main Challenges 18 1.Heterogeneous sources of knowledge are expressed in different languages. How can we transform them to a single constraint satisfaction problem? 2.Knowledge is expressed on sets of objects in a model called meta-constraints. Eg. There must be at least one Final State in the set of all States in a Finite State Machine. How can we resolve such meta-constraints? 3. Models can be of infinite size! How do we control the generation of finite sized models?

Technical Solution 19 Model Mx UML OCL … Modelling Domain specification Language Lx conforms to CSP as Alloy Model Transformation Class Diagram Invariants

Why Alloy ? 20 1.Sets of concepts/classes with properties can be modeled as Alloy signatures with fields in an Alloy model A. 1.Meta-constraints on the concepts in A can be modeled as Alloy facts and predicates (based on first- order relational logic with quantifiers) 1.Alloy model A can be solved in a finite scope to give solution instances. CSP as Alloy Model

Concrete Realization of Solution 21 Modelling Domain Specification MetamodelFeature Diagram Effective Model Discovery in any Modelling Language Effective Product Discovery in the Variability Domain of a Software Product

Outline Context and Problem The Solution Model Discovery Product Discovery Conclusion Future Work 22

23 Model Discovery in the domain of a Modelling Language Metamodel MM Set of Valid Models in a Modelling Language How to discover effective models for a given objective?

24 Effective Model Discovery Objectives in this Thesis Objective 1: Test model discovery for detect bugs in model transformations Objective 2: Partial Model Completion in Model Editors

25 Motivating Example UML2RDBMS (For Object Persistence) Model Transformation (MT) Adding a friend on FB (Change in Social Graph) Examples Input Models Output Models Input metamodel MM_in Output metamodel MM_out

26 Motivating Example Model Transformation (MT) Input Models Output Models There may be BUGS in the transformation MT!

27 Motivating Example Model Transformation (MT) Input Models Output Models Can we discover the Effective Modelling Domain and Effective TEST Models that detect these bugs?

What is the effective modelling domain for testing a transformation? 28

29 Input Models MM_in Heterogeneous Sources of Knowledge define the Effective Modelling Domain

28 Input Models MM_in Heterogeneous Sources of Knowledge define the Effective Modelling Domain Required Types and Properties

28 Input Models MM_in Invariants I on MM (OCL) Heterogeneous Sources of Knowledge define the Effective Modelling Domain Required Types and Properties

28 Input Models MM_in Invariants I on MM (OCL) Pre-condition pre Heterogeneous Sources of Knowledge define the Effective Modelling Domain Pre MT Input Domain Required Types and Properties

28 Input Models MM_in Invariants I on MM (OCL) Pre-condition pre Test Knowledge Model fragments F Heterogeneous Sources of Knowledge define the Effective Modelling Domain Pre MT Test Models Input Domain Required Types and Properties

Automatic test model discovery : The Approach 34

35 Running Example :UML2RDBMS Transformation Pre UML2RDBMS Post UML Class Diagram Models RDBMS Models Bezivin, J., Rumpe, B., Schurr, A., Tratt, L.: Model transformations in practice workshop, october 3rd 2005, part of models In: Proceedings of MoDELS. (2005)

36 Input meta-model MM_in Invariants I on MM_in (OCL) Test Knowledge as set of model fragments F Test Model Discovery : The Approach Pre MT Pre-condition pre Alloy Model 1.Metamodel Pruning Effective metamodel eMM_in 2. MM2Allo y 4. MF2Allo y 3. OCL2Alloy (Manual/Auto matic) Required Types and Properties T_req, P_req

Metamodel Pruning : The Process 37 Input meta-model MM_in Effective metamodel eMM_in 1.Metamodel Pruning MT (UML2RDBMS) MT (UML2RDBMS) Required Types and Properties Class Association Attribute PrimitiveType Package NamedElement:name static analysis All UML Models Class Diagrams

Metamodel Pruning : The Results 38 1.Required types and properties and their obligatory dependencies are preserved. 2.All instances of effective MM are instances of input MM 3.Effective MM is a super type and a subset of of MM_in. This implies that all operations on eMM_in are valid for MM_in subsetOf superTypeOf UML Class Diagram Subset

39 Input meta-model MM_in Invariants I on MM_in (OCL) Test Knowledge as set of model fragments F Test Model Discovery : The Approach Pre MT Pre-condition pre Alloy Model 1.Metamodel Pruning Effective metamodel eMM_in 2. MM2Allo y 4. MF2Allo y 3. OCL2Alloy (Manual/Auto matic) Required Types and Properties T_req, P_req

40 Why MM2Alloy ? Structural Similarity Metamodel classes specify a Set of Objects with Properties Alloy signatures specify sets of atoms and Alloy fields, facts, and predicate specify allowed relations between atoms Object and Properties Atoms and Relations meta-level similarity instance-level similarity

What aspects to transform by MM2Alloy? 41 Metamodel MM Meta-metamodelling language expressed in Meta-metamodel (Eg. Ecore is an industrial standard) specified by

MM2Alloy : Bridging the Semantic Gap 42 Birds Eye view of the meta-metamodel Ecore that specifies the set of all metamodels

MM2Alloy : Bridging the Semantic Gap 43 We zoom into the concept of Classes and Properties (Attributes and References) (abstract-)Class and Properties to Alloy Signatures and Fields

MM2Alloy : Bridging the Semantic Gap 44 We zoom into the concept of multiple inheritance Flattening Multiple Inheritance to Single Inheritance in Alloy

MM2Alloy : Bridging the Semantic Gap 45 Zoom into characteristics of properties Opposite, Multiplicity of, Composite Properties to Alloy Facts

Example Transformations to Alloy 46

47 Transformation to Alloy (1) Lets take the Classifier concept from the UML Class Diagram meta-model Transformed to an Alloy Signature abstract sig Classifier { name : String } Lets take the Class concept concept that inherits from Classifier Transformed to an Alloy Signature sig Class extends Classifier { is_persistent: one Bool, parent : lone Class, attrs : some Attribute } UML Class Diagram Subset

48 Transformation to Alloy (2) Implicit Constraints to Facts Containment is transformed to an Alloy fact fact Association_containers { (all o : Association |o in ClassModel.association) }

MM2Alloy : Bridging the Semantic Gap 49 Example from Kermeta Type System MM to illustrate multiple containers Multiple Containers to Alloy Facts Containment is transformed to an Alloy fact fact TypeVariableBinding_containers { (all o1,o2,o3 : TypeVariableBinding| (o1 in ParameterizedType.typeParamBinding and o2 in ParameterizedType.virtualTypeBinding and o3 in VirtualType.typeParameterBinding ) and disj[o1,o2,o3] }

50 Input meta-model MM_in Invariants I on MM_in (OCL) Test Knowledge as set of model fragments F Test Model Discovery : The Approach Pre MT Pre-condition pre Alloy Model 1.Metamodel Pruning Effective metamodel eMM_in 2. MM2Allo y 4. MF2Allo y 3. OCL2Alloy (Manual/Auto matic) Required Types and Properties T_req, P_req

Transformation of OCL to Alloy 51 No Cyclic Inheritance OCL Version: context Class: inv : not self.allParents()->includes(self ) Alloy Version: fact noCyclicInheritance { no c: Class | c in c.^parent } manually to an Alloy fact ?

52 Input meta-model MM_in Invariants I on MM_in (OCL) Test Knowledge as set of model fragments F Test Model Discovery : The Approach Pre MT Pre-condition pre Alloy Model 1.Metamodel Pruning Effective metamodel eMM_in 2. MM2Allo y 4. MF2Allo y 3. OCL2Alloy (Manual/Auto matic) Required Types and Properties T_req, P_req

53 Test Model Fragments 1.We generate partitions of the effective metamodel using input domain partitioning to cover various structural properties of the input domain. Franck Fleurey, Benoit Baudry, Pierre-Alain Muller, Yves Le Traon. Qualifying Input Test Data for Model Transformations. Software and Systems Modeling (SoSyM), Volume 8, Number 2, April We combine partitions using two strategies: AllRanges and AllPartitions to give model fragments that cover wide range of characteristics the entire input metamodel 3. A model fragment is transformed to a predicate that must satisfied by a test model.

54 Test Model Fragment to Alloy Predicate pred modelFragment1 { some a1:Class,a2:Class, a3:Class|#a1.name=0 and a2.name=0 and #a3.name>0 } automatically transformed to an Alloy predicate a1 : Class name : {null} a2 : Class name : {« »} a3 :Class name : {.+}

55 Input meta-model MM_in Invariants I on MM_in (OCL) Test Knowledge as set of model fragments F Test Model Discovery : The Approach Pre MT Pre-condition pre Alloy Model 1.Metamodel Pruning Effective metamodel eMM_in 2. MM2Allo y 4. MF2Allo y 3. OCL2Alloy (Manual/Auto matic) Required Types and Properties T_req, P_req

56 Test Model Discovery: The Approach Test Models Alloy Model 5. Solve

57 Can these test models detect bugs? 1.Mutation Analysis: Detecting artificially injected bugs 2.We perform an empirical study to compare test models generated using different sources of knowledge. We use mutation score as a measure of comparison Sources: (a) AllPartition (5 fragments) of 8 sets of 50 models/set = 400 models (b) Random in Input Domain of 8 sets of 50 models/set = 400 models (c) AllRanges (15 fragments) of 8 sets of 150 models/set = 1200 models (d) ) Random in Input Domain of 8 sets of 150 models/set = 1200 models Total of 3200 models with multiple Alloy solutions (10) per fragment Time to automatically discover: 3 hours on MacBook Pro with Intel Core Duo processor and 2GB of RAM Time for mutation analysis: 7 days on a grid. Executing 200 mutant transformations for each 3200 models

Mutation Analysis 58 Y-axis Mutation Score X-axis Test Strategy

Mutation Analysis 59 AllPartitions (93.3%)AllRanges (92.27%)

Summary of Model Discovery 60 1.Automatic Discovery of Effective Models is Possible! 1.1 We present a software framework to transform knowledge from various sources to a CSP in Alloy 1.2 We solve the CSP in Alloy model to discovery models 1.Empirical studies validate the effectiveness of models! 2.1 Automatically discovered test models help us detect 93 % bugs in the UML2RDBMS transformation! 3. The whole approach is implemented in the tool Pramana previously known as Cartier conforming to industry standard inputs

Outline Context and Problem The Solution Model Discovery Product Discovery Conclusion Future Work 61

Product Discovery 62 Feature Diagram (FD) specifies a set of product configurations How to automatically discover effective products for a given objective?

Motivating Example 63 E-Health Emergency Composite Service Insurance Service Treatment Service Ambulance Service Documentation Service

E-Health Service Workflow 64 Atomic Services in a Workflow

Variability in E-Health Service 65 E-Health Emergency Service is a DYNAMIC SERVICE In Traffic Jam!Choices in Insurance

Reconfiguration of Workflow 66

Variation Captured in a Feature Diagram 67 CHOICE MANDATORY OPTIONAL

Variability in E-Health Composite Service 68 reconfigures Removing unselected services Orchestration Workflow Selection of a set of features

69 All Possible Configurations 53,248 Configurations!

70 Variation in QoS such as Response Time 53,248 Possible response time distributions for E-health! (Based on Sidney et. al. Probabilistic QoS and soft contracts for transaction based Web services, ICWS 2010) Example response time distributions

71 Can this sample be used to better analyze E-healths overall QoS ? Is there small but effective set of configurations that widely samples the configuration space?

How do we sample? 72 What are the configurations that cover all pairwise interactions between features (or services)? J. Czerwonka, Pairwise Testing in Real World Proceedings of 24th Pacific Northwest Software Quality Conference, 2006

Pairwise Interactions 73 Ambulance_fDoctor_s Pair1N/A Pair2N/AA Pair3AN/A Pair4AA Eg: Pairs say what must be in a configuration that satisfy it. Total of 2380 pairs

74 We discover only 185 configurations covering all valid pairs in the 2380 feature interaction pairs Configuration Discovery Methodology Feature Diagram FD T-wise Strategy T (Eg. T=2 implies pairwise) 1. FD2Alloy 2. Generation of Alloy Predicates for Tuples (pairs for T=2) Alloy Model 3. Detection of Consistent Tuples (w.r.t FD) 4. Generate Configurations covering valid pairs (Divide and Compose) 4. Generate Configurations covering valid pairs (Divide and Compose)

Resulting Variation in QoS for 185 Configurations 75

Box Plot of 37 configurations out of 185 configurations 76 Y-axis Response Time X-axis Configurations Probability Distribution

Box Plot of 37 configurations out of 185 configurations 77 Extreme cases show up Only mandatory services Maximum number of services Robust Determination of Service Level Agreement

Summary of Product Discovery 78 1.Automatic Discovery of Effective Product Configurations is Possible! 1.1 This thesis presents a software framework to transform a feature diagram and T-wise interactions to a CSP in Alloy to discover products 2. Empirical studies validate the effectiveness of products! 2.1 We automatically discover configurations of a dynamic web service that helped determine better SLA (In collaboration with Ajay Kattepur, Team DISTRIBCOM) 2.2 Two large case studies: Crisis Management, E-Health 3. The whole approach is implemented in the tool Avishkar

Outline Context and Problem The Solution Model Discovery Product Discovery Conclusion Future Work 79

80 Conclusion 1. Can we automatically discover effective models and products ? Answer: Yes! 2.Were we able to qualify discovery using empirical studies? Answer: Yes! Eg. Via mutation analysis in Test Model Discovery 3.Is there a software prototype for automatic model discovery? Answer: Yes! Pramana (previously Cartier) 4.Did we apply the general approach to other domains? Answer: Yes! The domain of Feature Diagrams implemented in tool Avishkar 5.Were we able to qualify product discovery using empirical studies? Answer: Yes! To sample QoS and determine SLA in a composite dynamic service

81 Conclusion 8. Any key scientific contributions? Answer: In our opinion, a) First results in automatic discovery of models satisfying knowledge from various sources. Notably discovery of test models! Why not other forms of discovery? b) We leverage the idea of model transformations to scale formal methods to address very specific and modular needs (Eg. using metamodel pruning) in MDE. Theory truly meets practice! c) Automatic discovery facilitates large-scale empirical studies in MDE

Outline Context and Problem The Solution Model Discovery Product Discovery Conclusion Future Work 82

83 Future Work 1.Discovery of large and complex models using discovery of small models followed by their composition. 1.Discovery of models using feedback knowledge from fitness computation 2.What John Koza did with Genetic programming we want to do with Automatic effective model discovery! 1.Industrial Impact: Automatic discovery and composition of products in a Very Large Feature Diagram (Video Surveillance System from Keeneo) for a given intention.

Publications 84 JOURNAL ARTICLES (During Thesis) (12) Sen S., Moha N. Baudry B., Mahe V., Barais Olivier, and J.-M.Jezequel, Reusable Model Transformations 2010 (Under Review) (11) Sen S., Mottu J.M. Baudry B., Automatic Model Generation for Transformation Testing 2010 (Under Review) (10). N. Moha, Sen S., Faucher C., Barais O., and J.-M.Jezequel, Evaluation of Kermeta on Graph Transformation Problems, Journal of Software Tools and Technology Transfer, 2010 (9). Sen S., B. Baudry, Vangheluwe H., Towards Domain-specific Model Editors with Automatic Model Completion, Simulation : Transactions of the Modeling and Simulation Society, 2009 CONFERENCE ARTICLES (During Thesis) (8) Kattepur Ajay, Sen S., Baudry B., Beneveniste A., Jard C. Variability Modeling and QoS Analysis of Web Services Orchestrations (7) G. Perrouin, Sen S., Klein J., Baudry B, Le Traon Yves, Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines, ICST 2010 (6) Sen, S., N. Moha, B. Baudry, and J.-M.Jezequel, Meta-model Pruning. In Proceedings of MoDeLs (5) Sen, S., B. Baudry, and J.-M. Mottu. Automatic Model Generation Strategies for Model Transformation Testing. In Proceedings of International Conference on Model Transformation (4) Sen S., Baudry B., Mottu Jean-Marie. On Combining Multi-formalism Knowledge to Select Models for Model Transformation Testing. ICST'08, 2008 (3) Sen S., Baudry B. Partial Model Completion in Model Driven Engineering. In Proc. of International Conference of Applications of Declarative Programming 2007 (2) Sen S., Baudry B., Vangheluwe Hans, Domain Specific Model Editors with Model Completion, LNCS vol. on Models in Software Engineering,Workshops and Symposia at MoDELS 2007 (Best Paper) (1) Sen S., and Baudry B., Mutation-based Model Synthesis in Model Driven Engineering, Proc of IEEE ISSRE, Mutation Workshop Model Discovery Articles Effective Modelling Domain Articles Product Discovery Articles

THANK YOU I AM PLEASED TO ADDRESS YOUR QUESTIONS 85