1. Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Benchmarking for Graph Transformation Gergely Varró gervarro@cs.bme.hu Andy Schürr andy.schuerr@es.tu-darmstadt.de Dániel Varró varro@mit.bme.hu

2. 2 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Talk overview Introduction GT overview Paradigm & tool features Benchmarking terminology Measurements

3. 3 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Model transformation in MDD Model Driven Development (MDD) –thorough system modeling before implementation –precise and visual SW models on a high abstraction level from different viewpoints –automatic code generation Model transformation –automated translations within and between visual modeling languages MDD requirements on model transformation: –Specification: high-level specification language –Execution: efficient algorithms and techniques –Industrialization: extensive tool support Upcoming standard: QVT

4. 4 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Specification techniques for MT QVT Main features: –object based patterns –precondition –unidirectional –bidirectional Problems: –immature tool support –lack of experience GT Main features: –similar to QVT –wide tool support Problems: –non-bidirectional (except for TGGs) Pattern matching phase: –most critical issue from performance viewpoint –same as in QVT

5. 5 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Benchmarking Aim: –systematic and reproducable measurements –on performance –under different and precisely defined circumstances Overall goal: Ihelp system engineers in selecting tools Popular approach in different fields –AI –relational databases –rule-based expert systems

6. 6 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Benchmarking in model transformation Specification examples for MT –Goal: assessing expressiveness –UML-to-XMI, object-relational mapping, UML-to-EJB, etc. No performance benchmarks for MT In the paper: Benchmarks for graph transformation tools –terminology for benchmarking –quantitative comparison of tools Ongoing research: – Benchmarks for general model transformation tools

7. 7 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Talk overview Introduction GT overview Paradigm & tool features Benchmarking terminology Measurements

8. 8 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Metamodel Metamodeling h1:held_by n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by next blocked held_by releasetoken request * 1 * 1 * 111 ** * * Class Association Multiplicity constraint At most one Arbitrary p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource Resource Process Object Link Slot Inheritance Attribute Instance model

9. 9 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Graph transformation rule r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS Negative applicationc ondition p1:Process r:Resource GiveRule n:next p2:Process rel:release p1:Process r:Resource n:next p2:Process t:token LHSRHS Left-Hand Side Right-Hand Side

10. 10 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Pattern matching phase h1:held_by n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS

11. 11 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Updating phase h1:held_by n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS rl:release

12. 12 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Talk overview Introduction GT overview Paradigm & tool features Benchmarking terminology Measurements

13. 13 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Paradigm features Definition: problem characteristics Model size –number of nodes and edges in the model graph Pattern size –number of nodes and edges in the LHS graph Fan-out (maximum degree of model nodes) –number of edges adjacent to a certain node Number of matchings Transformation sequence length

14. 14 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Tool features Definition: typical optimization strategies supported by different tools Multiplicity based optimization –More powerful strategy when traversing edges with ‘at most one’ (0..1) multiplicity Parameter passing Parallel rule execution ‘As long as possible’ (ALAP) rule application Indexes (for key/index attributes) Incremental pattern matching...

15. 15 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Parameter passing I. h1:held_by n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS rl1:release Between consecutive rule applications No recalculation in later steps

16. 16 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Parameter passing II. rl1:release n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource p1:Process r:Resource GiveRule n:next p2:Process rel:release p1:Process r:Resource n:next p2:Process t:token LHSRHS t1:token

17. 17 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Parallel rule execution (PM) h1:held_by n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS 4 12 3 1 4 3 2 4 3 2 1 Calculation of all matchings Independent matchings

18. 18 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Parallel rule execution (Update) h1:held_by n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS 4 12 3 1 4 3 2 4 3 2 1 rl1:release rl3:release rl2:release rl4:release No re-evaluation of valid matchings Transaction block

19. 19 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Talk overview Introduction GT overview Paradigm & tool features Benchmarking terminology Measurements

20. 20 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology I. Metamodel GT rules Instance model Transformation sequence Paradigm features Tool features Scenario –––––– –Broad application field of graph transformation model analysis model transformation visual languages with dynamic operational semantics –Informal characteristics

21. 21 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology II. Metamodel GT rules Instance model Transformation sequence Paradigm features Tool features Scenario –––––– Benchmark example ++–––– next blocked held_by releasetoken request * 1 * 1 * 111 ** * * Resource Process TakeRule RequestRule GiveRule ReleaseRule LHSRHS Incarnation of a scenario –fulfills the informal characteristics –e.g. Mutex for the simulation of visual languages

22. 22 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology III. Metamodel GT rules Instance model Transformation sequence Paradigm features Tool features Scenario –––––– Benchmark example ++–––– Test set ++ PD – h1:held_by nN:next n1:next hN:held_by p1:Process p2:Process pN:ProcessrN:Resource r1:Resource Instance model 1.N ReleaseRule 2.N GiveRule 3.N RequestRule 4.N TakeRule Parametric and deterministic specification

23. 23 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology IV. Metamodel GT rules Instance model Transformation sequence Paradigm features Tool features Scenario –––––– Benchmark example ++–––– Test set ++ PD – Test case ++ PD + Multiplicity based optimization: OFF Parameter passing: OFF Parallel rule execution: ON

24. 24 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology V. Metamodel GT rules Instance model Transformation sequence Paradigm features Tool features Scenario –––––– Benchmark example ++–––– Test set ++ PD – Test case ++ PD + Run ++++++ Complete characterization Reproducable measurements N = 10

25. 25 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Talk overview Introduction GT overview Paradigm & tool features Benchmarking terminology Measurements

26. 26 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Categorization Style –Interpreted: AGG, GTDB –Compiled: PROGRES, Fujaba Base algorithm –Constraint satisfaction: AGG, GTDB variables + constraints –Local searches: Fujaba, PROGRES step-by-step extension of the matching

27. 27 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Environment Hardware and OS –1500 MHz Pentium –768 MB RAM –Linux kernel of version 2.6.7 Tool related –Runs without GUI –Rule applications guided by Java programs PROGRES: guided by C program –Standard services of the default distribution fine-tuning based on developers’ suggestions

28. 28 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Tool specific issues AGG –Manual implementation of parameter passing Progres –GRAS (default underlying graph-oriented database) –Prolog-style cuts (determinism) –User undo/redo mechanism being switched off Fujaba –Additional node for storing processes and resources –Slightly altered models to provide better performance GTDB –MySQL (version 4.1.7) –No additional optimization

29. 29 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Tool feature support Multiplicity based optimization Parameter passing Parallel rule application AGG –+– PROGRES +++ Fujaba +++ GTDB –++

30. 30 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Quantitative comparison Result set clean-up procedure Constant overhead distributed over more rule applications Compilation step from graphs to categories in each GT step

31. 31 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Effect of multiplicity based optimization GiveRule ReleaseRule AGGPROGRESFUJABAGTDB

32. 32 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Effect of parameter passing AGGPROGRESFUJABAGTDB STS test set, Pattern matching phase, ReleaseRule

33. 33 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Effect of parallel rule application AGGPROGRESFUJABAGTDB ALAP test set, Pattern matching phase, ReleaseRule

34. 34 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Thanks for your kind attention Contributions to model transformation benchmarking are welcome –additional benchmark examples, test sets, test cases –participation of further tools –new optimization strategies Benchmark example descriptions, specifications, and measurement results –http://www.cs.bme.hu/~gervarro/benchmark/

35. 35 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Tool feature support Parameter passing Parallel rule application Multiplicity based optimization ALAP rule application AGG +––– PROGRES +++– Fujaba +++– GTDB ++––

36. 36 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology Metamodel GT rules Instance model Transformation sequence Paradigm features Tool features Scenario –––––– Benchmark example ++–––– Test set ++ PD – Test case ++ PD + Run ++++++

37. 37 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Model driven development Aim: –thorough system modeling before implementation Key idea: –SW models on various abstraction levels and from different viewpoints –automatic code generation Key requirement: –decision and design support for domain experts –precise and visual models on a high abstraction level

38. 38 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Model transformations in MDD Aim: –automated translations within and between (visual) modeling languages MDD requirements on model transformation: –Specification: high-level specification language –Execution: efficient algorithms and techniques –Industrialization: extensive tool support Model transformation tool categories –direct manipulation, relational, graph transformation based, structure driven, hybrid, XSLT based

39. 39 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Benchmarking in model transformation Specification benchmarks for MT –Goal: assessing expressiveness –UML-to-XMI, object-relational mapping, UML-to-EJB, etc. No performance benchmarks for MT In the paper: Benchmarks for graph transformation tools –terminology for benchmarking –quantitative comparison of tools Complete solution (ongoing research) (vagy elején vagy végén): –Adaptation to general model transformation tools QVT which is going to be the standard for capturing/defining model transformation, uses a notation that is similar to graph transformation

40. 40 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology III. (still cont.) Scenario Benchmark example Test set –Parametric and deterministic specification –Additionally fixed structure of instance models transformation sequences up to numerical parameters (e.g. number of processes)  calculation of paradigm features –Undefined tool feature (optimization strategy) settings

41. 41 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology IV. Scenario Benchmark example Test set Test case –Additionally fixed tool feature (optimization strategy) –Still parametric paradigm feature (characteristics of the model, transformation)

42. 42 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Terminology V. Scenario Benchmark example Test set Test case Run –Additionally fixed runtime parameters –Complete characterization –Reproducable measurements

43. 43 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Graph transformation rules I. p1:Process r:Resource GiveRule n:next p2:Process rel:release p1:Process r:Resource n:next p2:Process t:token r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS LHSRHS Negative applicationc ondition LHS

44. 44 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab Graph transformation rules II. p:Process r:Resource TakeRule t:token r:Resource req:requesthb:held_by LHSRHS p:Process RequestRule t:token r:Resource reqn:request p:Process t:token r:Resource req:request LHSRHS

45. 45 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab As long as possible rule execution I. h1:held_by n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS rl1:release Standard graph rewriting steps applied as long as possible Decreasing number of matchings

46. 46 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab As long as possible rule execution II. rl1:release n4:next n1:nextn2:next n3:next h2:held_by h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS rl2:release

47. 47 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab As long as possible rule execution III. rl1:release n4:next n1:nextn2:next n3:next rl2:release h3:held_by h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS rl3:release

48. 48 Budapest University of Technology and Economics Technical University of Darmstadt Real-Time Systems Lab As long as possible rule execution IV. n4:next n1:nextn2:next n3:next rl3:release h4:held_by p1:Processp3:Process p2:Process p4:Processr4:Resource r1:Resourcer2:Resource r3:Resource r:Resource p:Process r:Resource ReleaseRule hb:held_by reqn:request rel:release rn:Resource p:Process LHSRHS rl1:release rl2:release rl4:release