Overview of the ETSI Test Description Language Andreas Ulrich (Rapporteur) Siemens AG andreas.ulrich@siemens.com © ETSI 2012. All rights reserved 2nd MBT User Conference 2012 Tallinn, Estonia September 25–27, 2012

2nd MBTUC 2012, Tallinn, Estonia Overview What can I do with TDL? How is TDL designed? How does TDL compare to other approaches? How does TDL look like to users? What comes next? 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia What can I do with TDL? 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia Purpose of TDL Scope Distributed, concurrent, reactive real-time systems Black-box testing Indifferent to application domains Telecommunication / automotive / avionics / automation / others Different, user-definable testing languages Application scenarios Manual specification of tests from use cases, user stories etc. Intermediate representation of automatically generated tests Representing output from MBT tools Generation of Executable tests Test documentation Visualization of test logs 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia TDL Specifies… Interactions between SUT/tester at observable interfaces (only) System inputs/outputs: messages, calls, signals Expected flow of interactions Sequential, alternative, concurrent, recurrent behavior, etc. Mock-up (default) behavior, others Timing constraints System response time (system in/out, out/out constraints) Tester reaction time (system out/in constraints) Test data Concrete values Constraints on valid test data, i.e. ranges or intervals Parameterization of test specifications Individual test cases Entire test suites 2nd MBTUC 2012, Tallinn, Estonia

An Application Scenario Example for TDL Test Purpose Spec (ExTRA) System Model (State Chart) Automatic Test Case Generation Manual Derivation In Syntax #1 Test Case Spec (TDL) Test Documentation (MS Word) Generation In Syntax #2 Automatic Code Generation Executable Tests (TTCN-3) 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia How is TDL designed? 2nd MBTUC 2012, Tallinn, Estonia

Ingredients of a Software Language Abstract Syntax (abstract syntax graph, meta-model) Concrete Syntax (syntax tree, BNF grammar) Formal Semantics (transitional or operational sem.) 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia Coverage by TDL Abstract Syntax Standardized, incl. extension capabilities Concrete Syntax Formal Semantics Informative, by example only Standardized 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia Using TDL Static analysis Consistency checks, visualization Common TDL meta-model Test concepts expressed in a single abstract syntax Test generation Transformation Abstract test descriptions Various user-defined syntaxes, e.g. graphical, tabular, textual Executable tests Various test environments and languages, e.g. TTCN-3, C# 2nd MBTUC 2012, Tallinn, Estonia

Definition of the TDL Meta-Model In fact, two approaches exist OMG’s UML/MOF International standard Complex Eclipse Modeling Framework (EMF) Ecore Simpler than MOF Good tool support For practicability reasons, TDL meta-model will be based on a subset of Ecore 2nd MBTUC 2012, Tallinn, Estonia

TDL is Adjustable by User TDL meta-model can be extended by a user if need arises TDL concrete syntax may cover only parts of the meta-model User extensions of the TDL meta-model can be subjected to further TDL standardization and maintenance Extends Partially implements User-defined concrete syntax TDL meta-model w/ user extensions 2nd MBTUC 2012, Tallinn, Estonia

How does TDL look like to users? 2nd MBTUC 2012, Tallinn, Estonia

Examples in Possible TDL Concrete Syntax Sequential test with multiple SUT ports Structured test with default behavior Note: Examples only, concrete syntax is defined by user! 2nd MBTUC 2012, Tallinn, Estonia

Examples in Possible TDL Concrete Syntax A concurrent test in graphical and textual notation testcase joinTC() { UCAROL ! join(Carol, room1); par { { BOB ? joinPDU(Carol, ?); BOB ! joinPDU(Bob, room1) }, { ALICE ? joinPDU(Carol, ?); ALICE ! joinPDU(Alice, room1) } }; UCAROL ? joined() } A test suite model Note: Examples only, concrete syntax is defined by user! 2nd MBTUC 2012, Tallinn, Estonia

How does TDL compare to other approaches? 2nd MBTUC 2012, Tallinn, Estonia

TDL vs. UML Testing Profile UTP Focus on test descriptions of tester/SUT interactions Various concrete syntaxes Precise semantics Common meta-model for tests fosters reuse of tests and tools Amendable to user’s needs – extension, reduction of scope Wide scope inherited from UML, covers all kinds of test modeling Graphical language Semantics relies on UML Tool-specific solutions, models hardly transferrable Captures not all needs, further profiles needed, e.g. MARTE The semantic fuzz of UML 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia TDL vs. TTCN-3 TDL TTCN-3 Test specification level Captures requirements on a tester w.r.t. SUT behavior Leaves room for various tester code generation strategies Clean test spec possible – level of detail adjustable by user Supports validation of test spec due to concise semantics Test execution level Detailed tester implementation, incl. tester internal behavior Captures the user intension of an executable tester Tester code sometimes cluttered, unreadable Validation hard due to complex semantics 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia Business Value of TDL Provides guidance to users to define ‘their’ test language (concrete syntax) based on common concepts (TDL abstract syntax) Graphical UML-like / textual / tabular representation of tests Enables tool vendors to develop specialized test generation tools from common abstract test model Generation of TTCN-3 / Java/JUnit test code / test documentation Generation of sequential tests / concurrent tests Enables provision of analysis tools independent from a concrete user language Consistency checker Analysis for system input/output races Offers a way to exchange of test specs at an abstract level Common meta-model for all kind of tests 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia What comes next? 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia Status & Outlook Achievements so far Identification of the purpose and business value of TDL Collection of related work Discussion on TDL features Next steps Agreement on TDL core features Design of the TDL meta-model Validate the TDL concepts in a tool  Make a (draft) ETSI standard on TDL available in 2013! Do you want to join? 2nd MBTUC 2012, Tallinn, Estonia

2nd MBTUC 2012, Tallinn, Estonia Acknowledgments So far, the TDL initiative has attracted quickly 30+ persons and different organisations from industry, tool providers, and research from inside and outside of ETSI. The rapporteur of this activity is thankful for the many stimulating and controversial discussions by various people to lay out the cornerstones of TDL! Siemens AG acknowledges partial funding of this activity from the ARTEMIS Joint Undertaking, grant agreement no. 269335 and the German BMBF. 2nd MBTUC 2012, Tallinn, Estonia

Thank you! Your questions?