ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing Nikolay Pakulin Contributors: Ina Schieferdecker Jens Grabowski

TTCN-3 User Conference 2012, Bangelore, India Agenda  STF 430: TTCN3 Evolution  TTCN3 Overview  Advanced TRI  Continuous Signals  STF 433: TTCN3 Tool Conformance Testing  Motivation  Methodology  Challenges  Requirements-driven Test Development Slide 2 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 Evolution TTCN-3 User Conference 2012, Bangelore, India Nikolay Pakulin on behalf of STF430

TTCN-3 User Conference 2012, Bangelore, India STF administrative data  STF 430: April 2011 – December 2011 (78 mDays)  Members:  Gyorgy Rethy, Ericsson  Jens Grabowski, University of Goettingen  Ina Schieferdecker, Fraunhofer FOKUS  Jacob Wieland, Testing Technologies  STF 430 sessions in 2011  Session 1: May  Session 2: 27 June - 1 July  Session 3: September  Session 4: 28 November - 2 December Slide 4 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India IDLXSD Language mappings ASN.1 Extensions Advanced parameteri- zation Behaviour types Static configuration Documentation t3doc TTCN-3 Structuring: Imports, Groups, Attributes Core language TTCN-3 Technology Overview TTCN-3 Behaviour TTCN-3 Data CC++ TRI/TCI mapping Java XML C# Real-time support Continuous signal support Advanced TRI Slide 5 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India TTCN-3 language evolution  Tasks  Resolution and implementation of CRs  Prepare the drafts for ETSI publication: Parts 1 and 4 to 10 of the ES series The 4 extension packages (ES , ~782, ~784, ~785)  Cooperation with STF 160 on language requirements of LTE testing  Two new language extension packages  Advanced TRI  Continuous signal support Slide 6 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India The version forest  Approved new versions  TTCN-3 Part-1: Core Language V4.4.1 (44 CRs)  TTCN-3 Part-4: Operational SemanticsV4.4.1 (1 CR)  TTCN-3 Part-5: TTCN-3 Runtime Interface V4.4.1 (2 CRs)  TTCN-3 Part-6: TTCN-3 Control Interface V4.4.1 (8 CRs)  TTCN-3 Part-7: Using ASN.1 with TTCN-3 V4.4.1 (1 CR)  TTCN-3 Part-8: Using IDL with TTCN-3 V4.4.1 (2 CR)  TTCN-3 Part-9: Using XML with TTCN-3 V4.4.1 (12 CRs)  TTCN-3 Part-10: TTCN-3 docum. tagsV4.4.1 (1 CRs)  TTCN-3 Extension: Behaviour TypesV1.2.1 (3 CRs)  TTCN-3 Extension: Configuration and deployment V1.2.1 (1 CR)  TTCN-3 Extension: Real-Time and Performance Testing V1.2.2 (2 CRs)  Not republished:  TTCN-3 Extension: Advanced ParameterizationV1.3.1  New:  TTCN-3 Extension: Advanced TRI V1.1.1  TTCN-3 Extension: Continuous signal supportV1.1.1 Slide 7 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India New features in Part 1  5417: Support of parametrized map/unmap var MyConfigType cfg:={ option := 1, lock := false};... map(mtc:Port4, system:PCO2) param (cfg);  5262: Partially constrained structured types  5938: Type restriction by template li st type MyRecord MyRecordSub5 ( {f1 := *, f2 := "user", f3 := pattern "password|Password" }, { f1 := (1.. 10), f2 := "User“, f3 := ? })  5937: Generalized annotation of attributes to declarations/members Slide 8 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India New: Advanced TRI 1(3)  Historically, TTCN has been used to test communication protocols which typically use encoded messages. This has been reflected in the TRI SA and TCI CD design of TTCN-3 by encoding and decoding messages to/from bitstrings.  However, TTCN-3 also supports signature-based communication for which the transformation of objects into bitstrings and vice versa is cumbersome.  Therefore, an alternative API is being proposed along which TTCN-3 values can be directly passed to/from the SUT.  Old and new TRI can be used in combination! Slide 9 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India New: Advanced TRI 2(3)  Redefinition of TRI Data Interface  TriMessageType  use Value  TriAddressType  use Value  TriAddressListType  use ValueList  TriSignatureIdType  unchanged  TriParameterType  unchanged (not being used)  TriParameterPassingModeType  unchanged (not being used)  TriParameterListType  TciParameterListType (by that, the TciParameterPassingMode and Values for parameters will be used)  TriExceptionType  use Value Slide 10 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India New: Advanced TRI 2(3)  Changes to Message-based communication operations  triSend  xtriSend  triSendBC  xtriSendBC  triSendMC  xtriSendMC  triEnqueueMsg  xtriEnqueueMsg  New  Value xtriConvert(in any value, in Type typeHypothesis) This operation shall be called whenever the TE has to convert a value. The TE might convert immediately after reception of the value, or might for performance considerations postpone the conversion until the actual access to the value.  Changes to Procedure-based communication operations  triCall  xtriCall  etc.  Changes to Miscellaneous operations  triExternalFunction  xtriExternalFunction Slide 11 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India TTCN-3 Extension: Continuous signal support 1(4)  This new package defines concepts for testing systems using continuous signals.  The package introduces  notions of time and sampling,  notions of streams, stream ports and stream variables, and  definitions allowing an automaton alike control flow structure supporting the specification of hybrid behavior (i.e., combined definition of discrete and continuous behaviors). Slide 12 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India TTCN-3 Extension: Continuous signal support 2(4)  The following items shall give an impression about the contents of the package. Details and further operations can be found in the package definition.  Concepts for time and sampling:  access to global time: now operator  Setting stepsize for sampling: stepsize operator  Streams  New port type for streams, e.g.,:  type port StreamIn stream { in float }  type port StreamOut stream { out float }  Operations for accessing data streams:  retrieving or setting actual values: value operation  retrieval of time related information: timestamp operation  Retrieval of actual stepsize of a port: delta operation Slide 13 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India TTCN-3 Extension: Continuous signal support 3(4)  Streams (cont.)  navigation on streams  previous event: prev operation  event at a certain point in time: at operation  stream extraction  Retrieval of (partial) history: history operation  Modes  define states in hybrid automata. Modes may  be composed sequential or parallel  have entry and exit conditions  have invariant parts  have a duration  be parameterisable  etc. Slide 14 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India TTCN-3 Extension: Continuous signal support 4(4) 15 Slide 15 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

A Conformance Test Suite for TTCN-3 Tools Bogdan Stanca-Kaposta, Testing Technologies Andras Kovacs, Broadbit Nikolay Pakulin, ISPRAS

TTCN-3 User Conference 2012, Bangelore, India Motivation  Many TTCN-3 compilers (~7 commercial, 2 internal)  High interest in the language  Complex standard: hundreds of pages  No way to check the compliance of TTCN-3 tools  Quality of a test tool is critical  Relevance:  For users:  Is my tool standard compliant?  Can I switch tools easily / vendor lock-in?  For tool vendors:  Where can I improve the standard compliance of my tool?  Possibly later: compliance certification  For standard developers:  Where is need for clarification? Slide 17 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India STF 433  Specialist Task Force initiated by ETSI MTS  Manpower: 3 persons for 100 days total  Dates: November 2011 – Early 2012  Work is based on the existing framework.  Task:  Update the existing TTCN3 ATS to the newer TTCN-3 revision  Extend the conformance test suite for TTCN-3 tools.  Three deliverables:  ETSI TS V1.2.1 ( ) – ICS  ETSI TS V1.2.1 ( ) – TSS & TP  ETSI TS V1.2.1 ( ) – ATS & IXIT  Approved by MTS in May  Downloadable from the ETSI download area right now! Slide 18 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Methodology (1/7)  Main steps: 1.Validate existing tests against new revision of the standard. 2.Develop conformance tests.  Follow the priority list discussed with MTS and vendors. 3.Develop adapters for selected tools. Validate and fix tests. 4.Automate test execution. Slide 19 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Methodology (2/7)  Adjusting Conformance ATS from TTCN3 rev to rev :  Was: 733 tests, 130 pp. of test purposes  Validate that test purposes still hold  Update modified test purposes  Revoke outdated tests  Update ATS folder structure  Approximately 30% of the STF 433 effort  New tests development  Clause priority list:  Not all clauses can be covered. Selection was made: Vendor feedback + most basic functionality.  There is no perfect selection! Slide 20 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Methodology (3/7)  Development of conformance tests:  The ATS is not the test automation, but the test input!  In most cases: 1 module = 1 conformance test.  Test automation provided for certain vendors  Elvior and Testing Technologies  Test suite structure: organized according to clauses.  Expected output for verdict determination: pass accept/reject [expectedoutput]  Examples: pass reject pass accept, ttcn3verdict:pass pass accept, noexecution  Purpose specification: documentreference, description  Documentreference: part:clause  Examples: 1:5, Ensure that when the IUT loads a module containing some definitions before the module declaration then the module is rejected. Slide 21 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Methodology (4/7)  Development of conformance tests (cont.):  Complete TTCN-3 test input example: /*************************************************** 1:5.2.2, Ensure that the IUT correctly handles the uniqueness of variable names in its scope. pass accept, ttcn3verdict:pass ***************************************************/ module Sem_050202_Uniqueness_001 { import from Sem_050202_Uniqueness_001_import { const all; } type component GeneralComp {} function f_funcScope() { var boolean repeatedIdentifier := true; if(repeatedIdentifier==true) { setverdict(pass); } } testcase TC_Sem_050202_Uniqueness_001() runs on GeneralComp { f_funcScope(); } control { execute(TC_Sem_050202_Uniqueness_001()); } } module Sem_050202_Uniqueness_001_import { // repeated indentifier // from imported module // is allowed const integer repeatedIdentifier := 0; } Slide 22 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Methodology (5/7)  Creation of documentation:  Framework already provided by STF393 (Proforma).  Little time for documentation.  Working on a single source saves time.  Solution:  Write test purposes and expected tool outputs directly into the test inputs.  Generate tables for the documentation from the ATS:  Test purpose tables.  Test suite structure.  Much easier to maintain!  Documentation generation tools developed as part of STF409. Slide 23 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Methodology (6/7)  Validation and fixing:  Validation and fixing took more than half of the development time!  STF internal validation with 2 tools:  TTworkbench: compile-time and execution.  TestCast: compile-time and execution.  Developed necessary adapters  Loopback, proc. call  Lots of feedback from TTech and Elvior! Slide 24 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Methodology (7/7)  Executing the ATS: ATS Sem_060101_TopLevel_001.ttcn NegSem_060303_component_types_001.ttcn … TTCN-3 Tool (SUT) TTCN-3 File Sem_060101_TopLevel_001.ttcn Tool output Rejection as invalidRejection to execute Execution with results (e.g., pass/inconc/fail) Tool output evaluation pass accept, ttcn3verdict:pass failpass Test input Expected output Slide 25 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Discussion  Compliance does not imply safety from a vendor lock-in!  Testing is never complete!  With v1.1.1 of the ATS, about 32% of the core standard clauses are covered with at least one test.  Ideally, we should have many tests for each clause.  Compliance means less likelihood for vendor lock-in.  With better coverage and higher number of test cases, we improve the quality of this statement.  Compliance reduces the likelihood for major tool errors  Tests range from simple arithmetic operations to communication operations and verdict handling.  ATS checks syntax and semantics.  ATS does not check the correctness of every single tool message, but of the overall output. TTCN-3 Tool TTCN-3 Conformance ATS Meaning? pass Slide 26 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Challenges (1/2)  Various standard clarifications were necessary:  13 CRs for TTCN-3 core languages have been written.  Unclarified test cases have been excluded from the delivery ("on-hold").  Tools sometimes handle the standard differently.  Tests need to be expressed as tool independent as possible.  Test Automation:  No standard command line interface for batch run  Tool-specific adapters  No common test log format  TTCN3 semantics  TTCN3 standard part 1 defines both static and dynamic semantics  No definition of well- and ill- formed programs  What to check at compile time is tool-dependent Slide 27 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Challenges (2/2)  Synchronize ATS with new TTCN3 revisions  Time consuming manual activity  As ATS grows the effort will grow as well  To which core language standard version does the conformance test suite relate to?  How to maintain compatibility across different language versions?  Against which version should the conformance test suite be developed in the future?  Which TTCN3 revision to support  Should we certify newest TTCN3 revision only? Slide 28 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Conclusion  The conformance test suite is available now!  ATS Metrics:  Total number of conformance tests: 1077 (340 new)  20 tests fixed  9 removed since deprecated or duplicates  Number of positive syntactic conformance tests: 138  Number of negative syntactic conformance tests: 82  Number of positive semantic conformance tests: 517  Number of negative semantic conformance tests: 340  Number of clauses (at least partially) covered: 134 (24 new) Slide 29 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India  30 TTCN-3 CONFORMANCE TEST SUITE STF 433 STATUS REPORT Nikolay Pakulin Requirements tracking case study Slide 30 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Case study in a nutshell  4 chapters: total 7,5 pages  12 Declaring messages,  13 Declaring timers,  23 Timer operations,  24 Verdict operations;  56 requirements elicited,  7 requirements refined into 34 test purposes;  covered by 174 test cases  Average 7,5 requirements per page, 3 test cases per requirement Slide 31 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Requirements tracking case study  ISP RAS contributed to the project requirements management tool Requality  Hierarchical grouping of requirements  Requirements elicitation from DOC/HTML documents  Requirements are connected to certain locations in the documents  Requirements locations are highlighted  Requirements refinement through test purposes  Test purposes define specific cases to be tested in order to treat corresponding requirement ‘covered’  Requirements coverage by test cases report  Source document change management (beta)  Update requirements when the text of the standard is modified  Requality is an open-source plugin to eclipse  Integrated with version control software (SVN) Slide 32 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Sample view: Requality in TTWorkbench Slide 33 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India /*************************************************** STF :12, Ensure timer can be declared in components pass accept, ttcn3verdict:pass ***************************************************/ /* * #reqname /Requirements/12 Declaring timers/Timers can be declared in component type **/ module Sem_12_toplevel_timer_001 { type component TComp{ timer t_timer:=10.0; } testcase TC_Sem_12_toplevel_timer_001() runs on TComp{ t_timer.start; if (t_timer.running){ setverdict(pass) } else { setverdict(fail) } control{ execute(TC_Sem_12_toplevel_timer_001()) } Requirements Tracking (1/2) Fully qualified name of the requirement Slide 34 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Requirements Tracking (1/2)  Requirements coverage report  Integrated into Eclipse  Evaluates coverage of requirements and test purposes Slide 35 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 User Conference 2012, Bangelore, India Suggested use case  Add requirements markup to TTCN3 standard  Manual  Introduce references between requirements and TTCN3 test cases  Manual  Track requirements changes between TTCN3 standard revisions  Automated (beta)  Track required changes in TTCN3 ATS  Manual, to be automated TTCN3 standard updated Changes in reqs. reviewed New reqs. added ATS updated Slide 36 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

Thank You! Questions? Slide 37 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing