1. A conformance test suite for TTCN-3 tools Benjamin Zeiss, T-Systems Andras Kovacs, Broadbit Bogdan Stanca-Kaposta, Testing Technologies

2. Slide 2 Agenda  Motivation  STF 409  Methodology  Discussion  Challenges  Conclusion A conformance test suite for TTCN-3 tools

3. Motivation (1/2)  Many TTCN-3 compilers (~7 commercial, 2 internal)  High interest in the language  Complex standard: hundreds of pages  Estimation of ~5000 requirements  No way to check the compliance of TTCN-3 tools  TTCN-3 is based on ETSI standards  A conformance test suite for TTCN-3 tools is needed!  Quality of a test tool is critical  Errors in the test tool may influece test results. Slide 3 A conformance test suite for TTCN-3 tools

4. Motivation (2/2)  Relevance:  For users:  Is my tool standards compliant?  Can I switch tools easily / vendor lock-in?  For tool vendors:  Can I improve the standard compliance of my tool?  Possibly later: compliance certification  For standard developers:  Where is need for clarification? Slide 4 A conformance test suite for TTCN-3 tools

5. STF 409 (1/2)  Specialist Task Force initiated by ETSI MTS  Manpower: 3 persons w. 20 days each (60 days total)  Benjamin Zeiss (T-Systems/University of Göttingen)  Andras Kovacs (Broadbit)  Bogdan Stanca-Kaposta (Testing Technologies)  Dates: October 2010 – Early 2011  Work is based on a framework given by TTCN-3 STF.  Task:  Devel. of a conformance test suite for TTCN-3 tools.  Creation of a thorough documentation for the ATS. Slide 5 A conformance test suite for TTCN-3 tools

6. STF 409 (2/2)  Three deliverables:  ETSI TS 102 950-1 V1.1.1 (2011-04) – ICS  ETSI TS 102 950-1 V1.1.1 (2011-04) – TSS & TP  ETSI TS 102 950-2 V1.1.1 (2011-04) – ATS & IXIT  Approved by MTS in April 2011.  Downloadable from the ETSI download area right now!  Time spent:  10 days each (30 work days)  Development of conformance tests.  10 days each + extra time (30+ work days):  Validation, fixing of problems, documentation, tool development. Slide 6 A conformance test suite for TTCN-3 tools

7. Methodology (1/7)  Five main steps: 1.Create clause priority list. Discuss with MTS. 2.Develop conformance tests. 3.Create documentation framework. 4.Generate documentation content. 5.Validate and fix.  Step 1 (creation of clause priority list):  Not all clauses can be covered. Selection has to be made.  How to prioritize?  Vendor feedback + most basic functionality.  There is no perfect selection! Slide 7 A conformance test suite for TTCN-3 tools

8. Methodology (2/7)  Step 2 (development of conformance tests):  The ATS is not the test automation, but the test input!  In most cases: 1 module = 1 conformance test.  Two steps: 1.Specification of the test purpose: @purpose + @verdict 2.Implementation of the test in TTCN-3.  Conformance verdict:  Expected tool output is specified in the document tags (@verdict).  Purpose:  Also specified in the document tags (@purpose).  Test automation is not provided by the ATS!  Can be scripted easily by extracting the document tags.  Example Java source code can be provided!  Test suite structure: organized according to clauses. Slide 8 A conformance test suite for TTCN-3 tools

9. Methodology (3/7)  Step 2 (development of conformance tests) cont.:  Expected output for verdict determination:  @verdict pass accept/reject [expectedoutput]  Examples:  @verdict pass reject  @verdict pass accept, ttcn3verdict:pass  @verdict pass accept, noexecution  Purpose specification:  @purpose documentreference, description  Documentreference: part:clause  Examples:  @purpose 1:5, Ensure that when the IUT loads a module containing some definitions before the module declaration then the module is rejected. Slide 9 A conformance test suite for TTCN-3 tools

10. Methodology (4/7)  Step 2 (development of conformance tests) cont.:  Complete TTCN-3 test input example: Slide 10 A conformance test suite for TTCN-3 tools /*************************************************** ** @purpose 1:5.2.2, Ensure that the IUT correctly handles the uniqueness of variable names in its scope. ** @verdict pass accept, ttcn3verdict:pass ***************************************************/ module Sem_050202_Uniqueness_001 { import from Sem_050202_Uniqueness_001_import { const all; } type component GeneralComp {} function f_funcScope() { var boolean repeatedIdentifier := true; if(repeatedIdentifier==true) { setverdict(pass); } } testcase TC_Sem_050202_Uniqueness_001() runs on GeneralComp { f_funcScope(); } control { execute(TC_Sem_050202_Uniqueness_001()); } } module Sem_050202_Uniqueness_001_import { // repeated indentifier // from imported module // is allowed const integer repeatedIdentifier := 0; }

11. Methodology (5/7)  Step 3 (creation of documentation framework):  Already provided by STF393 (Proforma).  Split single document into three documents.  Step 4 (generate documentation content):  Only 60 days manpower, little time for documentation.  Working on a single source saves time.  Solution:  Write test purposes and expected tool outputs directly into the test inputs.  Generate tables for the documentation from the ATS:  Test purpose tables.  Test suite structure.  Much easier to maintain!  Documentation generation tools developed as part of STF409. Slide 11 A conformance test suite for TTCN-3 tools

12. Methodology (6/7)  Step 5 (validation and fixing):  Validation and fixing took more than half of the development time!  STF internal validation with 4 tools:  TTworkbench: compile-time and execution.  TestCast: compile-time and execution.  IBM: some initial compile-time validation.  MTP: as reports are sent in.  Lots of feedback from Elvior and MTP! Slide 12 A conformance test suite for TTCN-3 tools

13. Methodology (7/7)  Executing the ATS: Slide 13 A conformance test suite for TTCN-3 tools ATS Sem_060101_TopLevel_001.ttcn NegSem_060303_component_types_001.ttcn … TTCN-3 Tool (test subject) TTCN-3 File Sem_060101_TopLevel_001.ttcn Tool output Rejection as invalidRejection to execute Execution with results (e.g., pass/inconc/fail) Tool output evaluation ** @verdict pass accept, ttcn3verdict:pass failpass Test input Expected output

14. Discussion  Compliance does not imply safety from a vendor lock-in!  Testing is never complete!  With v1.1.1 of the ATS, about 32% of the core standard clauses are covered with at least one test.  Ideally, we should have many tests for each clause.  Compliance means less likelihood for vendor lock-in.  With better coverage and higher number of test cases, we improve the quality of this statement.  Compliance reduces the likelihood for major tool errors  Tests range from simple arithmetic operations to communication operations and verdict handling.  ATS checks syntax and semantics.  ATS does not check the correctness of every single tool message, but of the overall output. A conformance test suite for TTCN-3 tools Slide 14 TTCN-3 Tool TTCN-3 Conformance ATS Meaning? pass

15. Challenges (1/2)  Various standard clarifications were necessary:  9 CRs for TTCN-3 core languages have been written.  Unclarified test cases have been excluded from the delivery ("on-hold").  Tools sometimes handle the standard differently.  Tests need to be expressed as tool independent as possible.  Time management:  Proper validation and fixing is extremely time consuming.  More than 10 voluntary days have been actually spent.  Responsibilities:  Everyone mixes up STF393 and STF409 responsibilities (proforma). Slide 15 A conformance test suite for TTCN-3 tools

16. Challenges (2/2)  Version numbers:  To which core language standard version does the conformance test suite relate to?  How to maintain compatibility across different language versions?  Against which version should the conformance test suite be developed in the future?  Target quality:  The target quality of the test suite should be fixed right in the beginning (Terms of Reference). Slide 16 A conformance test suite for TTCN-3 tools

17. Conclusion  The conformance test suite is available now!  ATS Metrics:  Total number of conformance tests: 733  Number of positive syntactic conformance tests: 110  Number of negative syntactic conformance tests: 34  Number of positive semantic conformance tests: 335  Number of negative semantic conformance tests: 254  Total number of clauses in the TTCN-3 standard: 342  Total number of clauses (at least partially) covered: 110  About 140 pages of test purposes.  Free to use for tool developers. Use it to improve your tool!  Bug reports: http://t-ort.etsi.org Slide 17 A conformance test suite for TTCN-3 tools

18. Slide 18 Special thanks: Tool validation and comments Elvior, MTP, Conformiq, Ericsson, ISPRAS Jens Grabowski / University of Göttingen Questions? Feedback? A conformance test suite for TTCN-3 tools