TTCN CONFORMANCE TEST SUITE TTCN-3 Part 1 & Part 9 STF 487 MILESTONE 2 Bogdan Stanca-Kaposta/ Testing Technologies Tomas Urban / Elvior Andras Kovacs / Broadbit

Contents Introduction Results Technical details Proposed Method ATS

Objectives Detect changes of the TTCN-3 Standard Part 1 4.7.1 Part 9 (XML Schema) 4.6.1 Cover new/changed Part 1 and Part 9 Features Upgrade test execution tools pre-processing, test adaptation Validation of tests with at least two TTCN-3 Compilers Tool vendors feedback (Elvior, Testing Technologies) Raising of CRs Document and publish the results ICS, TSS/TP, IXIT, ATS, Progress Reports

Objectives Detect changes of the TTCN-3 Standard (done) Part 1 4.7.1 Part 9 (XML Schema) 4.6.1 Cover new/changed Part 1 and Part 9 Features (wip) Upgrade test execution tools (nearly done) pre-processing, test adaptation Validation of tests with at least two TTCN-3 Compilers Tool vendors feedback (Elvior, Testing Technologies) (wip) Raising of CRs (open) Document and publish the results (draft done) ICS, TSS/TP, IXIT, ATS, Progress Reports

Results Part 1 4.7.1 (184 new tests, 15 modified) 84 (25%) changed sections (±36% done) Enhanced converage of older sections Part 9 (XML Schema) 4.6.1 11 (8%) changed sections (±10% done) Administrative work Updated SVN structure to ETSI standards Compacted ATS folder structure to match windows 260 chars path limitations Execution tools updated Documents & reporting Effort 36 days used out of 100 by end of May 2015 Updating existing tests to new requirements is the most of the work

Validation 4 teams Issues Team members: Testing Technologies, Elvior (wip) External: IBM, Devoteam Issues Revalidating sections we usually find untested requirements, tests exists but not all details are tested 1 bug in XmlDiff –1 acknowledged Caused by incorrect handling of xsi:type attribute in Xerces XML processor – bug report filed, not fixed yet Triggers on 2 TCs for union type Hampered 2 vendors

TECHNICAL DETAILS

Part 1: Proposed Method ATS = ∑ test cases Test case = TTCN3 script SUT = TTCN-3 tool SUT executes TTCN3 script and Produces some output: verdict, log Output is validated against expected SUT launch, execution of tests, collecting outputs is vendor specific.

Part 1: Proposed Method TTCN-3 ATS Test case _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ TTCN-3 Tool (SUT) _ _ _ _ _ _ _ _ _ _ _ _ Expected output Invalid TTCN-3 Execution results Execution error Evaluation PASS FAIL

Overview: using XML Schema with TTCN3 Direct import of definitions of types, elements, attributes as TTCN3 types from XML Schema Definition (XSD) file Automagic transformation Inbound XML message -> TTCN3 template Sending TTCN3 template -> Outbound XML message ES 201 873-9 “TTCN-3: Using XML schema with TTCN-3” references XSD imports TTCN-3 uses XML ATS Test case produces

Part 9: Proposed Method ATS = ∑ test cases Test case = TTCN3 script + XSD to import + reference XML Reference XML to validate encoding result SUT = TTCN-3 tool + XSD import facility + XML codec SUT executes TTCN3 script and Produces some output: verdict, log Output is validated against expected SUT launch, execution of tests, collecting outputs is vendor specific.

Part 9: Proposed Method XSD TTCN-3 XML ATS Test case references XSD imports TTCN-3 uses XML ATS Test case _ _ _ _ _ _ _ _ _ _ _ TTCN-3 Tool (SUT) _ _ _ _ _ _ _ _ _ _ _ _ Expected output Invalid XSD,TTCN-3 Execution results Execution error Evaluation PASS FAIL

Part 1 & 9: Proposed method Expected SUT output is encoded in the header comment of TTCN-3 script: E. g.: @verdict pass accept, ttcn3verdict:pass Same as TTCN-3 Conformance Test Suite Reused tools to execute SUT & validate outputs validation framework from TTCN-3 Conformance TS

Test case construction If matches test case PASS, otherwise FAIL Types XSD TTCN-3 Template XML Data Decoded Value references encode decode XML Compare XML references Positive test: valid TTCN-3, valid XSD, must compile & execute with verdict PASS Import definitions from XSD Construct a template; encode template into string Compare string with the reference XML document Decode string Check whether decoded value matches original template

Test case construction XSD TTCN-3 Template Test Case FAIL references encode Types Invalid XSD and TTCN-3 Template not compatible Test Case is PASS Negative test: valid TTCN-3, valid XSD, must compile & execute with verdict ERROR Import definitions from XSD Construct a template; encode template into string Encoding must fail -> TTCN-3 verdict error If encoding succeeds, it is an SUT error

Sample TTCN-3 Script module Pos_060101_length_001 { import from schema_Pos_060101_length_001 language "XSD" all; /* target NS: “schema:Pos_060101_length_001” */ template E1 m_msg := "length_010"; /* Aux. definitions */ testcase TC_Pos_060101_length_001() runs on C system C { /* loopback: sends back string with encoded XML*/ map(self:p, system:p); p.send(m_msg); /* encode */ alt { /* The actual matching happens here */ }

Sample TTCN-3 Script: alt alt { // compare the encoded message with the XML file [] p.check(receive(universal charstring:?) -> value v_rcv) { if (matchFile(v_rcv, "Pos_060101_length_001.xml", { "Pos_060101_length_001.xsd" }, v_matchError) { alt { // match decoded value to pass test [] p.receive(m_msg) { setverdict(pass, "Decoded value matches"); } [] p.receive { setverdict(fail, "XML decoding failure"); } } else { setverdict(fail, v_matchError); } [] p.receive { setverdict(fail, “Failed to decode XML"); }

Sample XSD <?xml version="1.0" encoding="UTF-8"?> <schema xmlns=http://www.w3.org/2001/XMLSchema targetNamespace="schema:Pos_050101_namespaces_001“ xmlns:ns="schema:Pos_050101_namespaces_001"> <element name="MyType" type="integer"/> </schema>

XML Comparison Two XML files are considered equal if: Tree of element nodes are equal, and qualified names of elements are the same for corresponding nodes. NOTE: take care about for unordered subtrees corresponding to choice Set of attributes for each element node are equal. Qualified names of attributes must match. Literal values of attributes must match. float type are compared by their numerical values. Textual contents of elements must match, whitespaces are ignored. CDATA is considered equal to plain text with equal contents. Comments and processing instructions are ignored. Invoked as external function Reference implementation is based on XmlUnit toolkit, written in Java.