1 TTCN CONFORMANCE TEST SUITE TTCN-3 Part 1 & Part 9 STF 487 Bogdan Stanca-Kaposta/ Testing Technologies Tomas Urban / Elvior Andras Kovacs / Broadbit

Contents Introduction Results Technical details – Proposed Method – ATS 2

Objectives Detect changes of the TTCN-3 Standard – Part – Part 9 (XSD/XML Schema) Cover new/changed Part 1 and Part 9 Features Upgrade test execution tools – pre-processing, test adaptation Validation of tests with at least two TTCN-3 Compilers – Tool vendors feedback (Elvior, Testing Technologies) – Raising of CRs Document and publish the results – ICS, TSS/TP, IXIT, ATS, Progress Reports 3

Objectives Detect changes of the TTCN-3 Standard (done) – Part – Part 9 (XSD/XML Schema) Cover new/changed Part 1 and Part 9 Features (done) Upgrade test execution tools (done) – pre-processing, test adaptation Validation of tests with at least two TTCN-3 Compilers – Tool vendors feedback (Elvior, Testing Technologies) ( done ) – Raising of CRs ( done ) Document and publish the results (draft done) – ICS, TSS/TP, IXIT, ATS, Progress Reports 4

Results Part (now 2956, 1221 new, reviewed tests in the changed sections) – 84 (25%) changed sections (done) – Enhanced converage of older sections (done) Part 9 (XML Schema) (now 426, 79 new, reviewed more than half) – 11 (8%) changed sections (done) – Enhanced converage of older sections (done) Change requests – 37 CRs filled to Evolution STF – 20 CRs Processed by STF 487 Administrative work – All open CRs fixed (10 from previous STF + 10 STF487 ) – Updated SVN structure to ETSI standards – Compacted ATS folder structure to match windows 260 chars path limitations – Execution tools updated – Documents & reporting Effort – Spent more time than the 100 days – Updating existing tests to new requirements 40% – Creating new tests for untested requirements 55% – Documentation & reporting ±5% 5

Validation 3 teams – Team members: Testing Technologies, Elvior (wip) – External: IBM, CRs Issues – While revalidating sections we usually find untested requirements, tests exists but not all details are tested – (STF475) 1 bug in XmlDiff –1 acknowledged Caused by incorrect handling of xsi:type attribute in Xerces XML processor – bug report filed, not fixed yet Triggers on 2 TCs for union type Hampered 2 vendors 6

TECHNICAL DETAILS 7

Part 1: Proposed Method ATS = ∑ test cases Test case = TTCN3 script SUT = TTCN-3 tool – SUT executes TTCN3 script and – Produces some output: verdict, log – Output is validated against expected SUT launch, execution of tests, collecting outputs is vendor specific. 8

Part 1: Proposed Method 9 TTCN-3 ATS Test case TTCN-3 Tool (SUT) Evaluation Execution results Execution error Invalid TTCN-3 FAILPASS _ _ _ _ _ _ _ _ _ _ _ _ Expected output

Overview: using XML Schema with TTCN3 Direct import of definitions of types, elements, attributes as TTCN3 types from XML Schema Definition (XSD) file Automagic transformation – Inbound XML message -> TTCN3 template – Sending TTCN3 template -> Outbound XML message ES “TTCN-3: Using XML schema with TTCN-3” 10 XSDTTCN-3 importsuses references ATS Test case XML produces

Part 9: Proposed Method ATS = ∑ test cases Test case = TTCN3 script + XSD to import + reference XML – Reference XML to validate encoding result SUT = TTCN-3 tool + XSD import facility + XML codec – SUT executes TTCN3 script and – Produces some output: verdict, log – Output is validated against expected SUT launch, execution of tests, collecting outputs is vendor specific. 11

Part 9: Proposed Method 12 XSDTTCN-3 importsuses references ATS Test case TTCN-3 Tool (SUT) Evaluation Execution results Execution error Invalid XSD,TTCN-3 FAILPASS _ _ _ _ _ _ _ _ _ _ _ _ XML Expected output

Part 1 & 9: Proposed method Expected SUT output is encoded in the header comment of TTCN-3 script: – E. pass accept, ttcn3verdict:pass – Same as TTCN-3 Conformance Test Suite Reused tools to execute SUT & validate outputs – validation framework from TTCN-3 Conformance TS 13

Test case construction Positive test: valid TTCN-3, valid XSD, must compile & execute with verdict PASS – Import definitions from XSD – Construct a template; encode template into string – Compare string with the reference XML document – Decode string – Check whether decoded value matches original template 14 referencesencode Types TTCN-3 Template XSD XML Data Decoded Value decode If matches test case PASS, otherwise FAIL XML references XML Compare

Test case construction Negative test: valid TTCN-3, valid XSD, must compile & execute with verdict ERROR – Import definitions from XSD – Construct a template; – encode template into string – Encoding must fail -> TTCN-3 verdict error – If encoding succeeds, it is an SUT error 15 referencesencode Invalid XSD and TTCN-3 Template not compatible Test Case is PASS Test Case FAIL Types TTCN-3 Template XSD

Sample TTCN-3 Script module Pos_060101_length_001 { import from schema_Pos_060101_length_001 language "XSD" all; /* target NS: “schema:Pos_060101_length_001” */ template E1 m_msg := "length_010"; /* Aux. definitions */ testcase TC_Pos_060101_length_001() runs on C system C { /* loopback: sends back string with encoded XML*/ map(self:p, system:p); p.send(m_msg); /* encode */ alt { /* The actual matching happens here */ } 16

Sample TTCN-3 Script: alt alt { // compare the encoded message with the XML file []p.check(receive(universal charstring:?) -> value v_rcv) { if (matchFile(v_rcv, "Pos_060101_length_001.xml", { "Pos_060101_length_001.xsd" }, v_matchError) { alt { // match decoded value to pass test [] p.receive(m_msg) { setverdict(pass, "Decoded value matches"); } [] p.receive { setverdict(fail, "XML decoding failure"); } } else { setverdict(fail, v_matchError); } } [] p.receive { setverdict(fail, “Failed to decode XML"); } } 17

Sample XSD <schema xmlns= targetNamespace="schema:Pos_050101_namespaces_001“ xmlns:ns="schema:Pos_050101_namespaces_001"> 18

XML Comparison Two XML files are considered equal if: – Tree of element nodes are equal, and qualified names of elements are the same for corresponding nodes. NOTE: take care about for unordered subtrees corresponding to choice – Set of attributes for each element node are equal. Qualified names of attributes must match. – Literal values of attributes must match. float type are compared by their numerical values. – Textual contents of elements must match, whitespaces are ignored. CDATA is considered equal to plain text with equal contents. float type are compared by their numerical values. – Comments and processing instructions are ignored. Invoked as external function Reference implementation is based on XmlUnit toolkit, written in Java. 19