1. TTCN CONFORMANCE TEST SUITE TTCN-3 Part 1 & Part 9 STF 487 MILESTONE 2
Bogdan Stanca-Kaposta/ Testing Technologies
Tomas Urban / Elvior
Andras Kovacs / Broadbit

2. Contents
Introduction
Results
Technical details
Proposed Method
ATS

3. Objectives Detect changes of the TTCN-3 Standard
Part
Part 9 (XML Schema) 4.6.1
Cover new/changed Part 1 and Part 9 Features
Upgrade test execution tools
pre-processing, test adaptation
Validation of tests with at least two TTCN-3 Compilers
Tool vendors feedback (Elvior, Testing Technologies)
Raising of CRs
Document and publish the results
ICS, TSS/TP, IXIT, ATS, Progress Reports

4. Objectives Detect changes of the TTCN-3 Standard (done)
Part
Part 9 (XML Schema) 4.6.1
Cover new/changed Part 1 and Part 9 Features (wip)
Upgrade test execution tools (nearly done)
pre-processing, test adaptation
Validation of tests with at least two TTCN-3 Compilers
Tool vendors feedback (Elvior, Testing Technologies) (wip)
Raising of CRs (open)
Document and publish the results (draft done)
ICS, TSS/TP, IXIT, ATS, Progress Reports

5. Results Part 1 4.7.1 (184 new tests, 15 modified)
84 (25%) changed sections (±36% done)
Enhanced converage of older sections
Part 9 (XML Schema) 4.6.1
11 (8%) changed sections (±10% done)
Administrative work
Updated SVN structure to ETSI standards
Compacted ATS folder structure to match windows 260 chars path limitations
Execution tools updated
Documents & reporting
Effort
36 days used out of 100 by end of May 2015
Updating existing tests to new requirements is the most of the work

6. Validation 4 teams Issues
Team members: Testing Technologies, Elvior (wip)
External: IBM, Devoteam
Issues
Revalidating sections we usually find untested requirements, tests exists but not all details are tested
1 bug in XmlDiff –1 acknowledged
Caused by incorrect handling of xsi:type attribute in Xerces XML processor – bug report filed, not fixed yet
Triggers on 2 TCs for union type
Hampered 2 vendors

7. TECHNICAL DETAILS

8. Part 1: Proposed Method ATS = ∑ test cases Test case = TTCN3 script
SUT = TTCN-3 tool
SUT executes TTCN3 script and
Produces some output: verdict, log
Output is validated against expected
SUT launch, execution of tests, collecting outputs is vendor specific.

9. Part 1: Proposed Method TTCN-3 ATS Test case _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _
TTCN-3 Tool (SUT)
_ _ _ _ _ _ _ _ _ _ _ _
Expected
output
Invalid
TTCN-3
Execution
results
Execution
error
Evaluation
PASS
FAIL

10. Overview: using XML Schema with TTCN3
Direct import of definitions of types, elements, attributes as TTCN3 types from XML Schema Definition (XSD) file
Automagic transformation
Inbound XML message -> TTCN3 template
Sending TTCN3 template -> Outbound XML message
ES “TTCN-3: Using XML schema with TTCN-3”
references
XSD
imports
TTCN-3
uses
XML
ATS
Test case
produces

11. Part 9: Proposed Method ATS = ∑ test cases
Test case = TTCN3 script + XSD to import + reference XML
Reference XML to validate encoding result
SUT = TTCN-3 tool + XSD import facility + XML codec
SUT executes TTCN3 script and
Produces some output: verdict, log
Output is validated against expected
SUT launch, execution of tests, collecting outputs is vendor specific.

12. Part 9: Proposed Method XSD TTCN-3 XML ATS Test case
references
XSD
imports
TTCN-3
uses
XML
ATS
Test case
_ _ _ _ _ _ _ _ _ _ _
TTCN-3 Tool (SUT)
_ _ _ _ _ _ _ _ _ _ _ _
Expected
output
Invalid
XSD,TTCN-3
Execution
results
Execution
error
Evaluation
PASS
FAIL

13. Part 1 & 9: Proposed method
Expected SUT output is encoded in the header comment of TTCN-3 script:
E. pass accept, ttcn3verdict:pass
Same as TTCN-3 Conformance Test Suite
Reused tools to execute SUT & validate outputs
validation framework from TTCN-3 Conformance TS

14. Test case construction
If matches test case PASS, otherwise FAIL
Types
XSD
TTCN-3
Template
XML Data
Decoded
Value
references
encode
decode
XML Compare
XML
references
Positive test: valid TTCN-3, valid XSD, must compile & execute with verdict PASS
Import definitions from XSD
Construct a template; encode template into string
Compare string with the reference XML document
Decode string
Check whether decoded value matches original template

15. Test case construction
XSD
TTCN-3
Template
Test Case
FAIL
references
encode
Types
Invalid
XSD and TTCN-3 Template not compatible
Test Case is PASS
Negative test: valid TTCN-3, valid XSD, must compile & execute with verdict ERROR
Import definitions from XSD
Construct a template;
encode template into string
Encoding must fail -> TTCN-3 verdict error
If encoding succeeds, it is an SUT error

16. Sample TTCN-3 Script
module Pos_060101_length_001 { import from schema_Pos_060101_length_001 language "XSD" all; /* target NS: “schema:Pos_060101_length_001” */ template E1 m_msg := "length_010"; /* Aux. definitions */ testcase TC_Pos_060101_length_001() runs on C system C { /* loopback: sends back string with encoded XML*/ map(self:p, system:p); p.send(m_msg); /* encode */ alt { /* The actual matching happens here */ }

17. Sample TTCN-3 Script: alt
alt { // compare the encoded message with the XML file [] p.check(receive(universal charstring:?) -> value v_rcv) { if (matchFile(v_rcv, "Pos_060101_length_001.xml", { "Pos_060101_length_001.xsd" }, v_matchError) { alt { // match decoded value to pass test [] p.receive(m_msg) { setverdict(pass, "Decoded value matches"); } [] p.receive { setverdict(fail, "XML decoding failure"); } } else { setverdict(fail, v_matchError); } [] p.receive { setverdict(fail, “Failed to decode XML"); }

18. Sample XSD
<?xml version="1.0" encoding="UTF-8"?> <schema xmlns= targetNamespace="schema:Pos_050101_namespaces_001“ xmlns:ns="schema:Pos_050101_namespaces_001"> <element name="MyType" type="integer"/> </schema>

19. XML Comparison Two XML files are considered equal if:
Tree of element nodes are equal, and qualified names of elements are the same for corresponding nodes.
NOTE: take care about for unordered subtrees corresponding to choice
Set of attributes for each element node are equal. Qualified names of attributes must match.
Literal values of attributes must match.
float type are compared by their numerical values.
Textual contents of elements must match, whitespaces are ignored. CDATA is considered equal to plain text with equal contents.
Comments and processing instructions are ignored.
Invoked as external function
Reference implementation is based on XmlUnit toolkit, written in Java.