1. Automated Interoperability Testing
ETSI STF 370 Status
Stephan Schulz STF Leader
© ETSI All rights reserved
MTS#49, Rome, Italy

2. Project Background Project funded by EC and ETSI Main stakeholders
Objective is to extend existing ETSI interoperability testing concepts for the purpose of automation and in context of distributed systems
Main stakeholders
ETSI TC Methods for Testing and Specification (owner of ETSI WIs)
ETSI TC IMS Network Testing
ETSI TC GRID
B2B community (mainly around HL7)
TETRA Association
WiMax Forum (NWIIOT)
Further signalled interest
IPv6 community
ITS community
Testing labs

3. Project Overview Planned duration Jan 2009 to Jun 2010
12 experts with various background led by ETSI CTI
WP1: Methodology and Framework
Output is ETSI Guide
WP2: IMS case study based on use of TTCN-3
Application of WP1 in context of IMS core network IOT
Basis is ETSI TC INT test specification for 3rd ETSI IMS Plugtest
Output is ETSI Technical Report (ATS document), TTCN-3 ATS, codecs & adapters, ETSI TR on Plugtest experience
IMS IOT tool and auto IOT concepts will be validated at IMS Plugtest!
WP3: Dissemination
White paper & training material
Presentation to TETRA Forum and at T3UC T3UC Asia 2009

4. WP1 Status
First draft of methodology produced by looking at various areas
IOT in IMS, WiMax, IPv6, HL7, ROHC, IPTV, WiMedia, SIP VoIP, etc
Draft sent out for review and comments
Draft expected to further evolve during project
Methodology extends ETSI’s generic approach to IOT (EG )
Adds aspects of automation and test system implementation
Main points for now
Independent of technology to be tested
Collection of key terminology
Separate verdicts for end-to-end and conformance assessment
Limitations, feasibility, and degree of automation
Controllability of Equipment Under Test (EUT) interfaces
Definition of generic means of interoperability testing
Definition of test system development (independent of TTCN-3)

5. Generic Auto IOT Test Architecture
Means of
Interoperability Testing
EUT 1
Interface
Monitor
Test Coordinator
Configuration interface
Application Support Nodes
Monitoring interface
SUT
Test Oracle
EUT 2
EUT N
Interconnecting Network
Stimulating interface
Equipment User

6. WP2 Status Case study on IMS core network IOT ongoing
Stable draft of IMS architecture documentation
Draft sent out for review and comments
TTCN-3 ATS specification phase completed
Completed implementation TTCN-3 framework
Library based test design
Handling of EUT information in IOT event context
En/disabling of interface checks upon need
Separate management of CON and IOP verdicts
Real-time vs. offline IOT test execution
Essentially completed specification of stable TTCN-3 IMS IOP tests
Current TTCN-3 implementation status
93 % done (but 14 tests marked for now as “skip” as not stable)
Number of issues reported to TC INT on IMS IOP test specification
Number of tests have been validated

7. WP2 Status (contd.) TCI SIP and SDP codec implementation completed
Based on open source IRISA t3dev codec development kit
Excludes handling of XML (only required by 2 tests)
Validated with Elvior tool
TCI DNS codec implementation “not started”
In principle already available
Competing proposals for DNS TTCN-3 type system (TT vs. IRISA)
Best type system compromise in the middle – but then no codec 
For now issue on hold (SIP checks take precedence for now)
TCI ISUP codec implementation not started
Very complex type structure and only very basic checks in tests
Unclear test adapter tool support since not Ethernet based
No resource for this activity planned in project plan
Required resource for codec implementation not available

8. WP2 Status (contd.) Test adapter implementation completed
Based on open source IRISA t3dev codec development kit
Protocol independent, extensible C++ implementation including GUI based upper tester, lower tester, and PCAP capture process
Designed test adapter protocol and implemented C++ codecs for test adapter protocol
Only been able to integrate with Elvior tool
Cygwin dependency of t3devkit has been very difficult to handle with tools
“old” TTWB adapter updated to fit new adapter interface

9. Road ahead
Use of STF370 IMS test system as part of ETSI TC INT IMS Plugtest in mid Oct
Using Elvior and TT WB based test system implementations
Two experts to attend for test system support in context of WP2 to make updates to TTCN-3 code, codec, and adapters
Two experts to attend for Plugtest observation in context of WP3
Write experience report after event (WI)
Updates of methodology and IMS architecture documents
First part of WP3 STF sessions planned to start in Nov
Drafting of White paper & Training material (WI)

10. Topics for discussion with MTS
STF 370 comments and questions on EG (49TD10)
ISUP adaptation support
Adjusting the schedule of dissemination WIs reflecting the project timelines

11. Related documents
WI DTR/MTS-00116: “Methods for Testing and Specification (MTS); Automated Interoperability Testing; Specific Architectures”
Uploaded as 49TD09r1
WI DEG/MTS-00119: Methods for Testing and Specification (MTS); Automated Interoperability Testing; Methodology and Framework
Uploaded as 49TD12r1
TTCN-3 code and design documents will be made available after the IMS Plugtest (for next meeting)

12. Thank you Questions? Comments?