1. LTE CONFORMANCE TESTING: EXPERIENCES IN A LONG TERM PROJECT TTCN-3 User Conference 2011 – Bled, Slovenia © ETSI 2011. All rights reserved Presented by MCC TF160: Wolfgang Seka Authors: Wolfgang Seka, Hellen Griffiths, Shicheng Hu wolfgang.seka@etsi.orgwolfgang.seka@etsi.org, hellen.griffiths@etsi.org, shicheng.hu@etsi.orghellen.griffiths@etsi.orgshicheng.hu@etsi.org

2. LTE Conformance Testing: Experiences in a long term project The Project Technical Issues Common Aspects of Long Term Projects Conclusions 2 © ETSI 2011. All rights reserved

3. THE PROJECT 3 © ETSI 2011. All rights reserved

4. MCC TF160 - General Task Force – Mobile Competence Centre: Project Group at ETSI Pool of TTCN expertise used by 3GPP 3GPP: 3rd Generation Partnership Project (http://www.3gpp.org) Telecommunication Standardisation Bodies TSG RAN: Radio Access Network WG RAN5: Mobile terminal conformance testing Conformance Tests Specification (Prose): RAN5 Implementation (TTCN): MCC TF160 Verification and Validation: Test Industry MCC TF160: Signalling Conformance Tests for 3GPP (RAN5: Testing) Task: Develop Conformance Test Suites for UE world-wide certification since 2000 Conformance Tests for UMTS Signalling (TTCN-2) since 2006 Conformance Tests for IMS (TTCN-3) 2007..2008 Pre-evaluation of TTCN-3 for LTE Signalling 2008..now 3GPP LTE/SAE UE Conformance Test © ETSI 2011. All rights reserved 4

5. MCC TF160 – LTE/SAE Project Size: 17 experts all over the world Duration: since 2008 and at least until 2015 (LTE-advanced will follow) Deliveries2011: 8 deliveries planned Test cases: 264 approved 391 fully implemented 468 in total 80..100 planned for IMS Code size: nearly 200 Modules nearly 190 000 Lines of code (TTCN-3) Type Definitions 25 TTCN-3 modules, 3 ASN.1 modules, 6 XML modules TTCN-3:> 17 500 lines of code ASN.1:> 38 000 lines of code XML:< 1 000 lines of code Tools: 6 different compilers (all available at ETSI) quality checks (naming conventions, template restrictions etc.) code generation (top-level test case definitions, parameters, etc.) code analysis (structure, approved objects) TTCN-3 code is officially published and widely used © ETSI 2011. All rights reserved 5

6. Component Structure - 1 MTC Start of PTCs, monitoring of done and killed RAT independent interfaces (e.g. AT-/MMI-commands to control the UE) In general no pass/fail verdicts PTCs for each RAT (radio access technology) LTE, UMTS, GSM/GPRS, CDMA2000 May be connected to any other RAT PTC Only RAT specific interfaces Assignment of pass/fail verdicts PTCs for other protocols IP data (e.g. DHCP, ICMPv6), IMS and protocol layer underneath LTE PTC (NAS Emulator) To keep system interface simple and deal with parallel behaviour In general no test characteristic ( no pass/fail verdicts) Ports and interfaces Connections: only one-to-one ( no send to, receive from; no use of address data type) No duplicated interfaces: e.g. only one interface for AT-/MMI-commands hosted by the MTC Message based communication only © ETSI 2011. All rights reserved 6

7. Component Structure - 2 7 © ETSI 2011. All rights reserved7 7 System Interface E_SYSIND E_DRB E_SYSE_SRB EUTRA PTC (LTE) NAS Emulator CTRL TC_SRB SYS_SRB SYSIND DRB SYSNASCTRL SRB UT UTRAN GERAN MTC E_Ut Ut G_Ut U_Ut UTRAN PTC C2K_Ut GERAN PTC CDMA2000 PTC UTRAN PTC GERAN PTC CDMA2000 PTC C2K Coordination Upper Tester (AT, MMI) Configuration Signalling User Data

8. SUT Test System © ETSI 2011. All rights reserved 8 Host PC System Simulator HW complex configuration delay of messages no matter what test purpose is UE RF AT/MMI Test Executable system specific System/ Platform Adaptor Codec e.g. requirements regarding real-time behaviour for System Simulator and TTCN-3 code Test Control, Logging

9. TECHNICAL ISSUES 9 © ETSI 2011. All rights reserved

10. Real Time Issues - 1 Requirements Time resolution at the air interface: 1ms Due to significant delay between air interface and test case executable: Timing needs to be monitored/controlled at the air interface (not at the TRI) All test equipment shall have similar behaviour at any time Test results shall not depend on how fast or slow a UE or system simulator is specific combinations of UEs and system simulators System Restrictions Delay by codec and matching: up to some milliseconds; typically < 5ms Host system: typically Windows or Linux no real-time OS Architecture: Host-PC + System Simulator Estimation delay between test executable and air interface: 1ms < delay < 80ms © ETSI 2011. All rights reserved 10

11. Real Time Issues - 2 Impact on System Interface LTE timing information is part of all LTE primitives is provided by the System Simulator for all received messages is used to schedule sending of messages in the system simulator can be retrieved from the system simulator Impact on writing TTCN-3 Scheduled and non-scheduled sending shall not be mixed up When there is no reaction (e.g. from the UE) it is hard to know in the TTCN code when the message has been sent out Scheduling vs. TTCN timers TTCN timers are not as accurate as scheduling Rules are required when to use TTCN timers or scheduled sending Restrictions and Issues No common system time: uncorrelated timing between different RATs Legacy primitive definitions for UMTS and GSM/GPRS: less support of timing information Short duration only: For LTE, wrap-around after 10s No bell mechanism (yet): to schedule a wake-up call in the system simulator to send back a trigger after a given time © ETSI 2011. All rights reserved 11

12. Race Conditions Race conditions occur when messages appear at the same time at different ports messages appear in any order at one port Impact on TTCN implementation and Design considerations TTCN implementation interleave (in general not more than two messages) check operation evaluation of timing information Clever port definitions (few ports rather than many) Component structure (e.g. signalling and data at different PTCs) © ETSI 2011. All rights reserved 12

13. COMMON ASPECTS OF LONG TERM PROJECTS 13 © ETSI 2011. All rights reserved

14. Extendibility Extensions of TTCN-3 code Modification of common objects shall not cause changes in the whole test suite reduced maintenance effort Impact on writing TTCN-3 foresighted TTCN implementation introduction of additional optional parameters Example: © ETSI 2011. All rights reserved 14 template MyType c_MyType := { field1 := 1, field2 := 2 }; template MyType c_MyType(integer p_Int := 1) := { field1 := p_Int, field2 := 2 }; Note: Optional Parameters vs. Modified Templates Optional parameters need to be added to modified templates too modification of templates and addition of optional parameters may serve the same purpose possible conflicts, potential contradiction of both approaches

15. Backward Compatibility - 1 Backward Compatibility at the System Interface Modifications are caused by Release upgrade for signalling messages may have impact on configuration of the system simulator New requirements (due to new test cases) Corrections Backward Compatibility Aspects Old TTCN-3 code on new system simulator implementation New TTCN-3 code on old system simulator implementation Impact system adaptor and codec interface definition top-level records or unions (even when not needed in the first place) definitions of place holders © ETSI 2011. All rights reserved 15

16. Backward Compatibility - 2 Examples: Backward Compatibility at System Interface © ETSI 2011. All rights reserved 16 type union MyUnion_Type := { R8_Type R8 }; type union MyUnion_Type := { R8_Type R8, R9_Type R9 }; New branch: branches are mutual exclusive; requires new templates type record MyRecord_Type := { R8_Type R8Only }; type record MyRecord_Type := { R8_Type R8Only, R9Extension_Type R9Ext optional }; Extension: templates may be enhanced by optional parameter

17. Tools Large, long-term project code reviews and manual checks are not sufficient anymore Requirements Quality checks Approved objects: objects used by approved test cases shall not be changed without change request Analysis of module dependencies Recursive selection Enhanced tool support Quality checks by T3Q (on behalf of ETSI) Common tool to get dependencies based on T3D (on behalf of ETSI) Project-specific front-ends (approved objects, recursive selection) Further project specific tools: replace tabs, find non-ASCII characters in comments © ETSI 2011. All rights reserved 17

18. CONCLUSIONS 18 © ETSI 2011. All rights reserved

19. Conclusions (TTCN-3 language) Important TTCN-3 language features optional parameters backward compatibility, maintenance template restrictions in combination with naming conventions and appropriate tools no problems with receive templates used in send direction anymore encvalue/decvalue maintenance (no external functions needed) check operation, interleave race conditions testcase.stop well-defined termination of a test case in case of unrecoverable error situations visibility: "private", "friend not used yet (may improve quality regarding module dependencies) © ETSI 2011. All rights reserved 19

20. Common Conclusions Feedback of long-terms projects to the core language is vital Tool Compatibility is needed Shortened release cycle of TTCN-3 standards TTCN-3 development is SW Development SW Engineering System Engineering Long-term characteristic of a project needs to be considered from the beginning Conceptual errors will be expensive Tool support is vital Common and project-specific tools © ETSI 2011. All rights reserved 20

21. ADDITIONAL SLIDES 21 © ETSI 2011. All rights reserved

22. Release Upgrades - 1 Release Upgrade for Protocol and Signalling (Baseline Moving) Typically every year In general changes are backward compatible Critical and non-critical extensions (TS 36.331 Annex A) Critical extensions redefinition of message content or parts of it (branching, e.g. r8 and r9 branch) sender needs to know which release the receiver supports Non-critical extensions Provision of additional information e.g. new fields at the end of a record, new enumerated value when not supported receiver shall skip the information Messages in general contain hooks for critical and non-critical extensions © ETSI 2011. All rights reserved 22

23. Release Upgrades - 2 Impact on writing TTCN-3 Non-critical extensions In general no code duplication is necessary Enhancement of existing templates by new optional parameters (initialised with * for receiving and omit for sending) Critical extensions New templates are needed Challenge: minimised code duplication Example: template of a plain RRC PDU for UMTS up to several hundred lines of code clever template structure reduces duplication of code parameterised templates modified templates © ETSI 2011. All rights reserved 23