1. TTCN-3 based Implementation of ETSI TISPAN IMS Benchmark George Din, Diana Vega, Razvan Petre, Andreas Hoffman Fraunhofer Institute for Open Communication Systems

2. TTCN-3 User Conference, 30.05.20072 Contents  Motivation  What do we test?  How do we test?  Implementation details  Benchmark Results  Conclusions

3. TTCN-3 User Conference, 30.05.20073 Motivation  Goal - performance benchmark for IMS components Performance and scalability testing of all IMS and related components with simulated real-world traffic Measurement and analysis of important QoS parameters  Why Creation of objective means to compare overall IMS of different systems by performance (and price) Check ability of hardware/software to run the IMS  How Define standard scenarios and traffic models for the work load Define the metrics to be measured Standardize the test procedure, the test parameters and the Benchmark test report  Where Standardization of IMS benchmarking at ETSI TISPAN WG6 Version 1.0 of IMS benchmarking standard has been released.

4. TTCN-3 User Conference, 30.05.20074 System under Test Charging Functions IP Transfer (Access and Core) MRFP BGCFMRFC SLF MGCF Other IP Networks Resource and Admission Control Subsystem IWF I-BGF IBCF AS PSTN / ISDN SGF T-MGF Network Attachement Subsystem Gm Gq’ MpMn Mx Mg Mj Gq’ Dh Dx Rf / Ro Mw Mx Sh Cx Ut ISC/Ma Iw Ic Ie Ib Mr Mk Mle2 I/S-CSCF UE UPSF IMS Core Fokus Open IMS UE P-CSCF

5. TTCN-3 User Conference, 30.05.20075 Test Report Benchmark Information Model Use case 1 (e.g.: registration) Scenario 1.1 (e.g. user not registered) Message flow Metrics & graphs to collect SuT parameters Design objectives Scenario 1.2 (e.g. already registered) Benchmark Test Traffic Set Preamble Registration Phase SuT Characteristics Traffic Profile Initial load Step duration Step Increase Amount Number Of Steps Other parameters Use case 2 (e.g.: session setup) Scenario 2.1 (e.g. successful call) Message flow Metrics & graphs to collect SuT parameters Design objectives Scenario 2.2 (e.g. rejected call) Background Load X% of scenario 1.1 y% of scenario 2.1 z% of scenario 2.2 DisclaimerBenchmark Test Run SuT configuration and parameters Metrics and graphics with Design Objective Capacity (overall and for each scenario) Test System configuration and parameters Observations and interpretations Exceptions Metrics and graphics with SuT resource consumption

6. TTCN-3 User Conference, 30.05.20076 Scenario: Successful Call INVITE 100 Trying INVITE 180 Ringing200 OK180 Ringing200 OK ACK BYE 200 OK REGISTER 401 REGISTER 200 OK RegistrationSection Tear Up Tear Down Client 1IMS CN REGISTER 401 REGISTER 200 OK Client 2IMS CN Client1Client2CN Talking Time Ringing Time

7. TTCN-3 User Conference, 30.05.20077 Scenario: Successful Call INVITE 100 Trying INVITE 180 Ringing200 OK180 Ringing200 OK ACK BYE 200 OK Section Tear Up Tear Down Client1 Client2 CN TRT-SES1 TRT-REL1 TRT-SES2 Talking Time Ringing Time

8. TTCN-3 User Conference, 30.05.20078 Estimated Maximum Load No Load Initial Load Initial Call Load Stop Condition Met Time To Stop Condition Plus Load Plus Load Plus Load Ramp Load Steady Plus Load Stability Or Settling Time Linear, Immediate? Stability? Rate Increase Fluctuations, Steady state? Fluctuations, Steady state? Benchmark Procedure

9. TTCN-3 User Conference, 30.05.20079 TTCN-3 ATS Architecture Scenario-Independent Scenario-dependant User State Machine User State Machine Imports from Benchmar k Testcase Benchmar k Testcase Module for Instant Messaging Scenarios Data Templates Module for Registration Scenarios Data Templates Module for Calling Scenarios Data Templates Common Types Common Behavior Common Templates Benchmark Constants Benchmark Parameters

10. TTCN-3 User Conference, 30.05.200710 Test Adapter usrN+1 Component Handler 2 usrN+2usr2N Users Database usr1usr2usrN Component Handler 1 Users Database Test Configuration SuT Component Sender User 1 stackUser 2 stackUser N stack Load Generator

11. TTCN-3 User Conference, 30.05.200711 User Concept 1.A user is a state machine running a scenario 2.A user may be „callee“ or/and „caller“ 3.A user may create > 1 calls 4.A user may be reused to create other calls 5.A user may call randomly any other user

12. TTCN-3 User Conference, 30.05.200712 Simultaneous Scenarios calling messaging calling Same user involved in more than another instance of same scenario Same user involved in different scenario

13. TTCN-3 User Conference, 30.05.200713 Population Clusters all other scenarios like messaging, calling, reregistration b c registration (not registerred yet) a deregistration (registerred and not involved in any other scenario)

14. TTCN-3 User Conference, 30.05.200714 TTCN-3 Message Types // general type for a SIP Request // can be extended by any // other request type (i.e. INVITE_Request) type record Request { integer transactionId, RequestLine requestLine, MessageHeader msgHeader, charstring messageBody optional } type set MessageHeader { Authorization authorization optional, CallId callId optional, Contact contact optional, CSeq cSeq optional, Expires expires optional, From fromField optional, RecordRoute recordRoute optional, Route route optional, ServiceRoute serviceroute optional, To toField optional, Via via optional, MaxForwards maxForwards optional, ContentLength contentLength optional, WwwAuthenticate wwwAuthenticate optional, Event event optional } // [20.7 RFC2617 3.2.2] type record Authorization { FieldName fieldName(AUTHORIZATION_E), //Credentials body charstring body optional }

15. TTCN-3 User Conference, 30.05.200715 TTCN-3 State Handling alt { // P-CSCF --------REQUEST--------> User [] p2SUT.receive (TRequest) -> value request { // P-CSCF --------INVITE--------> User if (match(request, Request_INVITE_r)) { } // P-CSCF --------MESSAGE-------> User else if (match(request, Request_MESSAGE_r)) { } // P-CSCF --------ACK--------> User else if (match(request, Request_ACK_r)) { } ………………. } // P-CSCF --------RESPONSE--------> User [] p2SUT.receive (TResponse) -> value response { // P-CSCF -------- 200 OK --------> User if (match(response, Response_200_r)) { if (response.msgHeader.cSeq.seqMethod == "INVITE") {} else if (response.msgHeader.cSeq.seqMethod == "BYE") {} ………………. else if (response.msgHeader.cSeq.seqMethod == "REGISTER") {} } // P-CSCF -------- 401 OK --------> User else if (match(response, Response_401_r)) { } ………………. }

16. TTCN-3 User Conference, 30.05.200716 Messages Creation // P-CSCF --------RESPONSE--------> User [] p2SUT.receive (TResponse) -> value response { // P-CSCF -------- 200 OK --------> User if (match(response, Response_200_r)) { if (response.msgHeader.cSeq.seqMethod == "INVITE") { ackReq := ACK_Request_s; ackReq.transactionId := getNewTransactionId(); ackReq.requestLine.method := ACK_E;................................... ackReq.msgHeader.callId := response.msgHeader.callId; ackReq.msgHeader.cSeq := response.msgHeader.cSeq; ackReq.msgHeader.cSeq.seqMethod := "ACK";................................... ackReq.msgHeader.fromField := response.msgHeader.fromField; ackReq.msgHeader.toField := response.msgHeader.toField; ackReq.msgHeader.via := response.msgHeader.via; ackReq.msgHeader.route := getServiceRoute(response); userChannel := getChannel(response); p2SUT.send (ackReq) to userChannel; delTransactionId(response.transactionId);

17. TTCN-3 User Conference, 30.05.200717 Processing of Messages message processor RESP REQ identify update state validate transaction state process response SUT test component Platform adaptor

18. TTCN-3 User Conference, 30.05.200718 Adaptation Layer Aim: Maximize TTCN-3 system performance !  Handles multiple parallel connections with SUT  Uses Pools of Threads and Listeners  Smaller number of threads and context switches mutex-es and semaphores contended message queues Test Management (TM) Component Handling (CH) Coding Decoding (CD) TTCN-3 Executable (TE) System Adapter (SA) Platform Adapter (PA) TTCN-3 Executable (TE) System under Test Test System System Adapter (SA)

19. TTCN-3 User Conference, 30.05.200719 Adapter Transactions Stack ListeningPoint SipProvider SipListener SIP Messages SIP Events Stack ListeningPoint SipProvider SipListener SIP Messages SIP Events SUT PTC

20. TTCN-3 User Conference, 30.05.200720 Test Adapter Design Test System (TS) 6: handle message TE SA 3: read request Message Handlers Thread Pool 9: write ready Initiation Dispatcher (Reactor) 2: read ready 1: request message System under Test (SUT) 4: message ready 5: notify TE (put message in the appropriate port queue) 7: response message 10: write message 8: send response message I/O operations are made completely separately by the processing of the message. Concurrent I/O are possible on a SMP system. Advantages:

21. TTCN-3 User Conference, 30.05.200721 Overall Traffic

22. TTCN-3 User Conference, 30.05.200722 CPU Monitoring on SuT

23. TTCN-3 User Conference, 30.05.200723 Distribution of SAPS

24. TTCN-3 User Conference, 30.05.200724 Average Delay for Invite Requests

25. TTCN-3 User Conference, 30.05.200725 Conclusions  TTCN-3 specification has a big impact on the performance of the hole test system Using a component to simulate more than one user Using a component sender Using a tree like mechanism for matching the messages  Efficient Adaptation Layer Design has an important role in the overall performance Smart codec/decoder system Non-blocking I/O for protocol stack Optimized runtime for specific needs of the benchmark

26. TTCN-3 User Conference, 30.05.200726 Thank you ! Time for questions …