TTCN-3 based Implementation of ETSI TISPAN IMS Benchmark George Din, Diana Vega, Razvan Petre, Andreas Hoffman Fraunhofer Institute for Open Communication Systems

TTCN-3 User Conference, Contents  Motivation  What do we test?  How do we test?  Implementation details  Benchmark Results  Conclusions

TTCN-3 User Conference, Motivation  Goal - performance benchmark for IMS components Performance and scalability testing of all IMS and related components with simulated real-world traffic Measurement and analysis of important QoS parameters  Why Creation of objective means to compare overall IMS of different systems by performance (and price) Check ability of hardware/software to run the IMS  How Define standard scenarios and traffic models for the work load Define the metrics to be measured Standardize the test procedure, the test parameters and the Benchmark test report  Where Standardization of IMS benchmarking at ETSI TISPAN WG6 Version 1.0 of IMS benchmarking standard has been released.

TTCN-3 User Conference, System under Test Charging Functions IP Transfer (Access and Core) MRFP BGCFMRFC SLF MGCF Other IP Networks Resource and Admission Control Subsystem IWF I-BGF IBCF AS PSTN / ISDN SGF T-MGF Network Attachement Subsystem Gm Gq’ MpMn Mx Mg Mj Gq’ Dh Dx Rf / Ro Mw Mx Sh Cx Ut ISC/Ma Iw Ic Ie Ib Mr Mk Mle2 I/S-CSCF UE UPSF IMS Core Fokus Open IMS UE P-CSCF

TTCN-3 User Conference, Test Report Benchmark Information Model Use case 1 (e.g.: registration) Scenario 1.1 (e.g. user not registered) Message flow Metrics & graphs to collect SuT parameters Design objectives Scenario 1.2 (e.g. already registered) Benchmark Test Traffic Set Preamble Registration Phase SuT Characteristics Traffic Profile Initial load Step duration Step Increase Amount Number Of Steps Other parameters Use case 2 (e.g.: session setup) Scenario 2.1 (e.g. successful call) Message flow Metrics & graphs to collect SuT parameters Design objectives Scenario 2.2 (e.g. rejected call) Background Load X% of scenario 1.1 y% of scenario 2.1 z% of scenario 2.2 DisclaimerBenchmark Test Run SuT configuration and parameters Metrics and graphics with Design Objective Capacity (overall and for each scenario) Test System configuration and parameters Observations and interpretations Exceptions Metrics and graphics with SuT resource consumption

TTCN-3 User Conference, Scenario: Successful Call INVITE 100 Trying INVITE 180 Ringing200 OK180 Ringing200 OK ACK BYE 200 OK REGISTER 401 REGISTER 200 OK RegistrationSection Tear Up Tear Down Client 1IMS CN REGISTER 401 REGISTER 200 OK Client 2IMS CN Client1Client2CN Talking Time Ringing Time

TTCN-3 User Conference, Scenario: Successful Call INVITE 100 Trying INVITE 180 Ringing200 OK180 Ringing200 OK ACK BYE 200 OK Section Tear Up Tear Down Client1 Client2 CN TRT-SES1 TRT-REL1 TRT-SES2 Talking Time Ringing Time

TTCN-3 User Conference, Estimated Maximum Load No Load Initial Load Initial Call Load Stop Condition Met Time To Stop Condition Plus Load Plus Load Plus Load Ramp Load Steady Plus Load Stability Or Settling Time Linear, Immediate? Stability? Rate Increase Fluctuations, Steady state? Fluctuations, Steady state? Benchmark Procedure

TTCN-3 User Conference, TTCN-3 ATS Architecture Scenario-Independent Scenario-dependant User State Machine User State Machine Imports from Benchmar k Testcase Benchmar k Testcase Module for Instant Messaging Scenarios Data Templates Module for Registration Scenarios Data Templates Module for Calling Scenarios Data Templates Common Types Common Behavior Common Templates Benchmark Constants Benchmark Parameters

TTCN-3 User Conference, Test Adapter usrN+1 Component Handler 2 usrN+2usr2N Users Database usr1usr2usrN Component Handler 1 Users Database Test Configuration SuT Component Sender User 1 stackUser 2 stackUser N stack Load Generator

TTCN-3 User Conference, User Concept 1.A user is a state machine running a scenario 2.A user may be „callee“ or/and „caller“ 3.A user may create > 1 calls 4.A user may be reused to create other calls 5.A user may call randomly any other user

TTCN-3 User Conference, Simultaneous Scenarios calling messaging calling Same user involved in more than another instance of same scenario Same user involved in different scenario

TTCN-3 User Conference, Population Clusters all other scenarios like messaging, calling, reregistration b c registration (not registerred yet) a deregistration (registerred and not involved in any other scenario)

TTCN-3 User Conference, TTCN-3 Message Types // general type for a SIP Request // can be extended by any // other request type (i.e. INVITE_Request) type record Request { integer transactionId, RequestLine requestLine, MessageHeader msgHeader, charstring messageBody optional } type set MessageHeader { Authorization authorization optional, CallId callId optional, Contact contact optional, CSeq cSeq optional, Expires expires optional, From fromField optional, RecordRoute recordRoute optional, Route route optional, ServiceRoute serviceroute optional, To toField optional, Via via optional, MaxForwards maxForwards optional, ContentLength contentLength optional, WwwAuthenticate wwwAuthenticate optional, Event event optional } // [20.7 RFC ] type record Authorization { FieldName fieldName(AUTHORIZATION_E), //Credentials body charstring body optional }

TTCN-3 User Conference, TTCN-3 State Handling alt { // P-CSCF REQUEST > User [] p2SUT.receive (TRequest) -> value request { // P-CSCF INVITE > User if (match(request, Request_INVITE_r)) { } // P-CSCF MESSAGE > User else if (match(request, Request_MESSAGE_r)) { } // P-CSCF ACK > User else if (match(request, Request_ACK_r)) { } ………………. } // P-CSCF RESPONSE > User [] p2SUT.receive (TResponse) -> value response { // P-CSCF OK > User if (match(response, Response_200_r)) { if (response.msgHeader.cSeq.seqMethod == "INVITE") {} else if (response.msgHeader.cSeq.seqMethod == "BYE") {} ………………. else if (response.msgHeader.cSeq.seqMethod == "REGISTER") {} } // P-CSCF OK > User else if (match(response, Response_401_r)) { } ………………. }

TTCN-3 User Conference, Messages Creation // P-CSCF RESPONSE > User [] p2SUT.receive (TResponse) -> value response { // P-CSCF OK > User if (match(response, Response_200_r)) { if (response.msgHeader.cSeq.seqMethod == "INVITE") { ackReq := ACK_Request_s; ackReq.transactionId := getNewTransactionId(); ackReq.requestLine.method := ACK_E; ackReq.msgHeader.callId := response.msgHeader.callId; ackReq.msgHeader.cSeq := response.msgHeader.cSeq; ackReq.msgHeader.cSeq.seqMethod := "ACK"; ackReq.msgHeader.fromField := response.msgHeader.fromField; ackReq.msgHeader.toField := response.msgHeader.toField; ackReq.msgHeader.via := response.msgHeader.via; ackReq.msgHeader.route := getServiceRoute(response); userChannel := getChannel(response); p2SUT.send (ackReq) to userChannel; delTransactionId(response.transactionId);

TTCN-3 User Conference, Processing of Messages message processor RESP REQ identify update state validate transaction state process response SUT test component Platform adaptor

TTCN-3 User Conference, Adaptation Layer Aim: Maximize TTCN-3 system performance !  Handles multiple parallel connections with SUT  Uses Pools of Threads and Listeners  Smaller number of threads and context switches mutex-es and semaphores contended message queues Test Management (TM) Component Handling (CH) Coding Decoding (CD) TTCN-3 Executable (TE) System Adapter (SA) Platform Adapter (PA) TTCN-3 Executable (TE) System under Test Test System System Adapter (SA)

TTCN-3 User Conference, Adapter Transactions Stack ListeningPoint SipProvider SipListener SIP Messages SIP Events Stack ListeningPoint SipProvider SipListener SIP Messages SIP Events SUT PTC

TTCN-3 User Conference, Test Adapter Design Test System (TS) 6: handle message TE SA 3: read request Message Handlers Thread Pool 9: write ready Initiation Dispatcher (Reactor) 2: read ready 1: request message System under Test (SUT) 4: message ready 5: notify TE (put message in the appropriate port queue) 7: response message 10: write message 8: send response message I/O operations are made completely separately by the processing of the message. Concurrent I/O are possible on a SMP system. Advantages:

TTCN-3 User Conference, Overall Traffic

TTCN-3 User Conference, CPU Monitoring on SuT

TTCN-3 User Conference, Distribution of SAPS

TTCN-3 User Conference, Average Delay for Invite Requests

TTCN-3 User Conference, Conclusions  TTCN-3 specification has a big impact on the performance of the hole test system Using a component to simulate more than one user Using a component sender Using a tree like mechanism for matching the messages  Efficient Adaptation Layer Design has an important role in the overall performance Smart codec/decoder system Non-blocking I/O for protocol stack Optimized runtime for specific needs of the benchmark

TTCN-3 User Conference, Thank you ! Time for questions …