Buenos Aires - Argentina July Equipment for Testing Mobile QoS Joachim Pomy OPTICOM, Germany ITU Workshop on “Monitoring and Benchmarking of QoS and QoE of Multimedia Services in Mobile Networks” (Buenos Aires, Argentina, July 2014)

Buenos Aires - Argentina July  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

Buenos Aires - Argentina July Measurement of QoS in Mobile  Measured from the perspective of an expert  Assessing the network to see if it delivers the quality that the network planner targeted (cf. Rec. ITU-T G.1000)  Tools and Methods for Operation and Maintenance  Protocol analysers are connected to BTSs, BSCs, and MSCs for a period of time to check for problems in the cellular network  Drive test (also walk tests) are used for monitoring operationally achieved QoS

Buenos Aires - Argentina July Measurement of QoS in Mobile  Drive tests allow the mobile network to be tested through the use of a team of people who take the role of users  Rate the QoS of the network  Not applicable to the entire network  Always a statistical sample  In Operation and Maintenance Centres (OMCs), counters are used for various events  Providing the network operator with vendor specific information on the state and quality of the network  Also, customer complaints are a vital source of feedback on the QoS, and must not be ignored

Buenos Aires - Argentina July End-to-end QoS Testing in Mobile  Minimum requirement for QoS measurement equipment to ensure that  for all QoS parameters  trigger points are well defined  values are determined following wqell defined procedures  Test-equipment fulfilling the specified minimum requirements  Will allow performing the proposed measurements  In a reliable and reproducible way

Buenos Aires - Argentina July Parameters are based on Field Measurements  Measurements from customers point of view  Full End-to-End perspective, taking into account needs of testing  Assumed end customer can handle his equipment and services to use  Operability is not evaluated at this time  For the purpose of measurement it is assumed that  service is not barred for any reason  routing is defined correctly without errors  target subscriber equipment is ready to process service request  speech and video quality values measured – Only to be applied to calls ended successfully – In order to receive proper statistical analysis

Buenos Aires - Argentina July Not yet any ITU Standards in Place  ETSI TS part 4  Speech and multimedia Transmission Quality (STQ); QoS aspects for popular services in GSM and 3G networks; Part 4: Requirements for Quality of Service measurement equipment  globally recognized as "the requirement" in the area  extremely compehensive  detailed instructions

Buenos Aires - Argentina July  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

Buenos Aires - Argentina July Test Scenarios  Test scenarios need to distinguish the following principal user cases  User-to-user services (typically telephony)  Store-and-forward services (e.g. SMS)  Information services (e.g. accessing the internet or FTP download)  Push services (sender initiated transfer)

Buenos Aires - Argentina July  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

Buenos Aires - Argentina July Considerations on trigger points  It can be assumed that  any feasible test equipment will contain some kind of communication terminal (UE) which may be a special type (e.g. a Trace Phone)  or a standard UE  Also, it can be assumed that each such device will provide information from different communication layers  from Application Layer (close to the user interface)  down to lower layers, e.g. operating-system events, TCP/IP layer, or Layer 3 signalling information, which is used as trigger points for QOS PARAMETERS processing  Typically AT commands can be used in majority of cases  Action is typically triggered  by some emulated user action  causing some action on the air interface

Buenos Aires - Argentina July Considerations on trigger points  Process of event propagation is deterministic  allowing for kind of mapping between layers  associated with communication and processing delay in each stage.  Choice of the layer for trigger point information retrieval  determines the view expressed in a QOS PARAMETER  choosing lower-level events ( Layer 3) gives more network-centric view  events on higher levels to produce views more user-related  for same QoS PARAMETER source layer for events used as trigger points should be the same  In benchmarking, for all all networks under test  use same type of UE, and QOS PARAMETERS  use trigger points from the same layer.  Changing the source layer for a given trigger point  changes in QOS PARAMETERS should be expected  calibration measurements required  to assess influence on QOS PARAMETERS both quantitatively and qualitatively.

Buenos Aires - Argentina July  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

Buenos Aires - Argentina July Mobile QoS Test-equipment (MQT)  Geographical data collection during a mobile measurement  Position, speed and heading  Taken from GPS whenever possible.  If no GPS available use other tools  e.g. navigation on a geo-referenced bitmap

Buenos Aires - Argentina July  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

Buenos Aires - Argentina July Telephony Voice  Means to generate necessary calls or to answer automatically incoming calls  Received speech sample is measured  via the analogue output of the User Equipment  hardware setup to include necessary electrical adaptation  Hardware setup of such a system is essential for the correctness of the measured QoS Parameters  Includes a UE specific part for the electrical adaptation  audio output of handset to input of the soundcard

Buenos Aires - Argentina July Video Telephony  Means to generate necessary calls or to answer automatically incoming calls  Capability to provide the following measurement methods.  Video Telephony using the User Equipments application; or  Video Telephony using a Video Telephony Stack on the PC.

Buenos Aires - Argentina July MQT for FTP  FTP client to support the following functions:  Active / Passive Mode  Common Firewall support  Downloading from subdirectories.  Uploading to subdirectories

Buenos Aires - Argentina July Mobile based measurement equipment  Functionalities of the MQT can also be realised on a single mobile phone  Different types of QoS tests can run  Test results can be logged for post processing  Mobile based measurement equipment may be controlled  by an application, remotely or  may be operated by a human

Buenos Aires - Argentina July  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

Buenos Aires - Argentina July Practical Hints  Which Mobile Phone to use ?  should be one offered in the shop ?  should be one used by new customers ?  should be one hipe to young people ?  should be one with good QoS ?  BUT  it MUST be a so-called Trace Mobile  ASK YOUR PHONE PROVIDER !!!  they are RARE !!!!

Buenos Aires - Argentina July  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

Buenos Aires - Argentina July Hot Topic: POLQA™ - Rec. P.863 (1)

Buenos Aires - Argentina July POLQA™ - Rec. P.863 (2)  New POLQA standard winner of rigorous technical competition  Carried out by ITU-T to define technology update for PESQ/P.862  PESQ was state-of-the-art in almost any telecom service globally  POLQA - “Perceptual Objective Listening Quality Assessment” - offers new level of benchmarking capability  to determine the voice quality of mobile network services

Buenos Aires - Argentina July POLQA™ - Rec. P.863 (3)  POLQA unified solution by OPTICOM, SwissQual and TNOOPTICOMSwissQual TNO  was selected as the winning approach  the underlying algorithms were subjected to extensive tests  over 45,000 speech samples from a broad range of targeted usage scenarios  wide array of tests covers the entire range of today’s telephony quality  over the last ten years test and measurement vendors have widely adopted ITU-T Recommendation P.862/PESQ (“Perceptual Evaluation of Speech Quality”) and there are now more than installations worldwide  recently developed speech codecs for 3G and 4G, and new techniques for delivering an enhanced voice quality experience have generated strong demand for an update to this industry standard  POLQA is the answer the industry has been looking for

Buenos Aires - Argentina July POLQA™ - Rec. P.863 (4)  POLQA coalition formed by OPTICOM, SwissQual and TNO is the outcome of the competitive standardization process run by the ITU-T  An expert group within ITU-T spent more than four years evaluating various candidate algorithms to find a suitable successor to older standards  An enormous amount of effort has gone into ensuring POLQA works quickly and accurately  For example, more than 100,000 samples of mobile and VoIP phone calls were used to train and optimize the POLQA algorithm  POLQA is superior to existing standards, and clearly has overcome all known issues and limitations of PESQ

Buenos Aires - Argentina July POLQA™ - Rec. P.863 (5)  The limitations of existing standards that are now addressed by POLQA  CDMA  Chinese 3G TD-SCDMA  POLQA will also offer immediate, strong support for testing of new wideband 4G/LTE networks delivering HD-quality voice services  Tests carried out during the POLQA evaluation included future technologies such as  Unified Communications  Next Gen Networks  4G/LTE  HD Voice, i.e. "wide-band" and "super-wide-band"

Buenos Aires - Argentina July POLQA™ - Rec. P.863 (6)  More information on POLQA can be found in the technical white paper “POLQA: The Next Generation in Voice Quality Testing” available for download from

Buenos Aires - Argentina July

Buenos Aires - Argentina July POLQA Introduction - (c) OPTICOM GmbH PESQPOLQA Acoustic measurements  Not easy Correct scoring with high background noise  AMR vs EVRC codec comparison  Representative scoring of reference signals  Effects of speech level in samples  Narrowband (300Hz -3400Hz) Wideband (100Hz-7000Hz) Use SWB Superwideband, SWB (50Hz – 14000Hz)  Linear Frequency distortion sensitivity 

Buenos Aires - Argentina July What we Perceive … POLQA Introduction - (c) OPTICOM GmbH In a subjective ACR experiment POLQA, PESQ and human beings perceive the following distortions: FactorHumanPOLQAPESQ Level too high or too lowXXno Strong linear filteringXXno Noise in the reference signalXXno High timbre in referenceXXno Level variationXXpoor SWB noise on NB/WB signalXXno

Buenos Aires - Argentina July POLQA Introduction - (c) OPTICOM GmbH Performance Validation The ITU has validated POLQA on: Languages included in the POLQA validation: German Swiss German Italian, Japanese, Swedish American English and British English Chinese (Mandarin), Czech, Dutch, French, file pairs across 64 subjective experiments

Buenos Aires - Argentina July Any questions ? Contact: