Version : 11 December 2008 Workshop on “Monitoring Quality of Service and Quality of Experience of Multimedia Services in Broadband/Internet Networks”

Slides:



Advertisements
Similar presentations
Nairobi, Kenya, 26 – 27July 2010 Quality of Service Monitoring Tool Tahitii OBIOHA Network Planning & Optimisation Engineer Planet Network International.
Advertisements

International Telecommunication Union Workshop on Standardization in E-health Geneva, May 2003 Basic requiremenst to Quality of Service (IP centric)
Cotonou, Benin, July 2012 Mobile QoS Framework: Counters, KPI, KQI Joachim Pomy, SG 12 Rapporteur Consultant, Opticom GmbH
Evolution of NGN and NGA scenario in Nepal Nepal Telecommunications Authority.
ETSI Workshop on Quality Issues for IP Telephony 8-9 June 1999, Sophia Antipolis, France ETSI PROJECT TIPHON overview of QoS activities ETSI Workshop on.
Kampala, Uganda, 23 June 2014 ITU-T Recommendation E.807: “Definitions and associated measurement methods of user-centric parameters for call handling.
Fundamentals of Multimedia Part III: Multimedia Communications and Networking Chapter 15 : Network Services and Protocols for Multimedia Communications.
ITU Regional Standardization Forum For Africa Dakar, Senegal, March 2015 QoS/QoE Assessment Methodologies (Subjective and Objective Evaluation Methods)
AIMS Workshop Heidelberg, 9-11 March /20 A Telecom and IP Project from ETSI Gerald Meyer
Version : 11 December 2008 Workshop on “Monitoring Quality of Service and Quality of Experience of Multimedia Services in Broadband/Internet Networks”
1 ITU Workshop on “Quality of Service and Quality of Experience of Multimedia Services in Emerging Networks” (Istanbul, Turkey, 9-11 February 2015) Overview.
ADVICE (Auxiliary Data & Voice Integrated Channel for Education) A Comprehensive WorldSpace eLearning Solution.
Q.Int_speed_test Draft Recommendation Eva Ibarrola SG11 Q15 Acting Rapporteur JCA-CIT meeting (electronic meeting)
Quality of Service Measurements of Mobile Networks
DAKNET Presented By: rreema.
Networks Evolving? Justin Champion C208 Ext:3723
1 Computer Networks Transport Layer Protocols. 2 Application-layer Protocols Application-layer protocols –one “piece” of an app –define messages exchanged.
SG12 Regional Group for Africa Meeting 18 th to 19 th of July, 2013 Ouagadougou, Burkinafaso By Yvonne UMUTONI Quality of Service Development Group (QSDG)
The OSI Model A layered framework for the design of network systems that allows communication across all types of computer systems regardless of their.
Cotonou, Benin, July 2012 Equipment for Testing Mobile QoS Joachim Pomy, SG 12 Rapporteur Consultant, Opticom GmbH ITU.
Maputo, Mozambique, April 2014 QoS Framework for Broadband Internet Service Country Case: Mauritius Roubee GADEVADOO, Engineer/Licensing Officer,
ITU Workshop on “Quality of Service and Quality of Experience of Multimedia Services in Emerging Networks” (Istanbul, Turkey, 9-11 February 2015) Mobile.
© Omnitele Ltd UMTS Quality of Service enabling new mobile applications CANTO St. Kitts & Nevis, 19 th – 22 nd June 2005 Heidi Lagerström,
COnvergence of fixed and Mobile BrOadband access/aggregation networks Work programme topic: ICT Future Networks Type of project: Large scale integrating.
The need for further standards and technical developments Brian Moore ITU-T Study Group 13 Chairman Lucent Technologies.
Spectranet - Confidential 1 IPv6 ; Internet User’s Perspective Content delivery from the edge; Indian Perspective APNIC30 August 2010.
Ouagadougou, Burkina Faso, 18 July Content and presentation of Recommendation E.MQoS Joachim Pomy, SG 12 Rapporteur Consultant, Opticom GmbH
Dubai, UAE 2-3 November Measuring the QoS of Internet Access Joachim Pomy OPTICOM, Germany ITU Workshop on “Quality of.
© 2006 Cisco Systems, Inc. All rights reserved.Cisco Public 1 Version 4.0 Identifying Application Impacts on Network Design Designing and Supporting Computer.
1 Quality of Service (QoS), Quality of Experience (QoE) and Performance Joachim Pomy OPTICOM, Germany ITU Workshop on “Performance,
3/12/20031 QoS Recherche des critères, paramètres et indicateurs pertinents pour les utilisateurs Réunion QOSTIC Paris 10 Décembre 2003 P-Y Hébert - AFUTT/ETSI.
Quality of Service (QoS) of interconnected Packet- based Networks Workshop on “Monitoring Quality of Service and Quality of Experience of Multimedia Services.
Colombia, September 2013 The importance of models and procedures for planning, monitoring and control in the provision of communications services.
IP Networking & MEDIACOM 2004 Workshop April 2001 Geneva Characterising End to End Quality of Service in TIPHON Systems Characterising End to End.
Computer and Information Science Ch1.3 Computer Networking Ch1.3 Computer Networking Chapter 1.
Standards Convergence Dave Evans Philips Research Laboratories UK
© 2006 Cisco Systems, Inc. All rights reserved.Cisco PublicITE I Chapter 6 1 Identifying Application Impacts on Network Design Designing and Supporting.
Kampala, Uganda, 23 June 2014 Applicability of the ITU-T E.803 Quality of service parameters for supporting service aspects Kwame Baah-Acheamfuor Chairman,
Version : 11 December 2008 Workshop on “Monitoring Quality of Service and Quality of Experience of Multimedia Services in Broadband/Internet Networks”
Quality of Service (QoS) Monitoring and Functions of Internet ITU Regional Standardization Forum for Africa (Kampala, Uganda, June 2014) Yvonne UMUTONI.
Version : 11 December 2008 Workshop on “Monitoring Quality of Service and Quality of Experience of Multimedia Services in Broadband/Internet Networks”
Version : 11 December 2008 ITU Regional Standardization Forum for Africa (Kampala, Uganda, June 2014) Mobile QoS Framework: Counters, KPI, KQI Joachim.
ﺑﺴﻢﺍﷲﺍﻠﺭﺣﻣﻥﺍﻠﺭﺣﻳﻡ. Group Members Nadia Malik01 Malik Fawad03.
 Introduction – Consumer Market  Benefits – Operational Cost & Flexibility  Challenges – Quality of Service & Securing VOIP  Legal Issuers  Risk.
ITU Workshop on “Monitoring and Benchmarking of QoS and QoE of Multimedia Services in Mobile Networks” (Buenos Aires, Argentina, July 2014) Overview.
Buenos Aires - Argentina July Equipment for Testing Mobile QoS Joachim Pomy OPTICOM, Germany ITU Workshop on.
Dubai - UAE November Quality of Service (QoS), Quality of Experience (QoE) and Performance Joachim Pomy OPTICOM,
25/11/2015 ITU-T NGN - Progress and Plans Brian Moore Lucent Technologies Chairman of ITU-T Study Group 13 1GSC-9, Seoul SOURCE:ITU-T TITLE:ITU-T NGN -
Integrated Knowledge System on Climate Change Adaptation Conceptual & Technological Framework OneWorld South Asia December 2008.
QOS OF DATA SERVICES IN MOBILE NETWORKS Ganson Lewela Head of Regulatory Airtel Networks Kenya Ltd STAKEHOLDERS FORUM ON QUALITY OF SERVICE AND CONSUMER.
Next Generation Networks Australian Communications Industry Forum (ACIF) ITU-T NGN Workshop, July 2003 Peter Darling Manager, ACIF NGN Project.
E-Commerce & M-Commerce. Introduction Electronic commerce, commonly known as e- commerce, It is a type of industry where buying and selling of product.
QoS issues in Digital Financial Services Joachim Pomy OPTICOM GmbH Germany 1 ITU Regional Standardization Forum For Africa Dakar,
Voice Over Internet Protocol (VoIP) Copyright © 2006 Heathkit Company, Inc. All Rights Reserved Presentation 5 – VoIP and the OSI Model.
EACO Guidelines on Quality of Service on Data Service in Mobile Networks Orange Telkom Kenya November 2015.
AIMS’99 Workshop Heidelberg, May 1999 Multimedia Services using IP over Bearer Networks: Quality of Service Aspects based on ACTS Guidelines SII.
Introduction to ITIL and ITIS. CONFIDENTIAL Agenda ITIL Introduction  What is ITIL?  ITIL History  ITIL Phases  ITIL Certification Introduction to.
Version : 11 December 2008 ITU Workshop on “Monitoring and Benchmarking of QoS and QoE of Multimedia Services in Mobile Networks” (Buenos Aires, Argentina,
Organizations of all types and sizes face a range of risks that can affect the achievement of their objectives. Organization's activities Strategic initiatives.
WATRA/ARTP REGIONAL WORKSHOP
Parameters/Thresholds for Quality of Voice and Data Services (Case of Ghana) Isaac Annan Laryea-Monitoring and Compliance, NCA-Ghana
Quality of Service Development Group (QSDG) meeting
Lab A: Planning an Installation
IP Telephony (VoIP).
Introduction to Wireless Sensor Networks
An IP-based multimedia traffic generator
3 | Analyzing Server, Network, and Client Health
SUMMARY Context Objectives targeted through the Plan of Actions
Content and presentation of Recommendation E.MQoS
IP and NGN Projects in ITU-T Jean-Yves Cochennec France Telecom SG13 Vice Chair Workshop on Satellites in IP and Multimedia - Geneva, 9-11 December 2002.
Presentation transcript:

Version : 11 December 2008 Workshop on “Monitoring Quality of Service and Quality of Experience of Multimedia Services in Broadband/Internet Networks” (Maputo, Mozambique, April 2014) Mobile QoS Framework: Counters, KPI, KQI Joachim Pomy OPTICOM, Germany

Maputo - Mozambique April  Mobile QoS Framework: Counters, KPI, KQI  New ITU-T work on Mobile QoS  Layered Approach – Selection of Mobile QoS Parameters perceived by the User  Test Scenarios and Trigger Points  Voice Quality Assessment – Drive Testing  Statistics – Presentation of Results  Counters, KPI, KQI ITRs: Setting the stage for a connected world [Content]

Maputo - Mozambique April ITU-T work on Mobile QoS (1)  Mobile KPIs  centric to equipment / infrastructure  aggregation of proprietary network counters  not standardized  New work in ITU-T on Mobile QoS  Draft new Recommendation E.MQoS  QoS Aspects for Popular Services in Mobile Networks  Draft new Supplement to E.800 series of Recommendations  Guidelines on Regulatory Aspects of QoS  Other ITU-T Recommendations on QoS cover  Aspects of Terminal requirements  Aspects of end-to-end quality  Aspects of user perceptions

Maputo - Mozambique April  Mobile QoS Framework: Counters, KPI, KQI  New ITU-T work on Mobile QoS  Layered Approach – Selection of Mobile QoS Parameters perceived by the User  Test Scenarios and Trigger Points  Voice Quality Assessment – Drive Testing  Statistics – Presentation of Results  Counters, KPI, KQI ITRs: Setting the stage for a connected world [Content]

Maputo - Mozambique April QoS Layers in Mobile  QoS model for mobile has four layers.  First layer is the Network Availability  defines QoS rather from the viewpoint of the service provider than the service user  Second layer is the Network Access  from user's point of view basic requirement for all the other QoS aspects and parameters  Third layer contains other QoS aspects  Service Access, Service Integrity & Service Retainability  Different services are located in the fourth layer  Their outcome are the QoS parameters as perceived by the user

Maputo - Mozambique April QoS aspects of Mobile

Maputo - Mozambique April Layered Mobile QoS (1)  If we follow the layered approach  focussing on events that can be perceived by the user  focussing on most relevant services in the market  Telephony  SMS  MMS  Video Telephony  Video Streaming  Web Browsing

Maputo - Mozambique April Layered Mobile QoS (2)  Layer 1 Layer 1  Radio Network Unavailability [%] Radio Network Unavailability [%]  Layer 2 Layer 2  Network Selection and Registration Failure Ratio [%] Network Selection and Registration Failure Ratio [%]

Maputo - Mozambique April Layered Mobile QoS (3)  Telephony Layer 3 Telephony Layer 3  Telephony Service Non ‑ Accessibility [%] Telephony Service Non ‑ Accessibility [%]  Telephony Layer 4 Telephony Layer 4  Telephony Setup Time [s] Telephony Setup Time [s]  Telephony Cut ‑ off Call Ratio [%] Telephony Cut ‑ off Call Ratio [%]

Maputo - Mozambique April Layered Mobile QoS (4)  SMS Layer 3 SMS Layer 3  SMS Service Non ‑ Accessibility [%] SMS Service Non ‑ Accessibility [%]  SMS Layer 4 SMS Layer 4  SMS End ‑ to ‑ End Delivery Time [s] SMS End ‑ to ‑ End Delivery Time [s]  SMS Completion Failure Ratio [%] SMS Completion Failure Ratio [%]

Maputo - Mozambique April Layered Mobile QoS (5)  MMS Layer 3 MMS Layer 3  MMS Send Failure Ratio [%] MMS Send Failure Ratio [%]  MMS Layer 4 MMS Layer 4  End ‑ to ‑ End Delivery Time [s] End ‑ to ‑ End Delivery Time [s]  MMS End ‑ to ‑ End Failure Ratio [%] MMS End ‑ to ‑ End Failure Ratio [%]

Maputo - Mozambique April Layered Mobile QoS (6)  Video Telephony Layer 3 Video Telephony Layer 3  VT Service Non ‑ Accessibility [%] VT Service Non ‑ Accessibility [%]  Video Telephony Layer 4 Video Telephony Layer 4  VT Audio/Video Setup Time [s] VT Audio/Video Setup Time [s]  VT Cut ‑ off Call Ratio [%] VT Cut ‑ off Call Ratio [%]

Maputo - Mozambique April Layered Mobile QoS (7)  Streaming Video Layer 3 Streaming Video Layer 3  Streaming Service Non ‑ Accessibility [%] Streaming Service Non ‑ Accessibility [%]  Layer 4: Layer 4:  Streaming Service Access Time [s] Streaming Service Access Time [s]  Streaming Reproduction Cut-off Ratio [%] Streaming Reproduction Cut-off Ratio [%]

Maputo - Mozambique April Layered Mobile QoS (8)  Web Browsing (HTTP) Layer 3 Web Browsing (HTTP) Layer 3  HTTP Service Non ‑ Accessibility [%] HTTP Service Non ‑ Accessibility [%]  Web Browsing (HTTP) Layer 4 Web Browsing (HTTP) Layer 4  HTTP Setup Time [s] HTTP Setup Time [s]  HTTP Session Failure Ratio [%] HTTP Session Failure Ratio [%]  HTTP Data Transfer Cut ‑ off Ratio [%] HTTP Data Transfer Cut ‑ off Ratio [%]

Maputo - Mozambique April Layered Mobile QoS (9)  This approach would make us end up with > 20 QoS parameter  All relate to user perceptional events  But this is just end-to-end parameters  Many more parameters defined for events related to network segments, hand-over etc.

Maputo - Mozambique April  Mobile QoS Framework: Counters, KPI, KQI  New ITU-T work on Mobile QoS  Layered Approach – Selection of Mobile QoS Parameters perceived by the User  Test Scenarios and Trigger Points  Voice Quality Assessment – Drive Testing  Statistics – Presentation of Results  Counters, KPI, KQI ITRs: Setting the stage for a connected world [Content]

Maputo - Mozambique April Test Scenarios  Test scenarios need to distinguish the following principal user cases  User-to-user services (typically telephony)  Store-and-forward services (e.g. SMS)  Information services (e.g. accessing the internet or FTP download)  Push services (sender initiated transfer)

Maputo - Mozambique April Considerations on trigger points  It can be assumed that  any feasible test equipment will contain some kind of communication terminal (UE) which may be a special type (e.g. a Trace Phone)  or a standard UE  Also, it can be assumed that each such device will provide information from different communication layers  from Application Layer (close to the user interface)  down to lower layers, e.g. operating-system events, TCP/IP layer, or Layer 3 signalling information, which is used as trigger points for QOS PARAMETERS processing  Typically AT commands can be used in majority of cases  Action is typically triggered  by some emulated user action  causing some action on the air interface

Maputo - Mozambique April Considerations on trigger points  Process of event propagation is deterministic  allowing for kind of mapping between layers  associated with communication and processing delay in each stage.  Choice of the layer for trigger point information retrieval  determines the view expressed in a QOS PARAMETER  choosing lower-level events ( Layer 3) gives more network-centric view  events on higher levels to produce views more user-related  for same QoS PARAMETER source layer for events used as trigger points should be the same  In benchmarking, for all all networks under test  use same type of UE, and QOS PARAMETERS  use trigger points from the same layer.  Changing the source layer for a given trigger point  changes in QOS PARAMETERS should be expected  calibration measurements required  to assess influence on QOS PARAMETERS both quantitatively and qualitatively.

Maputo - Mozambique April  Mobile QoS Framework: Counters, KPI, KQI  New ITU-T work on Mobile QoS  Layered Approach – Selection of Mobile QoS Parameters perceived by the User  Test Scenarios and Trigger Points  Voice Quality Assessment – Drive Testing  Statistics – Presentation of Results  Counters, KPI, KQI ITRs: Setting the stage for a connected world [Content]

Maputo - Mozambique April Telephony Measurements in the Network (1)  Transaction definition and transaction types  The basic transaction for telephony testing is equivalent to a single call to a counterpart extension. It is assumed that the call partner is typically a fixed-network type extension to avoid uncertainties related to a second mobile connection.  Type is either Mobile Originated (MO) or Mobile Terminated (MT).  It is assumed that once a connection has been established, for further measurements it does not matter which side has triggered it. Therefore, the audio data flow parameter will not be logically linked to the call type.

Maputo - Mozambique April Telephony Measurements in the Network (2)  Content quality  Content quality assessment data generated at receiving end  For downlink content data storage is straightforward  For uplink content, at some point in time results have to be integrated  For assessing content quality of complete transmitted speech samples, at least the following methods are possible:

Maputo - Mozambique April Telephony Measurements in the Network (3)  Content quality (ctd.)  Real-time assessment  streaming mode  speech quality assessment algorithm determines MOS- LQO real time  Offline assessment  content first recorded  being processed later  Data processing must ensure that only valid content quality data is processed  inside the "connection active" time window

Maputo - Mozambique April Telephony Measurements in the Network (4)  Verification of usable two-way connection  Only calls with a valid two-way end-to-end information connection shall be considered for content quality assessment (valid calls).  Non-valid calls treated like dropped calls  with indicator for this particular cause

Maputo - Mozambique April Hot Topic: POLQA™ - Rec. P.863 (1)

Maputo - Mozambique April POLQA™ - Rec. P.863 (2)  New POLQA standard winner of rigorous technical competition  Carried out by ITU-T to define technology update for PESQ/P.862  PESQ was state-of-the-art in almost any telecom service globally  POLQA - “Perceptual Objective Listening Quality Assessment” - offers new level of benchmarking capability  to determine the voice quality of mobile network services

Maputo - Mozambique April POLQA™ - Rec. P.863 (5)  The limitations of existing standards that are now addressed by POLQA  CDMA  Chinese 3G TD-SCDMA  POLQA will also offer immediate, strong support for testing of new wideband 4G/LTE networks delivering HD-quality voice services  Tests carried out during the POLQA evaluation included future technologies such as  Unified Communications  Next Gen Networks  4G/LTE  HD Voice, i.e. "wide-band" and "super-wide-band"  See POLQA: The Next Generation in Voice Quality Testing

Maputo - Mozambique April

Maputo - Mozambique April POLQA Introduction - (c) OPTICOM GmbH Performance Validation The ITU has validated POLQA on: Languages included in the POLQA validation: German Swiss German Italian, Japanese, Swedish American English and British English Chinese (Mandarin), Czech, Dutch, French, file pairs across 64 subjective experiments

Maputo - Mozambique April  Mobile QoS Framework: Counters, KPI, KQI  New ITU-T work on Mobile QoS  Layered Approach – Selection of Mobile QoS Parameters perceived by the User  Test Scenarios and Trigger Points  Voice Quality Assessment – Drive Testing  Statistics – Presentation of Results  Counters, KPI, KQI ITRs: Setting the stage for a connected world [Content]

Maputo - Mozambique April Confidence Intervalls for Different Sample Sizes (1)  Effect of different sample sizes in a measurement campaign  based on the Pearson-Clopper formulas for calculation of confidence intervals  valid in a generic way and even for small sample sizes  for higher sample numbers, the calculation of confidence intervals based on the approximation of a normal distribution can be applied  Three different graphs are depicted: Sample sizes in the range:  between 100 and samples;  between and samples; and  between and samples.

Maputo - Mozambique April Confidence Intervalls for Different Sample Sizes (2)

Maputo - Mozambique April Confidence Intervalls for Different Sample Sizes (3)

Maputo - Mozambique April Confidence Intervalls for Different Sample Sizes (4)

Maputo - Mozambique April Reporting of results (1)  Which pieces of information should be given to the reader when generating a test report ?  When quantile values are used, it should be kept in mind that the computation of quantiles separates a low percentage of outlier data from the remaining data. This means:  If lower values represent a better outcome from the customer's perspective, a small percentage containing the highest values could be separated by calculating a 95 %- quantile or a 90 %-quantile. This is the case for example for duration values.

Maputo - Mozambique April Reporting of results (2)  If higher values represent a better outcome from the customer's perspective, a small percentage containing the lowest values could be separated by calculating a 5 %- quantile or a 10 %-quantile. This is the case for example for throughput values.  Related to content quality, the appropriate quantile computation orientates itself on the scale of the determined test results. In practice, some algorithms define a value of 0 on a scale from 0 to 5 as the best quality whereas others define the value of 5 as the highest possible quality

Maputo - Mozambique April  Mobile QoS Framework: Counters, KPI, KQI  New ITU-T work on Mobile QoS  Layered Approach – Selection of Mobile QoS Parameters perceived by the User  Test Scenarios and Trigger Points  Voice Quality Assessment – Drive Testing  Statistics – Presentation of Results  Counters, KPI, KQI ITRs: Setting the stage for a connected world [Content]

Maputo - Mozambique April KPIs based on Network Counters  Vendor specific = network internal KPIs  different strategies  how to count network events  which events are included in which counter(s)  Requires knowledge of specific system  specialists with detailed system knowledge  testing the counters  documentation may be faulty  approach to counter change with system update  Mobile operators struggling with this  most operator live in a multi vendor environment  counters from different vendors cannot be directly compared  requires continous attention and a strategy

Maputo - Mozambique April KPIs from Users' Perspective = KQIs  Key Quality Indicators (KQIs) = external indicators  can be assessed in the Field  For Monitoring, Regulation etc.  a subset can be selected  applicable across all vendors & operators  not limited to mobile, but also good for broadband

Maputo - Mozambique April KPIs versus KQIs  Sometimes confused  KPIs = internal indicators  part of network performance  based on network counters  essential for operation, maintenance, business model  could be reported, audited etc.  however, meaningless when out of context  KQIs = external indicators  basis for QoS assessment as perceived by the user  vendor independant  operator independant  ideal to compare different operators on a statistical basis  cannot be reported from the system itself  requires some kind of field testing, drive, walk etc.

Maputo - Mozambique April Any questions ? Contact: