Monitoring QoS & QoE System of Mobile & Fixed Networks Case study of Greece Dr. Stavroula Bouzouki Head of Regional Office of Hellenic Telecommunication & Post Commission

Outline QoS & QoE concept for NRA Motivation Solution (post-process and visualization) Monitoring System benefits Future work

NRA and QoS & QoE Recognizing the importance of this information to users and the telecommunications market in accordance with International Standards and Recommendations of International Organizations, a National Regulatory Authority (NRA) has a duty to take steps to ensure: NRA The healthy competition between providers The development of reliable electronic communications networks The protection of end users The provision of objective information to all entrepreneurship in telecommunications

Motivation The main motives of EETT for the development of a Monitoring QoS & QoE System are the following: 1. The ability to inform the end users objectively regarding the quality of different mobile services of all mobile providers in specific geographical locations 2. To protect the end users through the control of the service provider with whom they are affiliated 3. To strengthen the competition between providers of telecommunications services aimed at improving the provided service quality, availability and geographical coverage of their networks 4. To increase available information and enhance the auditing role of EETT

QoS & QoE measurement methodologies The QoS & QoE measurement methodologies fall in the following two categories: Category 1: measurement methodologies with controlled measuring devices for simultaneous measurement of all providers (benchmarking measurements). Category 2: measurement methodologies from each user terminal device, capable of measuring only the operator of the specific user (crowd sourcing measurements). The differences between the categories make impossible the comparison of the respective results

Basic features of EETT’s Monitoring System The implementation covers: QoS & QoE measurement methodology of category 1 Provision of the move measurements of mobile service quality indicators simultaneously for all mobile providers Measuring Network of broadband quality indicators at fixed points for all mobile providers, via a dynamic procedure covering all the country Information for measurements’ control and presentation system Comparison results GIS search results The measurements are performed according ETSI standards and ITU recommendations

Overview of the Project Project: System for Measurement and Presentation of Quality Indicators for Communication Networks Budget: € 910.000,00 Duration: 11 months Consortium: ΑΡΤΕΜΙΣ ITS & VECTOR Subcontractors: Anite Finland ltd (Nemo) & University of Piraeus Research Center Urbanit

Architecture of Monitoring System (1/2) High Level Architecture of the Approach Sub-System 1: Mobile Measurements Stations Sub-System 2: Fixed Measurements Stations Sub-System 3: System for the Presentation of Quality Indicators

Architecture of Monitoring System (2/2) The high level architecture consists of two parts: A) Measurements Equipment for performing campaigns by EETT (Sub-System 1&2) B) System for the Presentation of Quality Indicators (Sub-System 3) Sub-System 1 & 2 carries out quality indicators measurements with the use of mobile and fixed units. Sub-System 3 undertakes the visualization of the results through a Content Management System (CMS). Moreover Sub-System 3 receives and visualizes fixed line measurements from the operators. Sub-System 1 includes : One mobile measurement station (modified vehicle for long distance coverage measurements) One mobile measurement station (modified backpack) Sub-System 2 includes : Twenty (20) probes Sub-System 3 includes: Visualization of the Quality Indicators Operators' Measurements Results

Technological Solutions for mobile networks Nemo Invex - Nemo Walker Nemo Invex Outdoor Nemo Autonomous / Nemo Fleet Manager Nemo Analyze Server / Voice Server / Data Server Orchard CMS Visualization tools Google Maps API Orchard CMS custom module for the visualization of the quality Indicators Sub-System 1 & 2 Sub-System 3

Technical Characteristics for mobile networks Measurement Options CS and PS data measurements Voice, voice quality PESQ and POLQA (ITU-T P.862.1, WB-AMR, P862.2, ITU-T P.863) VoLTE voice call support (Qualcomm IMS, Samsung IMS client) Video call, video call quality, video streaming, PVI video quality streaming, YouTube video streaming, and video streaming analysis measurements (PEVQ-S) Multi measurements Indoor measurements Frequency scanning, pilot scanning, CW scanning and spectrum analyzer measurements SMS, MMS, and USSD measurements FTP testing for file transfers (multi-threaded) SFTP testing for file transfers (download and upload) HTTP/S testing for file transfers (download and upload) HTTP/S testing for web browsing SMTP/POP3/IMAP for email testing WAP testing Iperf for UDP/TCP testing SIP-based VoIP call, voice quality testing PTT call testing (QChat and Kodiak InstaPoC™) Facebook, Twitter, Instagram testing IP packet capturing and playback Missing neighbor detection for GSM, WCDMA, and CDMA CDMA, EVDO, and WCDMA pilot pollution analysis GSM interference analysis LTE ANR CGI reports Diagnostics data logging (Qualcomm) Carrier aggregation testing Emergency call testing Remote controlled measurements with Nemo Commander Supported Technologies  LTE/LTE-A (FDD/TDD, carrier aggregation) VoLTE HSPA+ Dual Cell /HSPA+ GSM, cdmaOne, AMPS TETRA, iDEN HSCSD, GPRS, EDGE WCDMA CDMA2000 1xRTT, CDMA2000 1xEVDO (Rel. 0, Rev. A, Rev. B) HSDPA, HSUPA, TD-SCDMA WiMAX (802.16e)

Quality Indicators categories of Monitoring System The quality indicators are divided into three main categories: Category 1: Quality indicators of fixed communications systems services Category 2: Quality indicators of mobile communications systems services Category 3: Consumer Services and directory inquiry services quality indicators

Quality indicators of fixed communications systems services Service parameters for Fixed Telephony services: Unsuccessful call percentage (ETSI EG 202 057-2 v1.3.1) Proportion of card and coin operated public pay-telephones in working order (only for providers with Universal Service Obligation) (ETSI ETR 138) Speech connection quality (ITU-T G.107 , G.108, G.109, G.177, Y.1541, ETSI EG 202 057-2 v1.3.1, ETSI TS 101 329-5 V1.1.2) Supply time for initial connection (ETSI EG 202 057-1 v1.3.1, ETSI EG 201 769 v1.1.2) Faults rate per access line (ETSI EG 202 057-1 v1.3.1, ETSI EG 201 769 v1.1.2) Fault repair time (ETSI EG 202 057-1 v1.3.1, ETSI EG 201 769 v1.1.2) Service parameters for Fixed Broadband services: Data transmission speed achieved (ETSI EG 202 057-4 v1.2.1) Interconnection capacity (N/A) VoIP unsuccessful call percentage (ETSI EG 202 057-2 v1.3.1) VoIP connection quality (ITU-T G.107 , G.108, G.109, G.177, Y.1541, ETSI EG 202 057-2 v1.3.1, ETSI TS 101 329-5 V1.1.2) Faults rate per connection (ETSI EG 202 057-1 v1.3.1, ETSI EG 201 769 v1.1.2) Limitations in use and provision of QoS guarantees/SLAs

Quality indicators of mobile communications systems services Radio Network availability Voice quality Call block rate Call drop rate Call Setup time  FTP Uplink Mean Data Rate UL FTP Upload IP Service Setup Time FTP Upload Data Transfer FTP server connection success rate   FTP Downlink Mean Data Rate DL FTP Download IP Service Setup Time FTP Upload Data Transfer Cut-Off Ratio FTP server connection success rate   HTTP Downlink HTTP Data Transfer Non-Stability HTTP Mean Data Rate IP Service setup Time HTTP transfer duration   HTTP Web Browsing HTTP Browsing attempt HTTP Browsing failure HTTP Browsing Success Ratio HTTP Session Time HTTP browsing transfer duration IP Service setup Time   Video Streaming Streaming Service Non-Accessibility Streaming Reproduction Cut-Off Ratio Streaming service access time Streaming Rebuffering Rate E-mail Send E-mail IP service access time E-mail Upload Service Non-Accessibility E-mail Upload Session Time E-mail Upload Session Failure Ratio

Consumer Services and directory inquiry services quality indicators Service parameters for Helpdesk services Availability and call charges of customer services (N/A) Unanswered calls percentage per customer service line (ETSI EG 202 057-1 v1.3.1) Mean response time per customer service line (ETSI EG 202 057-1 v1.3.1) Customer complaints’ resolution percentage (ETSI EG 202 057-1 v1.3.1) Bill correctness complaints (ETSI EG 202 057-1 v1.3.1) Directory services Response time for directory enquiry services (ETSI EG 202 009-2 v1.2.1) Service parameters for Fixed Telephony services: Supply time for initial connection (ETSI EG 202 057-1 v1.3.1, ETSI EG 201 769 v1.1.2) Faults rate per access line (ETSI EG 202 057-1 v1.3.1, ETSI EG 201 769 v1.1.2) Fault repair time (ETSI EG 202 057-1 v1.3.1, ETSI EG 201 769 v1.1.2) Service parameters for Fixed Broadband services: Faults rate per connection (ETSI EG 202 057-1 v1.3.1, ETSI EG 201 769 v1.1.2) Limitations in use and provision of QoS guarantees/SLAs

Monitoring System: Metis Portal

Monitoring System benefits (1/4) QoS & QoE monitoring system Investors

Monitoring System benefits (2/4) The benefits for the consumers are: Objective information on the quality of in-use electronic services offered by their providers Possibility of objective comparison between service providers Possibility of combining the aforementioned points with geographical criteria Objective information on the quality of customer service of all service providers regardless of technology

Monitoring System benefits (3/4) The benefits for investors are: An objective point of view about the size, extent and potential of electronic communications networks per provider and technology Information on the level of quality of electronic services in the whole Greek Territory Information about the dynamic of the market of electronic communications as a sector of economic development

Monitoring System benefits (4/4) The benefits for EETT are: Transparency in the providers control data in accordance with the terms of licenses per technology Encouraging competition and broadband promotion due to the publication of quality indicators Additional control mechanism

Future work (1/4) The future work of EETT as an NRA concerns: The legislative framework Technical issues of QoS & QoE parameters of mobile networks Technical issues of QoS & QoE parameters of fixed networks

Future work (2/4) Legislative framework The objectives to be fulfilled by the new legislative framework are the following: 1. Fast adaptation of the KPI’s of new technological developments which introduce new communication services 2. An accurate and quick response to new needs in measuring equipment (hardware & software) 3. Easy adjustment to the information needs of end users

Future work (3/4) Technical issues of QoS & QoE parameters of mobile networks Include crowd sourcing measurements Different end users profile

Future work (4/4) Technical issues of QoS & QoE parameters of fixed networks Include crowd sourcing measurements Measurement probes in end user site