QoE Evaluation and Enforcement Framework for Internet Services ITU Workshop on “Performance, QoS and QoE of Emerging Networks and Services Athens, Greece, September 2015 QoE Evaluation and Enforcement Framework for Internet Services Marcus Eckert, Thomas M. Knoll Research Assistant, TU-Chemnitz marcus.eckert@etit.tu-chemnitz.de

Contents Motivation Architecture Overview Architecture Components QMON – QoE Monitoring QRULE – QoE Policy and Rules QEN – QoE Enforcement Results Summary

Motivation Experienced quality of Internet services is crucial for customer satisfaction QoE monitoring and enforcement is thus required for business success Aim: application specific differentiated handling of traffic flows for major Internet services 3GPP standard based procedures using dedicated bearers are hardly used today Default bearer differentiated flow handling is missing Improved QoE measurement and enforcement framework required ISAAR Framework (ISAAR = Internet Service quality Assessment and Automatic Reaction) ISAAR augments existing QoS functions by flow based network centric QoE monitoring and enforcement functions

Architecture Overview Modular service specific QoE management architecture 3 functional components: QoE Monitoring (QMON) – flow detection and measurement, QoE Rules (QRULE) – policy rules and permission checking and QoE Enforcement (QEN) – respective flow manipulation Interworking with existing QoS mechanisms 3GPP PCC Priority marking (DiffServ, Ethernet prio, MPLS prio) Proprietary router QoS support (queueing, scheduling, shaping) SDN based QoS support (e.g. through OpenFlow Action Sets) PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses ! Page 4

Architecture Overview Page 5

Architecture Components – QMON QMON operation Flow classification With and without DPI Centralized / distributed With SDN match and action rules Flow capturing SDN support to tee out flows Flow Monitoring Application specific KPI calculation Standardization: G.102y QMON output Flow Information and QoE estimation currently implemented: “Video QoE estimation”

Measurement Procedure Measurement at end device Most precise Firmware / device specific User involvement Tampering possible Measurement within operator network User and device independent End device (buffer) model required  estimation Reliable results at scale Challenges: constant changes in video streaming (encoding + media container formats)  MPEG DASH PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Measurement Procedure Flow Detection and Classification Deep Packet Inspection (DPI) Built-in or external using 3GPP PCC Gx interface Video Quality Measurement TCP = reliable transport  no longer fine grained pixel and block structure errors Video stall events, duration and inter-stall timing is important Buffer fill level estimation as measurement result PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Measurement Procedure Buffer fill level estimation (exact method) Difference between TCP segment timestamp and playout time encoded in video data within the segment Each segment is traced and processed Use TCP ACKs to increase precision (segment loss, RTT measurement) Buffer model required for fill level estimation (initial buffering, re-buffering and play-out thresholds) Buffer depletion / stall events, re-buffering times and inter-stall timing as measurement results PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Measurement Improvements (speed & accuracy) Buffer fill level estimation (estimation method) Speed-up by chunk based throughput like measurement to avoid decoding every packet (“jump through the stream” method) Full header decoding needed and limited to suitable formats e.g. MP4 Variable look-up interval  trade off between processing speed-up and accuracy Loss of fill level estimation precision especially when high delays or even losses occur due to congestion PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Measurement Improvements (speed & accuracy) Buffer fill level estimation (combined method) Automatic switching between exact and estimation mode of operation to gain the speed-up during good times and to keep the precision during bad networking conditions. There is hardly any difference between the exact and the combined method result. However, the processing load increases (speed-up decreases) for bad case video streaming conditions PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

MOS calculation for Video QoE Mean Opinion Score (MOS) derived from P.862 (ITU-T: Perceptual evaluation of speech quality = pesq) 0 = worst quality / 4.5 = highest quality Assumption: initial 4.5 decreased by negative impact (NI) factor Initial buffering (e.g. 10s) does not raise NI Each following stall decreases the MOS; follows an e-function (exponential function) 𝑀𝑂𝑆=𝑒 − 𝑥 5 +1,5 where x = # of stall events PESQ, Perceptual Evaluation of Speech Quality  ITU P.862 ITU P.862.1  mapping of PESQ score to MOS Subjective tests  ITU P.800

MOS calculation for Video QoE Example video with 5 stalling events Resulting in a bad MOS value PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

MOS calculation for Video QoE Memory effect also influences the negative impact of each stall (D1) as well and dampens the impact the longer the play-out has run smoothly before 𝑀𝑂𝑆=𝑒 − 𝑥 5 +1,5−𝑎∙ 𝑒 𝑡 𝑥=𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑡𝑎𝑙𝑙𝑠 𝑡=𝑡𝑖𝑚𝑒 𝑠𝑖𝑛𝑐𝑒 𝑙𝑎𝑠𝑡 𝑠𝑡𝑎𝑙𝑙 𝛼=𝑚𝑒𝑚𝑜𝑟𝑦 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Architecture Components – QRULE QRULE input Flow information and corresponding QoE estimation from QMON QRULE operation Mapping input flow to service flow classes Check whether QoE enhancement is allowed by general operator policy Determine Per Flow Behaviour (PFB) based on the Enforcement Database of QEN QRULE output PFB specific commands for QEN for 3GPP PCC triggering and/or marking, shaping, dropping and even (SDN/LSP) path selection

Architecture Components – QRULE / PFB Determination Example: PFB commands for marking rules and SDN flow based path selection in high contention situations

Architecture Components – QRULE / PFB Example BE/LE marking CS5 marking EF or equivalent class marking

Architecture Components – QEN QEN input Flow information and QRULE action command set QEN operation Register enforcement capabilities Execute flow manipulation via: 3GPP – PCC (PCRF / PCEF) IETF & IEEE priority marking Automated router configuration with vendor specific QoS capabilities and settings SDN capabilities for marking and Traffic Engineering (TE) Granularity: per-flow or per-class PFB & class PHB / flow & class TE

Architecture Components – QEN QEN flow manipulation options 3GPP – PCC (PCRF / PCEF) QCI marking and/or dedicated bearer setup IETF & IEEE priority marking IP Diffserv, Ethernet priority, MPLS traffic class priority without the need to change the configuration of network elements Synchronized inside/outside GTP tunnel & IPSec tunnel marking Automated router configuration with vendor specific capabilities and settings Cisco / Juniper specific router configuration with flow-specific rules for scheduling, shaping, dropping as well as path (LSP) selection SDN capabilities for marking and TE marking via OpenFlow switch action list configuration flow-specific traffic engineering (LSP selection or flow-specific forwarding paths)

Results Demonstrator setup additionally to the field trials Field trials using packet traces at SGi interface of an operator Demonstrator Lab setup using 2 Laptops Online Buffer fill level estimation and MOS calculation PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Results Results for exact vs. estimation vs. combined method estimation interval stepping processing time # re-buffering events re-buffering time Both algorithms - good case video Human - 0 s Exact 6 s 10 packets 3 s 50 packets 100 packets 150 packets 250 packets Estimation algorithm - bad case video 10 58 s 12 s 56,6 s 56,0 s 54,4 s 53,7 s 9 51,3 s 5 s 6 49,1 s Combined algorithm - bad case video exact 8 s 7 s Results for exact vs. estimation vs. combined method of operation PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Results Results for exact vs. combined method of operation Good case video Bad case video PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Results Results for buffer fill level to MOS QoE calculation Each stall is sharply impacting the experienced quality (MOS score) Re-buffering times gradually impact the MOS score Periods of smooth play-out lead to slight MOS recovery Full recovery is possible if only a few stalls occur and a long smooth play-out follows PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !

Summary ISAAR addresses QoE management for Internet based services 3 components QMON, QRULE, QEN to monitor and manipulate flows Location aware service flow observation and steering Automated network based QoE estimation is feasible and produces accurate results in terms of MOS score calculations and underlying buffer fill level estimations. Aware of 3GPP standardized PCC Interworking with PCRF/PCEF 3GPP interfaces are supported (Sd, UD/Sp, Rx, Gx/Gxx) ISAAR is also able to work independently of 3GPP QoS functionality PCC = „Policy and charging control architecture” - 3GPP TS 23.203 QCI = 5-9 für Non-GBR -> hence, also for default bearer QCI = 1-4 for GBR Bearer -> only dedicated bearer Dedicated bearer only as add-on for default bearer Default bearer gets IP address  dedicated bearer thus gets no additional IP address! Multiple default bearers are possible  leads to multiple IP addresses !