1. QoE Evaluation and Enforcement Framework for Internet Services
ITU Workshop on “Performance, QoS and QoE of Emerging Networks and Services Athens, Greece, September 2015
QoE Evaluation and Enforcement Framework for Internet Services
Marcus Eckert, Thomas M. Knoll
Research Assistant, TU-Chemnitz

2. Contents Motivation Architecture Overview Architecture Components
QMON – QoE Monitoring
QRULE – QoE Policy and Rules
QEN – QoE Enforcement
Results
Summary

3. Motivation
Experienced quality of Internet services is crucial for customer satisfaction
QoE monitoring and enforcement is thus required for business success
Aim: application specific differentiated handling of traffic flows for major Internet services
3GPP standard based procedures using dedicated bearers are hardly used today
Default bearer differentiated flow handling is missing
Improved QoE measurement and enforcement framework required
ISAAR Framework (ISAAR = Internet Service quality Assessment and Automatic Reaction) ISAAR augments existing QoS functions by flow based network centric QoE monitoring and enforcement functions

4. Architecture Overview
Modular service specific QoE management architecture
3 functional components:
QoE Monitoring (QMON) – flow detection and measurement,
QoE Rules (QRULE) – policy rules and permission checking and
QoE Enforcement (QEN) – respective flow manipulation
Interworking with existing QoS mechanisms
3GPP PCC
Priority marking (DiffServ, Ethernet prio, MPLS prio)
Proprietary router QoS support (queueing, scheduling, shaping)
SDN based QoS support (e.g. through OpenFlow Action Sets)
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !
Page 4

5. Architecture Overview
Page 5

6. Architecture Components – QMON
QMON operation
Flow classification
With and without DPI
Centralized / distributed
With SDN match and action rules
Flow capturing
SDN support to tee out flows
Flow Monitoring
Application specific KPI calculation
Standardization: G.102y
QMON output
Flow Information and QoE estimation
currently implemented: “Video QoE estimation”

7. Measurement Procedure
Measurement at end device
Most precise
Firmware / device specific
User involvement
Tampering possible
Measurement within operator network
User and device independent
End device (buffer) model required  estimation
Reliable results at scale
Challenges: constant changes in video streaming (encoding + media container formats)  MPEG DASH
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

8. Measurement Procedure
Flow Detection and Classification
Deep Packet Inspection (DPI)
Built-in or external using 3GPP PCC Gx interface
Video Quality Measurement
TCP = reliable transport  no longer fine grained pixel and block structure errors
Video stall events, duration and inter-stall timing is important
Buffer fill level estimation as measurement result
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

9. Measurement Procedure
Buffer fill level estimation
(exact method)
Difference between TCP segment timestamp and playout time encoded in video data within the segment
Each segment is traced and processed
Use TCP ACKs to increase precision (segment loss, RTT measurement)
Buffer model required for fill level estimation (initial buffering, re-buffering and play-out thresholds)
Buffer depletion / stall events, re-buffering times and inter-stall timing as measurement results
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

10. Measurement Improvements (speed & accuracy)
Buffer fill level estimation
(estimation method)
Speed-up by chunk based throughput like measurement to avoid decoding every packet (“jump through the stream” method)
Full header decoding needed and limited to suitable formats e.g. MP4
Variable look-up interval  trade off between processing speed-up and accuracy
Loss of fill level estimation precision especially when high delays or even losses occur due to congestion
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

11. Measurement Improvements (speed & accuracy)
Buffer fill level estimation
(combined method)
Automatic switching between exact and estimation mode of operation to gain the speed-up during good times and to keep the precision during bad networking conditions.
There is hardly any difference between the exact and the combined method result.
However, the processing load increases (speed-up decreases) for bad case video streaming conditions
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

12. MOS calculation for Video QoE
Mean Opinion Score (MOS) derived from P.862 (ITU-T: Perceptual evaluation of speech quality = pesq)
0 = worst quality / 4.5 = highest quality
Assumption: initial 4.5 decreased by negative impact (NI) factor Initial buffering (e.g. 10s) does not raise NI
Each following stall decreases the MOS; follows an e-function (exponential function)
𝑀𝑂𝑆=𝑒 − 𝑥 5 +1,5 where x = # of stall events
PESQ, Perceptual Evaluation of Speech Quality  ITU P.862
ITU P  mapping of PESQ score to MOS
Subjective tests  ITU P.800

13. MOS calculation for Video QoE
Example video with 5 stalling events
Resulting in a bad MOS value
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

14. MOS calculation for Video QoE
Memory effect also influences the negative impact of each stall (D1) as well and dampens the impact the longer the play-out has run smoothly before
𝑀𝑂𝑆=𝑒 − 𝑥 5 +1,5−𝑎∙ 𝑒 𝑡
𝑥=𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑡𝑎𝑙𝑙𝑠
𝑡=𝑡𝑖𝑚𝑒 𝑠𝑖𝑛𝑐𝑒 𝑙𝑎𝑠𝑡 𝑠𝑡𝑎𝑙𝑙
𝛼=𝑚𝑒𝑚𝑜𝑟𝑦 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

15. Architecture Components – QRULE
QRULE input
Flow information and corresponding QoE estimation from QMON
QRULE operation
Mapping input flow to service flow classes
Check whether QoE enhancement is allowed by general operator policy
Determine Per Flow Behaviour (PFB) based on the Enforcement Database of QEN
QRULE output
PFB specific commands for QEN for 3GPP PCC triggering and/or marking, shaping, dropping and even (SDN/LSP) path selection

16. Architecture Components – QRULE / PFB Determination
Example: PFB commands for marking rules and SDN flow based path selection in high contention situations

17. Architecture Components – QRULE / PFB Example
BE/LE marking
CS5 marking
EF or equivalent class marking

18. Architecture Components – QEN
QEN input
Flow information and QRULE action command set
QEN operation
Register enforcement capabilities
Execute flow manipulation via:
3GPP – PCC (PCRF / PCEF)
IETF & IEEE priority marking
Automated router configuration with vendor specific QoS capabilities and settings
SDN capabilities for marking and Traffic Engineering (TE)
Granularity: per-flow or per-class
PFB & class PHB / flow & class TE

19. Architecture Components – QEN
QEN flow manipulation options
3GPP – PCC (PCRF / PCEF)
QCI marking and/or dedicated bearer setup
IETF & IEEE priority marking
IP Diffserv, Ethernet priority, MPLS traffic class priority without the need to change the configuration of network elements
Synchronized inside/outside GTP tunnel & IPSec tunnel marking
Automated router configuration with vendor specific capabilities and settings
Cisco / Juniper specific router configuration with flow-specific rules for scheduling, shaping, dropping as well as path (LSP) selection
SDN capabilities for marking and TE
marking via OpenFlow switch action list configuration
flow-specific traffic engineering (LSP selection or flow-specific forwarding paths)

20. Results Demonstrator setup additionally to the field trials
Field trials using packet traces at SGi interface of an operator
Demonstrator Lab setup using 2 Laptops
Online Buffer fill level estimation and MOS calculation
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

21. Results Results for exact vs. estimation vs. combined method
estimation interval stepping
processing time
# re-buffering events
re-buffering time
Both algorithms - good case video
Human -
0 s
Exact
6 s
10 packets
3 s
50 packets
100 packets
150 packets
250 packets
Estimation algorithm - bad case video 10
58 s
12 s
56,6 s
56,0 s
54,4 s
53,7 s 9
51,3 s
5 s 6
49,1 s
Combined algorithm - bad case video
exact
8 s
7 s
Results for
exact vs.
estimation vs.
combined method
of operation
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

22. Results Results for exact vs. combined method of operation
Good case video Bad case video
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

23. Results Results for buffer fill level to MOS QoE calculation
Each stall is sharply impacting the experienced quality (MOS score)
Re-buffering times gradually impact the MOS score
Periods of smooth play-out lead to slight MOS recovery
Full recovery is possible if only a few stalls occur and a long smooth play-out follows
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !

24. Summary ISAAR addresses QoE management for Internet based services
3 components QMON, QRULE, QEN to monitor and manipulate flows
Location aware service flow observation and steering
Automated network based QoE estimation is feasible and produces accurate results in terms of MOS score calculations and underlying buffer fill level estimations.
Aware of 3GPP standardized PCC
Interworking with PCRF/PCEF
3GPP interfaces are supported (Sd, UD/Sp, Rx, Gx/Gxx)
ISAAR is also able to work independently of 3GPP QoS functionality
PCC = „Policy and charging control architecture” - 3GPP TS
QCI = 5-9 für Non-GBR -> hence, also for default bearer
QCI = 1-4 for GBR Bearer -> only dedicated bearer
Dedicated bearer only as add-on for default bearer
Default bearer gets IP address  dedicated bearer thus gets no additional IP address!
Multiple default bearers are possible  leads to multiple IP addresses !