1. Quality of Service of Over-The-Top Services Cyril Lau
ITU Athens QoS and QoE of OTT 2015 © Ascom

2. Presentation Objective and Agenda
To discuss the Quality of Experience (QoE) aspects of over-the-top (OTT) and social media services
Agenda
Rise of OTT and mobile social media apps
Factors affecting subscriber experience and quality
Case study – OTT QoE measurement
Whatsapp, Facebook, youtube
Q&A
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

3. OTT services overview
ITU Athens QoS and QoE of OTT 2015 © Ascom

4. Rise of OTT services 1bn users 340m users OTT driven by:
faster, more reliable data connectivity on cellular mobile networks
Smartphone devices with better performance, interfaces and larger screens
Better codec development for voice and video transmission
Method of accessing a new market of users formerly controlled solely by operators
1bn users
340m users
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

5. Types of OTT Services Social Media Twitter, Facebook, Instagram
Communications
Viber, Skype, WhatsApp
Content
Netflix, YouTube
Trend is that social media and communications functions are consolidating (Facebook purchase of WhatsApp etc)
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

6. Factors affecting Quality of Experience
ITU Athens QoS and QoE of OTT 2015 © Ascom

7. The Carrier Challenge Performance Quality Availability
Service quality is critical to maintaining subscriber loyalty, even though some areas are outside operator’s control
Customer Satisfaction impacted by:
Performance
Quality
Availability
ITU Athens QoS and QoE of OTT 2015 © Ascom
TEMS Discovery Professional, October 2010, © Ascom

8. LTE QoE- QoS - KPIs MAPPING
Network Performance
Customer Experience
Application Layer
(integrity, accessibility, retainability)
Upper Layers (IP/UDP, IP/TCP)
(throughput, delay,
Layer 2&3(MAC, RLC, LLC (messaging reports related to QoE dimensions e.g. PDP context, HO info, codec usage)
Physical layer
(e.g. coverage, interference)
QoE
QoS
KPI
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

9. Factors Affecting Availability
Radio environment/coverage
Mobility issues with handovers, missing neighbours etc. giving interruptions to data sessions
Core Network
Including DNS, routing, IP, transport issues
Content Provider Servers
Capacity, resilience, redundancy issues
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

10. Factors Affecting Performance
Throughput
Low throughput creates buffering on video streaming, poor download speeds for files
Latency
Jitter and delay in packets causing interruptions with real-time UDP streams
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

11. QoE expectations for each OTT is different……
Video Streaming – requires good bandwidth. Not as much in latency as long as the buffer can handle.
Online Gaming – requires good latency. Not as much in Bandwidth.
Social media – high availability is essential to keep messages/posts up-to-date
Online Shopping – Security and availability is essential. Lower requirement on bandwidth and latency
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

12. Other Non Network Related Factors Affecting Quality
Quality of Experience is a function of the performance and availability of the underlying network and services, but also other factors such as:
Encoding and delivery mechanism (video/voice)
Adaptive content systems or lack of
Poorly encoded content
Devices
Poorly designed radios, CPU and memory restrictions, operating systems
Service Design
Interfaces, protocols
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

13. Case study – OTT QoE measurement
ITU Athens QoS and QoE of OTT 2015 © Ascom

14. What to consider when measuring OTT services?
From the consumer side:
Can I access the service?
Are my messages sent and received in a timely fashion?
Do I receive notifications correctly?
Is the audio quality good?
From the operator side:
What is the uptime of the service?
How fast are messages delivered?
What average MOS/POLQA do customers get in a voice call?
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

15. Presentation Name Month Year
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

16. Whatsapp case study: Whatsapp signon https://r.whatsapp.net
Ping r.whatsapp.net
DNS lookup r.whatsapp.net
DNS lookup
Ping
WhatsApp signon
But is this enough for an operator?
We can test the QoE of whatapp services with real devices:
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

17. Whatsapp case study User Device Mobile network operator Whatsapp
Clients
RAN
Core network
Whatsapp
Server
The whatsapp OTT value chain
How can an operator ensure whatsapp service delivery over the mobile network? Remember:
RAN and Core Network are controlled by the operator
User device and whatsapp servers are outside operator’s control
ITU Athens QoS and QoE of OTT 2015 © Ascom

18. Whatsapp case study: 24 by 7 monitoring and location of errors
Cell colour indicates average error
DNS
Ping
Whatsapp 1 2 3 4
Locations
Vertical red means general failure
Horizontal red means error at a specific location 1 2 3 4
Clients
Different locations or access methods
Different resolution actions in each case
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

19. Whatsapp case study: Results
By understanding expected behavior Monitor Master can identify and alert on failures
Core network
WhatsappService
OTT “value chain”
DNS time is the time to find the IP address of the server
Ping time is the time to locate the server
WhatsApp time is the time to connect and login to the server
Sign on time to WhatsApp application ~0.2s
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

20. Whatsapp case study: Pinpointing problems
Correlation between Ping and Logon times shows when an issue is outside the control of the operator
ITU Athens QoS and QoE of OTT 2015 © Ascom

21. “Be a subscriber” using test equipment to measure real experience
Facebook
“Be a subscriber” using test equipment to measure real experience
This example demonstrating a test to perform a FB update and measure time taken and outcome
ITU Athens QoS and QoE of OTT 2015 © Ascom

22. Facebook benchmarking between carriers
Internet
Complex multi party interactions
Internet
Carrier3
Carrier1
Carrier RANs
Carrier2
Clients
TEMS Monitor Master is used to actively test Facebook interactions for different carriers
Different aspects of the service interaction must be supported by the carrier network
Interactions include logon, messaging, content upload, homepage display
Results used to improve core network and interworking connection
Results also important for marketing
Facebook “value chain”
ITU Athens QoS and QoE of OTT 2015 © Ascom

23. Active testing access using multiple different carriers
SOLUTION OVERVIEW
TEMS Monitor Master probes set up with scripts to test Facebook interactions
Active testing access using multiple different carriers
Tests simulate different clients such including PC and Smartphone and use different bearers
Tests exercise all the important functionality of Facebook from the user’s perspective
Engineering detail gives radio and IP trace to help pinpoint where issues occur
Multiple reports configured specifically to meet the customer’s requirements and add value
ITU Athens QoS and QoE of OTT 2015 © Ascom

24. Detailed results and Value
Facebook
Carrier RANs
Reports shows average performance of key Facebook interactions for different carriers
Shows at a glance how different carriers are performing
Shows how different aspects of the Facebook service perform
Results used to support marketing campaigns
Detailed results used to troubleshoot errors and improve MTTR for the owning carrier
Differences between bearer also used to troubleshoot
Different clients also used to give further insight into performance
Facebook “value chain”
ITU Athens QoS and QoE of OTT 2015 © Ascom

25. ITU Athens QoS and QoE of OTT 2015 © Ascom

26. Case Study - Video test per geographic location
3rd party video streaming service
APN 1
APN 2 …
APN X
GGSN 1
GGSN 2 …
GGSN Y
Non compressed video for regular users during off peak hours
Compressed video for regular users during peak hours
3rd party live TV
3rd party subscription TV
Compressed video for “all you can eat data” users at all times
TEMS Monitor Master is used to actively test video streaming services
Content delivery network must deliver video with different encoding schemes correctly
Access point names (APNs) must deliver different service correctly
Mobile network must deliver from geographical locations 24x7
Video Compression applied in different scenarios must be within certain QoE
ITU Athens QoS and QoE of OTT 2015 © Ascom

27. Case Study - Video compression per subscriber profile
3rd party video streaming service
Video CDN
Non compressed video for regular users during off peak hours
Video optimisation and load balancing
Compressed video for regular users during peak hours
3rd party live TV
3rd party subscription TV
Compressed video for “all you can eat data” users at all times
TEMS Monitor Master is used to actively test video streaming services
Content delivery network must deliver video with different encoding schemes correctly
Access point names (APNs) must deliver different service correctly
Mobile network must deliver from geographical locations 24x7
Video Compression applied in different scenarios must be within certain QoE
ITU Athens QoS and QoE of OTT 2015 © Ascom

28. Video Quality Comparison: different compressions
Reports show key video quality KPIs for different compression methods
Video size is reduced for compressed videos
MOS is lower for compressed video as expected, but still within acceptable ranges
Stalling/re-buffering % is similar for both compressed/non-compressed videos
Content Delivery Network
Video content
Clients
Video CDN value chain
ITU Athens QoS and QoE of OTT 2015 © Ascom
TEMS Discovery Professional, October 2010, © Ascom

29. Conclusion
ITU Athens QoS and QoE of OTT 2015 © Ascom

30. Conclusion
Think about how users interact with OTT services, and what factors affect their experience
Identify methods of measuring those factors in the OTT delivery chain, especially the areas within operator’s control
Quantify and measure those factors through active testing
Act to improve those factors and measure impact
ITU Athens QoS and QoE of OTT 2015 © Ascom
Presentation Name Month Year

31. Q&A
ITU Athens QoS and QoE of OTT 2015 © Ascom

32. THANK YOU!
ITU Athens QoS and QoE of OTT 2015 © Ascom