Quality of Service of Over-The-Top Services Cyril Lau ITU Athens QoS and QoE of OTT 2015 © Ascom

Presentation Objective and Agenda To discuss the Quality of Experience (QoE) aspects of over-the-top (OTT) and social media services Agenda Rise of OTT and mobile social media apps Factors affecting subscriber experience and quality Case study – OTT QoE measurement Whatsapp, Facebook, youtube Q&A ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

OTT services overview ITU Athens QoS and QoE of OTT 2015 © Ascom

Rise of OTT services 1bn users 340m users OTT driven by: faster, more reliable data connectivity on cellular mobile networks Smartphone devices with better performance, interfaces and larger screens Better codec development for voice and video transmission Method of accessing a new market of users formerly controlled solely by operators 1bn users 340m users ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Types of OTT Services Social Media Twitter, Facebook, Instagram Communications Viber, Skype, WhatsApp Content Netflix, YouTube Trend is that social media and communications functions are consolidating (Facebook purchase of WhatsApp etc) ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Factors affecting Quality of Experience ITU Athens QoS and QoE of OTT 2015 © Ascom

The Carrier Challenge Performance Quality Availability Service quality is critical to maintaining subscriber loyalty, even though some areas are outside operator’s control Customer Satisfaction impacted by: Performance Quality Availability ITU Athens QoS and QoE of OTT 2015 © Ascom TEMS Discovery Professional, October 2010, © Ascom

LTE QoE- QoS - KPIs MAPPING Network Performance Customer Experience Application Layer (integrity, accessibility, retainability) Upper Layers (IP/UDP, IP/TCP) (throughput, delay, Layer 2&3(MAC, RLC, LLC (messaging reports related to QoE dimensions e.g. PDP context, HO info, codec usage) Physical layer (e.g. coverage, interference) QoE QoS KPI ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Factors Affecting Availability Radio environment/coverage Mobility issues with handovers, missing neighbours etc. giving interruptions to data sessions Core Network Including DNS, routing, IP, transport issues Content Provider Servers Capacity, resilience, redundancy issues ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Factors Affecting Performance Throughput Low throughput creates buffering on video streaming, poor download speeds for files Latency Jitter and delay in packets causing interruptions with real-time UDP streams ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

QoE expectations for each OTT is different…… Video Streaming – requires good bandwidth. Not as much in latency as long as the buffer can handle. Online Gaming – requires good latency. Not as much in Bandwidth. Social media – high availability is essential to keep messages/posts up-to-date Online Shopping – Security and availability is essential. Lower requirement on bandwidth and latency ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Other Non Network Related Factors Affecting Quality Quality of Experience is a function of the performance and availability of the underlying network and services, but also other factors such as: Encoding and delivery mechanism (video/voice) Adaptive content systems or lack of Poorly encoded content Devices Poorly designed radios, CPU and memory restrictions, operating systems Service Design Interfaces, protocols ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Case study – OTT QoE measurement ITU Athens QoS and QoE of OTT 2015 © Ascom

What to consider when measuring OTT services? From the consumer side: Can I access the service? Are my messages sent and received in a timely fashion? Do I receive notifications correctly? Is the audio quality good? From the operator side: What is the uptime of the service? How fast are messages delivered? What average MOS/POLQA do customers get in a voice call? ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Presentation Name Month Year ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Whatsapp case study: Whatsapp signon https://r.whatsapp.net Ping r.whatsapp.net DNS lookup r.whatsapp.net DNS lookup Ping WhatsApp signon But is this enough for an operator? We can test the QoE of whatapp services with real devices: ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Whatsapp case study User Device Mobile network operator Whatsapp Clients RAN Core network Whatsapp Server The whatsapp OTT value chain How can an operator ensure whatsapp service delivery over the mobile network? Remember: RAN and Core Network are controlled by the operator User device and whatsapp servers are outside operator’s control ITU Athens QoS and QoE of OTT 2015 © Ascom

Whatsapp case study: 24 by 7 monitoring and location of errors Cell colour indicates average error DNS Ping Whatsapp 1 2 3 4 Locations Vertical red means general failure Horizontal red means error at a specific location 1 2 3 4 Clients Different locations or access methods Different resolution actions in each case ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Whatsapp case study: Results By understanding expected behavior Monitor Master can identify and alert on failures Core network WhatsappService OTT “value chain” DNS time is the time to find the IP address of the server Ping time is the time to locate the server WhatsApp time is the time to connect and login to the server Sign on time to WhatsApp application ~0.2s ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Whatsapp case study: Pinpointing problems Correlation between Ping and Logon times shows when an issue is outside the control of the operator ITU Athens QoS and QoE of OTT 2015 © Ascom

“Be a subscriber” using test equipment to measure real experience Facebook “Be a subscriber” using test equipment to measure real experience This example demonstrating a test to perform a FB update and measure time taken and outcome ITU Athens QoS and QoE of OTT 2015 © Ascom

Facebook benchmarking between carriers Internet Complex multi party interactions Internet Carrier3 Carrier1 Carrier RANs Carrier2 Clients TEMS Monitor Master is used to actively test Facebook interactions for different carriers Different aspects of the service interaction must be supported by the carrier network Interactions include logon, messaging, content upload, homepage display Results used to improve core network and interworking connection Results also important for marketing Facebook “value chain” ITU Athens QoS and QoE of OTT 2015 © Ascom

Active testing access using multiple different carriers SOLUTION OVERVIEW TEMS Monitor Master probes set up with scripts to test Facebook interactions Active testing access using multiple different carriers Tests simulate different clients such including PC and Smartphone and use different bearers Tests exercise all the important functionality of Facebook from the user’s perspective Engineering detail gives radio and IP trace to help pinpoint where issues occur Multiple reports configured specifically to meet the customer’s requirements and add value ITU Athens QoS and QoE of OTT 2015 © Ascom

Detailed results and Value Facebook Carrier RANs Reports shows average performance of key Facebook interactions for different carriers Shows at a glance how different carriers are performing Shows how different aspects of the Facebook service perform Results used to support marketing campaigns Detailed results used to troubleshoot errors and improve MTTR for the owning carrier Differences between bearer also used to troubleshoot Different clients also used to give further insight into performance Facebook “value chain” ITU Athens QoS and QoE of OTT 2015 © Ascom

ITU Athens QoS and QoE of OTT 2015 © Ascom

Case Study - Video test per geographic location 3rd party video streaming service APN 1 APN 2 … APN X GGSN 1 GGSN 2 … GGSN Y Non compressed video for regular users during off peak hours Compressed video for regular users during peak hours 3rd party live TV 3rd party subscription TV Compressed video for “all you can eat data” users at all times TEMS Monitor Master is used to actively test video streaming services Content delivery network must deliver video with different encoding schemes correctly Access point names (APNs) must deliver different service correctly Mobile network must deliver from geographical locations 24x7 Video Compression applied in different scenarios must be within certain QoE ITU Athens QoS and QoE of OTT 2015 © Ascom

Case Study - Video compression per subscriber profile 3rd party video streaming service Video CDN Non compressed video for regular users during off peak hours Video optimisation and load balancing Compressed video for regular users during peak hours 3rd party live TV 3rd party subscription TV Compressed video for “all you can eat data” users at all times TEMS Monitor Master is used to actively test video streaming services Content delivery network must deliver video with different encoding schemes correctly Access point names (APNs) must deliver different service correctly Mobile network must deliver from geographical locations 24x7 Video Compression applied in different scenarios must be within certain QoE ITU Athens QoS and QoE of OTT 2015 © Ascom

Video Quality Comparison: different compressions Reports show key video quality KPIs for different compression methods Video size is reduced for compressed videos MOS is lower for compressed video as expected, but still within acceptable ranges Stalling/re-buffering % is similar for both compressed/non-compressed videos Content Delivery Network Video content Clients Video CDN value chain ITU Athens QoS and QoE of OTT 2015 © Ascom TEMS Discovery Professional, October 2010, © Ascom

Conclusion ITU Athens QoS and QoE of OTT 2015 © Ascom

Conclusion Think about how users interact with OTT services, and what factors affect their experience Identify methods of measuring those factors in the OTT delivery chain, especially the areas within operator’s control Quantify and measure those factors through active testing Act to improve those factors and measure impact ITU Athens QoS and QoE of OTT 2015 © Ascom Presentation Name Month Year

Q&A ITU Athens QoS and QoE of OTT 2015 © Ascom

THANK YOU! ITU Athens QoS and QoE of OTT 2015 © Ascom