1. 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang

2. QoE  $$$ 2 CDN Users Content Providers Content Providers $$$ Better Quality Video Higher Engagement The QoE model Diagram courtesy: Prof. Ramesh Sitaraman, IMC 2012

3. Adapting video bitrates quicker Picking the best server Why do we need a QoE model? 3 Comparing CDNs

4. Traditional Video Quality Metrics 4 Objective Scores (e.g., Peak Signal to Noise Ratio) Subjective Scores (e.g., Mean Opinion Score) Does not capture new effects (e.g., buffering, switching bitrates) Does not capture new effects (e.g., buffering, switching bitrates) User studies not representative of “in-the-wild” experience

5. Internet Video is a new ball game 5 Objective Scores (e.g., Peak Signal to Noise Ratio) Subjective Scores (e.g., Mean Opinion Score) Engagement (e.g., fraction of video viewed) Quality metrics

6. 6 Commonly used Quality Metrics Join Time Average Bitrate Buffering ratio Rate of buffering Rate of switching

7. Which metric should we use? 7 Objective Scores (e.g., Peak Signal to Noise Ratio) Subjective Scores (e.g., Mean Opinion Score) Engagement (e.g., fraction of video viewed) Quality metrics Buffering Ratio, Average bitrate? Today: Qualitative Single-metric

8. Unified and Quantitative QoE Model 8 Objective Scores (e.g., Peak Signal to Noise Ratio) Subjective Scores (e.g., Mean Opinion Score) Engagement (e.g., fraction of video viewed) Quality metrics Buffering Ratio, Average bitrate? ƒ (Buffering Ratio, Average bitrate,…)

9. Outline What makes this hard? Our approach Conclusion 9

10. Complex Engagement-to-metric Relationships Engagement Quality Metric 10 Non-monotonic Engagement Average bitrate Engagement Rate of switching Threshold [Dobrian et al. Sigcomm 2011] Ideal Scenario

11. Complex Metric Interdependencies 11 Join Time Average bitrate Rate of buffering Join Time Bitrate Buffering ratio Rate of buffering Rate of switching

12. Confounding Factors 12 Confounding Factors can affect: 1)Engagement Live and Video on Demand (VOD) sessions have different viewing patterns. CDF ( % of users) Engagement Type of Video Live VOD

13. Confounding Factors 13 Confounding Factors can affect: 1)Engagement 2)Quality Metrics Type of Video Live VOD CDF ( % of users) Join Time Live and Video on Demand (VOD) sessions had different join time distribution.

14. Confounding Factors 14 Confounding Factors can affect: 1)Engagement 2)Quality Metrics 3)Quality Metric  Engagement Type of Video Connectivity DSL/Cable Wireless (3G/4G) Engagement Rate of buffering Users on wireless connectivity were more tolerant to rate of buffering.

15. Confounding Factors 15 Type of Video Connectivity Device Location Popularity Time of day Day of week Need systematic approach to identify and incorporate confounding factors Need systematic approach to identify and incorporate confounding factors

16. Summary of Challenges 1.Capture complex engagement-to-metric relationships and metric-to-metric dependencies. 2.Identify confounding factors 3.Incorporate confounding factors 16

17. Outline What makes this hard? Our approach Conclusion 17

18. 18 Challenge 1: Capture complex relationships

19. Cast as a Learning Problem 19 MACHINE LEARNING Engagement Quality Metrics QoE Model Decision Trees performed the best. Accuracy of 40% for predicting within a 10% bucket.

20. 20 Challenge 2: Identify the confounding factors

21. Test Potential Factors 21 Engagement Confounding Factors Quality Metrics

22. Test Potential Factors 22 Test 1: Relative Information Gain Engagement Confounding Factors Quality Metrics

23. Test Potential Factors 23 Test 1: Relative Information Gain Test 2: Decision Tree Structure Test 3: Tolerance Level Engagement Confounding Factors Quality Metrics

24. Identifying Key Confounding Factors 24 Factor Relative Information Gain Decision Tree Structure Tolerance Level Type of video ✓✓✓ Popularity ✗✗✗ Location ✗✗✗ Device ✗✓✓ Connectivity ✗✗✓ Time of day ✗✗✓ Day of week ✗✗✗

25. Identifying Key Confounding Factors 25 Factor Relative Information Gain Decision Tree Structure Tolerance Level Type of video ✓✓✓ Popularity ✗✗✗ Location ✗✗✗ Device ✗✓✓ Connectivity ✗✗✓ Time of day ✗✗✓ Day of week ✗✗✗

26. 26 Challenge 3: Incorporate the confounding factors

27. Refine the Model 27 MACHINE LEARNING Engagement Quality Metrics QoE Model Confounding Factors Adding as a featureSplitting the data Confounding Factors 1 e.g., Live, Mobile ML Engmnt Quality Metrics Model 2 ML Engmnt Quality Metrics Model 1 ML Engmnt Quality Metrics Model 3 Confounding Factors 2 e.g., VOD, Mobile Confounding Factors 3 e.g., VOD, TV QoE Model

28. Comparing Candidate Solutions 28 Final Model: Collection of decision trees Final Accuracy- 70% (c.f. 40%) for 10% buckets

29. Summary of Our Approach 1.Capture complex engagement-to-metric relationships and metric-to-metric dependencies  Use Machine Learning 2.Identify confounding factors  Tests 3. Incorporate confounding factors  Split 29

30. Evaluation: Benefit of the QoE Model 30 Preliminary results show that using QoE model to select bitrate leads to 20% improvement in engagement

31. Conclusions Internet Video needs a unified and quantitative QoE model What makes this hard? – Complex relationships – Confounding factors (e.g., type of video, device) Developing a model – ML + refinements => Collection of decision trees Preliminary evaluation shows that using the QoE model can lead to 20% improvement in engagement What’s missing? – Coverage over confounding factors – Evolution of the metric with time 31

32. EXTRA SLIDES 32

33. Need a unified and quantitative QoE 33 Increase bitrate Avoid buffering events Don’t switch bitrates often Share the bandwidth

34. Choice of ML Algorithm Matters 34 ML algorithm must be expressive enough to handle the complex relationships and interdependencies Classify engagement into uniform classes Accuracy = # of accurate predictions/ # of cases