A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang

Internet Video is taking off Improve Users’ Quality of Experience

Video Quality Metrics: The State of the Art Objective Score (e.g., Peak Signal to Noise Ratio) Subjective Scores (e.g., Mean Opinion Score)

Problem 1: New Effects, New Metrics PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate

Problem 1: New Effects, New Metrics PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate

Problem 1: New Effects, New Metrics PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate

Problem 1: New Effects, New Metrics PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate

Problem 1: New Effects, New Metrics PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate

Problem 1: New Effects, New Metrics PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate

Problem 1: New Effects, New Metrics PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate Join Time Buffering Ratio Rate of buffering Rate of switching Average bitrate

Problem 2: Opinion Scores  Engagement Opinion Scores - Not representative of “in the wild” experience - Combinatorial explosion of parameters Engagement as replacement for opinion score. (e.g., Play time, customer return rate)

Internet Video QoE Subjective Scores MOS Objective Scores PSNR Subjective score replaced by eng. Objective Scores PSNR

(e.g., Fraction of video viewed) Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) PSNR doesn’t take into account different effects Objective Scores PSNR

(e.g., Fraction of video viewed) Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Replace it with the metrics. But which one? Each one use only cover one aspect of the session. Objective Scores PSNR Join Time, Avg. bitrate, …?

(e.g., Fraction of video viewed) f(Join Time, Avg. bitrate, …) Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Objective Scores PSNR Join Time, Avg. bitrate, …? f(Join Time, Avg. bitrate, …)

(e.g., Fraction of video viewed) f(Join Time, Avg. bitrate, …) Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Objective Scores PSNR Join Time, Avg. bitrate, …? f(Join Time, Avg. bitrate, …)

Outline Need for a unified QoE What makes this hard? Our proposed approach

Challenge: Complex Engagement-to-metric Relationships Quality Metric First main challenge. Relationship between quality metric and eng – we had a simplistic view. But in the real world the relationships are more complex.

Challenge: Complex Engagement-to-metric Relationships Non-monotonic Engagement Average bitrate Engagement Quality Metric Avg bitrate and engagement – non-monotonic [Dobrian et al. Sigcomm 2011]

Challenge: Complex Engagement-to-metric Relationships Non-monotonic Engagement Average bitrate Engagement Quality Metric Engagement Rate of switching Threshold And rate of switching and engagement – threshold effect Measurement study by Dobrian et al. in Sigcomm 2011 show many of these relationships. [Dobrian et al. Sigcomm 2011]

Challenge: Complex Metric Interdependencies Join Time Bitrate Rate of switching Rate of buffering Quality metrics, they are not really independent of each other. Buffering Ratio

Challenge: Complex Metric Interdependencies Join Time Bitrate Rate of switching Rate of buffering Buffering Ratio

Challenge: Complex Metric Interdependencies Join Time Bitrate Rate of switching Rate of buffering Buffering Ratio

Challenge: Complex Metric Interdependencies Join Time Avg. bitrate Rate of switching Rate of buffering There might be several other dependencies. Buffering Ratio

Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies

Casting as a Learning Problem Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies MACHINE LEARNING Engagement Quality Metrics QoE Model

Impact of the ML algorithm Classify engagement into uniform classes Accuracy = # of accurate predictions/ # of cases ML algorithm must be expressive enough to handle the complex relationships and interdependencies

Challenge: Confounding Factors Live and VOD sessions experience similar quality

Challenge: Confounding Factors However, user viewing behavior is very different

Challenge: Confounding Factors Devices User Interest Connectivity Need systematic approach to identify and handle confounding factors

Domain-specific Refinement Engagement Quality Metrics MACHINE LEARNING QoE Model

Domain-specific Refinement Engagement Confounding Factors Quality Metrics MACHINE LEARNING QoE Model

Improved prediction accuracy Refined ML models can handle confounding factors

Concluding Remarks Internet Video needs unified quantitative QoE What makes this hard? Complex engagement-to-metric relationships Complex metric-to-metric interdependencies Confounding factors (e.g., genre, device) Promising start Machine learning + domain-specific refinements Open Challenges Coverage over confounding factors System Design