SIGCOMM 2011

Outline  Introduction  Datasets and Metrics  Analysis Techniques  Engagement  View Level  Viewer Level  Lessons  Conclusion

Introduction  Internet video has become more and more popular  What impacts engagement?!  Not well understood yet

Introduction  Given the same video, does Quality impact Engagement?!  What are the most critical metrics?  Do these critical metrics differ across genres?  How much does optimizing a metric help?

Datasets and Metrics  Data Collection  A week of data from multiple premium video sites & full census measurement from video player  Video Genres  Live  LVoD  SVoD

Datasets and Metrics  Quality Metrics  Buffering Ratio  Rate of Buffering  Join time  Rendering Quality  Average Bit Rate

Datasets and Metrics  Two Engagement Granularities  View  Play time of a video session  Viewer  Total play time by a viewer in a period of time  Total number of views by a viewer in a period of time

Analysis Techniques  Which metrics matter most  Are metrics independent?  How do we quantify the impact?

Analysis Techniques  Qualitative  Correlation Coefficient  Information Gain  Linear Regression  Quantitative

Analysis Techniques  An simple example

View Level Engagement  Long VoD Content - Correlation

View Level Engagement  Long VoD Content - Correlation  Most important metric  Buffering ratio  Less important metrics  Rendering quality, Join time

View Level Engagement  Long VoD Content – Information Gain

View Level Engagement  Long VoD Content – Information Gain  Bit rate becomes the most important metric  Why??????

View Level Engagement  Live Content

View Level Engagement  Live Content  Buffering Ration remains the most significant  Bitrate and Rate of Buffering matter much more

View Level Engagement  Live Content  Rendering Quality negatively correlated?!  User behavior matters

View Level Engagement  Short VoD Content

View Level Engagement  Short VoD Content  Similar to long VoD content  Buffering ration remains the strongest  Rendering Quality is less important

View Level Engagement  Quantitative Impact  Not apply regression to all the data  Only apply regression to the segment that looks like linear  0-10% range of Buffering ratio

View Level Engagement  Summary  BufRatio is the most important quality metric.  For live content, AvgBitrate in addition to BufRatio is a key quality metric.  A 1% increase in BufRatio can decrease 1 to 3 minutes ofviewing time.  JoinTime has significantly lower impact on view- level engagement than the other metrics  RendQual in live video highlights the need of considering context of actual user and system behavior

Viewer Level Engagement  Buffering ratio vs. play time

Viewer Level Engagement  Buffering ratio vs. # of views

Viewer Level Engagement  Summary  Both the # of views and the total play time are impacted by the quality metrics  Correlation between the engagement metrics and the quality metrics becomes visually and quantitatively more striking at the viewer level  The join time, which seemed less relevant at the view level, has non-trivial impact at the viewer level

Lesson Learned  The need for complementary analysis  All of you are right. The reason every one of you is telling it differently is because each one of you touched a different part of the elephant. So, actually the elephant has all the features you mentioned.  Combination of Correlation and Information gain

Lesson Learned  The importance of context  Lies, damned lies, and statistics  Together with the context of the human and operating factors

Lesson Learned  Toward video quality index  Provide objective index for service providers and researchers ex: MOS  More dimensions  More play type  More Content type  Etc…

Conclusion