2. Measuring Quality of Experience for Successful IPTV Deployments Dr. Stefan Winkler

3. Outline Challenges –Digital Video Quality Issues –Traditional Measurements (QoS) vs. Quality of Experience (QoE) Possible Solutions –QoE Measurement Approaches –End-to-end QoE Management Conclusions

4. Digital Video Challenges Demanding traffic profiles High bandwidth streams High traffic volumes Live, VOD Network effects Video impacted heavily with minor network impairments Multi-vendor network complicates diagnosis / troubleshooting High end-user expectations Defined with decades of history Grow rapidly with HD Low tolerance for poor quality New architectures Sensitive video processing devices create possibility for various impairment sources Ad-insertion, middleware Service quality degradations Difficult diagnosis, troubleshooting Rising management and OPEX costs Higher customer churn

5. What Drives End-Users Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.comhttp://qoe.symmetricom.com

6. Service Providers View Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.comhttp://qoe.symmetricom.com

7. Service Providers’ View 7 Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.comhttp://qoe.symmetricom.com

8. Sources of Video Issues Consider all elements for true end-to-end solution

9. Compression Artifacts OriginalMPEG-2H.264

10. PSNR vs. QoE Same amount of distortion (PSNR) – different perceived quality Understand & model human vision system

11. QoS vs. QoE Quality of Service –Network-centric –Delay, packet loss, jitter –Transmission quality –Content agnostic Quality of Experience –Content impairments –Blockiness, Jerkiness, … –End-user quality –Application driven QoS QoE

12. Same network impairments Packet Loss: 1% Delay: 10ms Jitter: 50us Bandwidth: 500kbps Different perceived quality! QoS vs. QoE

13. MDI vs. QoE Media Delivery Index (MDI) MDI consists of two metrics: –Delay Factor (DF) –Media Loss Rate (MLR) MDI limitations: –MDI assumes constant bit rate (CBR) traffic –MDI does not consider video payload or content –MDI values are not intuitive –MDI doesn’t correlate with video quality

14. MDI vs. QoE Media Loss MOS

15. QoS/QoE Cycle Desired QoE Perceived QoE Targeted QoS Delivered QoS End-user Service provider Alignment gap Perception gap Value gapExecution gap Adapted from ITU-T Rec. G.1000 and COM12–C185–E

16. Outline Challenges –Digital Video Quality Issues –Traditional Measurements (QoS) vs. Quality of Experience (QoE) Possible Solutions –QoE Measurement Approaches –End-to-end QoE Management Conclusions

17. Full-Reference Approach Comparison of individual video frames Offline analysis (capture is required) – lab applications High detail and accuracy Alignment procedure Compression/ Transmission System SenderReceiver Full reference information Video Full Ref. Quality Measurement

18. No-Reference Approach Non-intrusive, in-service measurement Real-time monitoring applications No alignment required Compression/ Transmission System SenderReceiver Video No-Ref. Quality Measurement

19. Monitoring applications Correlation of content and network impairments Encrypted environments Compression/ Transmission System SenderReceiver Video Reduced Ref. Measurement Feature Extraction Reduced-Reference Approach

20. Content & Network Metrics 20 "Vision is the most highly developed of the human senses, so people are even more sensitive to flaws in video images than, say, the sound of a telephone conversation.” Ken Wirt, Cisco Vice President Consumer Marketing, Jan 2008 (Correlation Engine)

23. Contrast perception –Visibility of different patterns –Frequency dependencies Masking effects –Interaction of content and impairments –Texture, edges, luminance –Spatial and temporal masking Color perception Spatial frequency [cpd]Temporal frequency [Hz] Sensitivity Vision Modeling Masker contrast Visibility threshold Target contrast Masking curve Threshold without masker

24. End-to-end QoE Deep Content Analysis (pixel by pixel) Source content and encoder / transcoder validation Human Vision System Model Video Quality Reports Content Impairments: Blockiness, blur Jerkiness Freeze/black frame Noise, Color Network Impairments: Loss Delay Jitter Bandwidth Content Stream Analysis: PES inspection PCR jitter etc. Deep Content Analysis (bitstream) Detect content impairments Deep inspection to associate content to timestamps (eg: TS1 = I-Frame) Network (header or stream) Analysis Detect QoS issues Content analysis where possible (unencrypted) Inspection of QoS to associate timestamps to impairments (eg: TS1 = Packet Loss) Q-Advisor Correlation Engine TS1 = I-Frame TS1 = Packet Loss Packet Loss -> I-Frame

25. IPTV QoE Management 25 IssuePossible Causes BlockinessEncoderTranscoderNetwork Loss BlurCamera (focus)EncoderTranscoderSTB (bad filtering) Freeze Frame, JerkinessEncoder (dropped frames) Network lossBad synchronization Black Screen, Blue Screen No Video Signal (source) Ads not insertedMajor network loss ColorEncoderCameraTranscoder Video Noise (analog noise) CameraSTB Noise (digital)EncoderTranscoder AudioMicrophoneEncoder (bad mono stereo encoding Encoder (lip sync) STB 1.0 1. Understand the Service  Is there an issue?  Does it matter? 1. Understand the Service  Is there an issue?  Does it matter? 2. Understand the Problem  What does the customer see?  What is the exact cause? 2. Understand the Problem  What does the customer see?  What is the exact cause? 3. Understand the Solution  What is the impairment source? 3. Understand the Solution  What is the impairment source? 1 2 3 4 5 Very Annoying Annoying Slightly Annoying Perceptible Imperceptible

26. Conclusions QoE is application-driven –Measure both network and content impairments QoE is user-oriented –Measure how end-user perceives service issues End-to-end quality measurement –Cover different impairment sources –Identify problem causes

27. Stefan Winkler swinkler@symmetricom.com swinkler@symmetricom.com Company: qoe.symmetricom.com qoe.symmetricom.com Further Reading: stefan.winkler.net/book.html stefan.winkler.net/book.html Contact Info