1. The Effectiveness of a QoE - Based Video Output Scheme for Audio- Video IP Transmission Shuji Tasaka, Hikaru Yoshimi, Akifumi Hirashima, Toshiro Nunome Department of Computer Science and Engineering, Graduate School of Engineering, Nagoya Institute of Technology ACM Multimedia 2008 Shuji Tasaka, Hikaru Yoshimi, Akifumi Hirashima, Toshiro Nunome Department of Computer Science and Engineering, Graduate School of Engineering, Nagoya Institute of Technology ACM Multimedia 2008

2. Issue Conceal the impairment typified by packet loss, error, and delay in IP networks Two ways these impairments can be remedied at the receiver are: Error-concealment Video frame skipping These techniques approach this issue with a unique tradeoff Temporal vs. Spatial Typically the benefits are exclusive Conceal the impairment typified by packet loss, error, and delay in IP networks Two ways these impairments can be remedied at the receiver are: Error-concealment Video frame skipping These techniques approach this issue with a unique tradeoff Temporal vs. Spatial Typically the benefits are exclusive

3. Scheme Switching between error Concealment and frame Skipping (SCS) utilizes this tradeoff between spatial and temporal quality to cope with video packet loss SCS aims to improve Quality of Experience (QoE) as it depends on both spatial and temporal quality by mixing error concealment and frame skipping Switching between error Concealment and frame Skipping (SCS) utilizes this tradeoff between spatial and temporal quality to cope with video packet loss SCS aims to improve Quality of Experience (QoE) as it depends on both spatial and temporal quality by mixing error concealment and frame skipping

4. Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

5. Principle SCS switches from error-concealment to frame skipping when a percentage of video slices error- concealed in a frame (R c ) exceeds a threshold value (T h ) Frame skipping continues until a new intra-coded frame is decoded Optimal T h is dependent on the content type SCS switches from error-concealment to frame skipping when a percentage of video slices error- concealed in a frame (R c ) exceeds a threshold value (T h ) Frame skipping continues until a new intra-coded frame is decoded Optimal T h is dependent on the content type

6. Error Concealment Strategy I Frames Missing block is interpolated from its neighboring blocks in the current frame P Frames Missing block is replaced by the corresponding block in the previous output frame Instead of motion copy I Frames Missing block is interpolated from its neighboring blocks in the current frame P Frames Missing block is replaced by the corresponding block in the previous output frame Instead of motion copy

7. Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

8. Previous Limitations No audio accompanies the video Output quality determined by PSNR, which is a QoS parameter (non-perceptual) PSNR evaluates no temporal qualities No real-time estimation of QoE Full Reference (objective QoE) models compare stream with the original though not in real-time No audio accompanies the video Output quality determined by PSNR, which is a QoS parameter (non-perceptual) PSNR evaluates no temporal qualities No real-time estimation of QoE Full Reference (objective QoE) models compare stream with the original though not in real-time

9. Aims Find QoE through subject testing Want to estimate QoE through estimation equations, which can allow threshold values to be evaluated in real-time See how accurate estimations are to the subjective measurements Measure the percentage of the selected threshold by way of multiple regression lines Find QoE through subject testing Want to estimate QoE through estimation equations, which can allow threshold values to be evaluated in real-time See how accurate estimations are to the subjective measurements Measure the percentage of the selected threshold by way of multiple regression lines

10. Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

11. Setup 6 videos Video dominant (sports) Audio dominant (music video) Lower frame rate (animation) The second of each with greater Temporal perceptual Information (TI) value Recorded streams output by the media recipient were used as stimuli (432 total): Six different levels of average web traffic (20, 30, 40, 50, 75, 100) Lossy environments > 20 web client processes Four T h (100, 40, 20, 0) Three picture patterns (I, IPPPP, IPPPPPPPPPPPPPP) 6 videos Video dominant (sports) Audio dominant (music video) Lower frame rate (animation) The second of each with greater Temporal perceptual Information (TI) value Recorded streams output by the media recipient were used as stimuli (432 total): Six different levels of average web traffic (20, 30, 40, 50, 75, 100) Lossy environments > 20 web client processes Four T h (100, 40, 20, 0) Three picture patterns (I, IPPPP, IPPPPPPPPPPPPPP)

12. Setup Based on an interval scale derived from: Rating scale Law of categorical judgment Impairment rating-scale: 5 Imperceptible 4 Perceptible, but not annoying 3 Slightly annoying 2 Annoying 1 Very annoying Stimuli which gave large errors of Mosteller’s test are removed Ensures goodness of fit (psychological scale) Based on an interval scale derived from: Rating scale Law of categorical judgment Impairment rating-scale: 5 Imperceptible 4 Perceptible, but not annoying 3 Slightly annoying 2 Annoying 1 Very annoying Stimuli which gave large errors of Mosteller’s test are removed Ensures goodness of fit (psychological scale)

13. Sport 2 (I) Pure frame skipping achieves the highest QoE All other contents performed similarly for I Notice that no other frames are dropped with only (I) Pure frame skipping achieves the highest QoE All other contents performed similarly for I Notice that no other frames are dropped with only (I)

14. Sport 2 (IPPPP) In almost all lossy environments, T h 20% & 40% provide higher QoE than 0%

15. Animation 2 (IPPPP) | Music Video 1 (IPPPP) T h 0% is better than nonzero values in lossy environments Reason: Animation has less than 30 frames & the music video is audio dominant with low video motion

16. Music Video 1 (IPPPPPPPPPPPPPP) At 0%, T h is now less successful Greater QoE at lossy envioronments with a higher T h, as frame skip loses all succeeding P frames At 0%, T h is now less successful Greater QoE at lossy envioronments with a higher T h, as frame skip loses all succeeding P frames

17. Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

18. QoE Estimation Psychological scale is achieved by QoS mapping between the user-level and the application level Mapping is accomplished via multiple regression analysis QoS parameters = independent Psychological scale = dependent Psychological scale is achieved by QoS mapping between the user-level and the application level Mapping is accomplished via multiple regression analysis QoS parameters = independent Psychological scale = dependent

19. Application level QoS parameters These parameters represent both temporal and spatial quality To best estimate QoE, the variables chosen chosen should have low cross-correlations These parameters represent both temporal and spatial quality To best estimate QoE, the variables chosen chosen should have low cross-correlations

20. QoE estimation Principle component analysis allows us to find cross- correlations between introduced independent variables Variables that correlate strongly (a cumulative contribution rate> 90%) are placed in one of the five classes (A-E) Then we calculate a multiple regression line for every combination The line chosen is the one with the greatest multiple correlation coefficient adjusted for degrees of freedom (R*) Picture pattern is not taken into account One parameter from each class must be selected Principle component analysis allows us to find cross- correlations between introduced independent variables Variables that correlate strongly (a cumulative contribution rate> 90%) are placed in one of the five classes (A-E) Then we calculate a multiple regression line for every combination The line chosen is the one with the greatest multiple correlation coefficient adjusted for degrees of freedom (R*) Picture pattern is not taken into account One parameter from each class must be selected

21. QoS parameter classes based on PCA

22. Accuracy of estimation

23. Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

24. Setting the threshold In order to maximize QoE, the appropriate threshold must be chosen This is accomplished by implementing a “learning period” T h is first set to 100%, while the formal T h is computed by estimating the psychological scale values for each threshold At the end of the learning period, the T h with the max psychological value is chosen If there are two T h ’s with the same value, the larger T h is chosen In order to maximize QoE, the appropriate threshold must be chosen This is accomplished by implementing a “learning period” T h is first set to 100%, while the formal T h is computed by estimating the psychological scale values for each threshold At the end of the learning period, the T h with the max psychological value is chosen If there are two T h ’s with the same value, the larger T h is chosen

25. Sport 2 (I) Pure frame skipping is chosen in a lossy environment This was also true in subjective tests Pure frame skipping is chosen in a lossy environment This was also true in subjective tests

26. Sport 2 (IPPPP)

27. Music Video 1 (IPPPP)

28. Music Video 1 (IPPPPP…)

29. Limitations Realized Content contains audio and video QoE is a perceptual QoS Psychological scale uses QoS parameters In QoE estimation, QoS parameters account for spatial and temporal characteristics Estimation with learning period provide real- time QoE assessment Content contains audio and video QoE is a perceptual QoS Psychological scale uses QoS parameters In QoE estimation, QoS parameters account for spatial and temporal characteristics Estimation with learning period provide real- time QoE assessment

30. Conclusions The effectiveness of estimation and human subject testing for SCS’s was examined Subject testing of these estimated SCS should be done With picture pattern’s I and IPPPP the measured and estimated QoE’s are quite similar to each other when utilizing nonlinear multiple regression analysis Picture pattern I favored frame dropping, while IPPP… favored error concealment Threshold value selection must be further investigated The effects of motion copy should be used in future tests Is there a need for QoE estimation? The effectiveness of estimation and human subject testing for SCS’s was examined Subject testing of these estimated SCS should be done With picture pattern’s I and IPPPP the measured and estimated QoE’s are quite similar to each other when utilizing nonlinear multiple regression analysis Picture pattern I favored frame dropping, while IPPP… favored error concealment Threshold value selection must be further investigated The effects of motion copy should be used in future tests Is there a need for QoE estimation?

31. Thank you