The Effectiveness of a QoE - Based Video Output Scheme for Audio- Video IP Transmission Shuji Tasaka, Hikaru Yoshimi, Akifumi Hirashima, Toshiro Nunome Department of Computer Science and Engineering, Graduate School of Engineering, Nagoya Institute of Technology ACM Multimedia 2008 Shuji Tasaka, Hikaru Yoshimi, Akifumi Hirashima, Toshiro Nunome Department of Computer Science and Engineering, Graduate School of Engineering, Nagoya Institute of Technology ACM Multimedia 2008

Issue Conceal the impairment typified by packet loss, error, and delay in IP networks Two ways these impairments can be remedied at the receiver are: Error-concealment Video frame skipping These techniques approach this issue with a unique tradeoff Temporal vs. Spatial Typically the benefits are exclusive Conceal the impairment typified by packet loss, error, and delay in IP networks Two ways these impairments can be remedied at the receiver are: Error-concealment Video frame skipping These techniques approach this issue with a unique tradeoff Temporal vs. Spatial Typically the benefits are exclusive

Scheme Switching between error Concealment and frame Skipping (SCS) utilizes this tradeoff between spatial and temporal quality to cope with video packet loss SCS aims to improve Quality of Experience (QoE) as it depends on both spatial and temporal quality by mixing error concealment and frame skipping Switching between error Concealment and frame Skipping (SCS) utilizes this tradeoff between spatial and temporal quality to cope with video packet loss SCS aims to improve Quality of Experience (QoE) as it depends on both spatial and temporal quality by mixing error concealment and frame skipping

Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

Principle SCS switches from error-concealment to frame skipping when a percentage of video slices error- concealed in a frame (R c ) exceeds a threshold value (T h ) Frame skipping continues until a new intra-coded frame is decoded Optimal T h is dependent on the content type SCS switches from error-concealment to frame skipping when a percentage of video slices error- concealed in a frame (R c ) exceeds a threshold value (T h ) Frame skipping continues until a new intra-coded frame is decoded Optimal T h is dependent on the content type

Error Concealment Strategy I Frames Missing block is interpolated from its neighboring blocks in the current frame P Frames Missing block is replaced by the corresponding block in the previous output frame Instead of motion copy I Frames Missing block is interpolated from its neighboring blocks in the current frame P Frames Missing block is replaced by the corresponding block in the previous output frame Instead of motion copy

Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

Previous Limitations No audio accompanies the video Output quality determined by PSNR, which is a QoS parameter (non-perceptual) PSNR evaluates no temporal qualities No real-time estimation of QoE Full Reference (objective QoE) models compare stream with the original though not in real-time No audio accompanies the video Output quality determined by PSNR, which is a QoS parameter (non-perceptual) PSNR evaluates no temporal qualities No real-time estimation of QoE Full Reference (objective QoE) models compare stream with the original though not in real-time

Aims Find QoE through subject testing Want to estimate QoE through estimation equations, which can allow threshold values to be evaluated in real-time See how accurate estimations are to the subjective measurements Measure the percentage of the selected threshold by way of multiple regression lines Find QoE through subject testing Want to estimate QoE through estimation equations, which can allow threshold values to be evaluated in real-time See how accurate estimations are to the subjective measurements Measure the percentage of the selected threshold by way of multiple regression lines

Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

Setup 6 videos Video dominant (sports) Audio dominant (music video) Lower frame rate (animation) The second of each with greater Temporal perceptual Information (TI) value Recorded streams output by the media recipient were used as stimuli (432 total): Six different levels of average web traffic (20, 30, 40, 50, 75, 100) Lossy environments > 20 web client processes Four T h (100, 40, 20, 0) Three picture patterns (I, IPPPP, IPPPPPPPPPPPPPP) 6 videos Video dominant (sports) Audio dominant (music video) Lower frame rate (animation) The second of each with greater Temporal perceptual Information (TI) value Recorded streams output by the media recipient were used as stimuli (432 total): Six different levels of average web traffic (20, 30, 40, 50, 75, 100) Lossy environments > 20 web client processes Four T h (100, 40, 20, 0) Three picture patterns (I, IPPPP, IPPPPPPPPPPPPPP)

Setup Based on an interval scale derived from: Rating scale Law of categorical judgment Impairment rating-scale: 5 Imperceptible 4 Perceptible, but not annoying 3 Slightly annoying 2 Annoying 1 Very annoying Stimuli which gave large errors of Mosteller’s test are removed Ensures goodness of fit (psychological scale) Based on an interval scale derived from: Rating scale Law of categorical judgment Impairment rating-scale: 5 Imperceptible 4 Perceptible, but not annoying 3 Slightly annoying 2 Annoying 1 Very annoying Stimuli which gave large errors of Mosteller’s test are removed Ensures goodness of fit (psychological scale)

Sport 2 (I) Pure frame skipping achieves the highest QoE All other contents performed similarly for I Notice that no other frames are dropped with only (I) Pure frame skipping achieves the highest QoE All other contents performed similarly for I Notice that no other frames are dropped with only (I)

Sport 2 (IPPPP) In almost all lossy environments, T h 20% & 40% provide higher QoE than 0%

Animation 2 (IPPPP) | Music Video 1 (IPPPP) T h 0% is better than nonzero values in lossy environments Reason: Animation has less than 30 frames & the music video is audio dominant with low video motion

Music Video 1 (IPPPPPPPPPPPPPP) At 0%, T h is now less successful Greater QoE at lossy envioronments with a higher T h, as frame skip loses all succeeding P frames At 0%, T h is now less successful Greater QoE at lossy envioronments with a higher T h, as frame skip loses all succeeding P frames

Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

QoE Estimation Psychological scale is achieved by QoS mapping between the user-level and the application level Mapping is accomplished via multiple regression analysis QoS parameters = independent Psychological scale = dependent Psychological scale is achieved by QoS mapping between the user-level and the application level Mapping is accomplished via multiple regression analysis QoS parameters = independent Psychological scale = dependent

Application level QoS parameters These parameters represent both temporal and spatial quality To best estimate QoE, the variables chosen chosen should have low cross-correlations These parameters represent both temporal and spatial quality To best estimate QoE, the variables chosen chosen should have low cross-correlations

QoE estimation Principle component analysis allows us to find cross- correlations between introduced independent variables Variables that correlate strongly (a cumulative contribution rate> 90%) are placed in one of the five classes (A-E) Then we calculate a multiple regression line for every combination The line chosen is the one with the greatest multiple correlation coefficient adjusted for degrees of freedom (R*) Picture pattern is not taken into account One parameter from each class must be selected Principle component analysis allows us to find cross- correlations between introduced independent variables Variables that correlate strongly (a cumulative contribution rate> 90%) are placed in one of the five classes (A-E) Then we calculate a multiple regression line for every combination The line chosen is the one with the greatest multiple correlation coefficient adjusted for degrees of freedom (R*) Picture pattern is not taken into account One parameter from each class must be selected

QoS parameter classes based on PCA

Accuracy of estimation

Outline Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions Introduction SCS theory Aims of study QoE measurement QoE estimation Threshold selection Conclusions

Setting the threshold In order to maximize QoE, the appropriate threshold must be chosen This is accomplished by implementing a “learning period” T h is first set to 100%, while the formal T h is computed by estimating the psychological scale values for each threshold At the end of the learning period, the T h with the max psychological value is chosen If there are two T h ’s with the same value, the larger T h is chosen In order to maximize QoE, the appropriate threshold must be chosen This is accomplished by implementing a “learning period” T h is first set to 100%, while the formal T h is computed by estimating the psychological scale values for each threshold At the end of the learning period, the T h with the max psychological value is chosen If there are two T h ’s with the same value, the larger T h is chosen

Sport 2 (I) Pure frame skipping is chosen in a lossy environment This was also true in subjective tests Pure frame skipping is chosen in a lossy environment This was also true in subjective tests

Sport 2 (IPPPP)

Music Video 1 (IPPPP)

Music Video 1 (IPPPPP…)

Limitations Realized Content contains audio and video QoE is a perceptual QoS Psychological scale uses QoS parameters In QoE estimation, QoS parameters account for spatial and temporal characteristics Estimation with learning period provide real- time QoE assessment Content contains audio and video QoE is a perceptual QoS Psychological scale uses QoS parameters In QoE estimation, QoS parameters account for spatial and temporal characteristics Estimation with learning period provide real- time QoE assessment

Conclusions The effectiveness of estimation and human subject testing for SCS’s was examined Subject testing of these estimated SCS should be done With picture pattern’s I and IPPPP the measured and estimated QoE’s are quite similar to each other when utilizing nonlinear multiple regression analysis Picture pattern I favored frame dropping, while IPPP… favored error concealment Threshold value selection must be further investigated The effects of motion copy should be used in future tests Is there a need for QoE estimation? The effectiveness of estimation and human subject testing for SCS’s was examined Subject testing of these estimated SCS should be done With picture pattern’s I and IPPPP the measured and estimated QoE’s are quite similar to each other when utilizing nonlinear multiple regression analysis Picture pattern I favored frame dropping, while IPPP… favored error concealment Threshold value selection must be further investigated The effects of motion copy should be used in future tests Is there a need for QoE estimation?

Thank you