Scene-dependent Frequency Weighting for Subjective Quality Improvement of MPEG-4 Fine- Granularity-Scalability Sharon Peng and Mihaela van der Schaar Philips Research IEEE International Conference on Image Processing (ICIP) September 2002

Outline Introduction Introduction Frequency Weighting Frequency Weighting Frequency Weighting Matrix Selection Frequency Weighting Matrix Selection Scene Characteristic Dependent Adaptive Frequency Weighting Scene Characteristic Dependent Adaptive Frequency Weighting Experimental Results Experimental Results Conclusions Conclusions

Introduction This paper proposes a novel scene- characteristic-dependent adaptive FW method aimed at improving the visual quality of FGS. This paper proposes a novel scene- characteristic-dependent adaptive FW method aimed at improving the visual quality of FGS. Abbreviation Abbreviation BL: Base Layer BL: Base Layer EL: Enhancement Layer EL: Enhancement Layer SL: Single Layer SL: Single Layer FW: Frequency Weighting FW: Frequency Weighting

Frequency Weighting Frequency weighting exploits the different human visual system sensitivities to various frequencies to improve the FGS visual quality at low and medium bit-rates. Frequency weighting exploits the different human visual system sensitivities to various frequencies to improve the FGS visual quality at low and medium bit-rates. The “Frequency Weighting” method has been standardized that allows the prioritized transmission of “low frequency” DCT coefficient. The “Frequency Weighting” method has been standardized that allows the prioritized transmission of “low frequency” DCT coefficient.

Frequency Weighting (Cont.)

Frequency Weighting Matrix Selection

Frequency Weighting Matrix Selection (Cont.) P(i) : the original BL DCT coefficients P r (i) : the reconstructed BL DCT coefficients i : the location of the DCT coefficient along the zigzag scan line W : the BL weighting matrix mquant : the BL quantization step A : a constant D(i) : the EL DCT residuals

Frequency Weighting Matrix Selection (Cont.) Very low BL bit-rate coding: Very low BL bit-rate coding: Q w (i) : very large value Q w (i) : very large value many BL DCT blocks are quantized to zero many BL DCT blocks are quantized to zero the EL DCT residuals resemble the original BL DCT coefficients the EL DCT residuals resemble the original BL DCT coefficients the EL DCT residual coding is intra-frame coded the EL DCT residual coding is intra-frame coded the same quality improvement methods (e.g. frequency weighted quantization matrix) used for BL intra-frame coding the same quality improvement methods (e.g. frequency weighted quantization matrix) used for BL intra-frame coding Very high BL bit-rate coding Very high BL bit-rate coding Q w (i) : very small value Q w (i) : very small value many DCT blocks have very small residuals many DCT blocks have very small residuals the BL quality is considerably higher the BL quality is considerably higher the effect of frequency weighting on the perceptual quality of the BL becomes negligible the effect of frequency weighting on the perceptual quality of the BL becomes negligible Consequently, the BL bit-rate plays an important role in selecting the FW matrix. Consequently, the BL bit-rate plays an important role in selecting the FW matrix.

Frequency Weighting Matrix Selection (Cont.) In most FGS applications, the BL is coded at a low bit-rate R BL and a large EL is used for gradually improving the quality as the transmission bit-rate R t increases. In most FGS applications, the BL is coded at a low bit-rate R BL and a large EL is used for gradually improving the quality as the transmission bit-rate R t increases. The quality gap between non-scalable single layer coding and FGS is for most sequences around R t = 3 R BL The quality gap between non-scalable single layer coding and FGS is for most sequences around R t = 3 R BL Therefore, this paper propose a FW matrix selection method Therefore, this paper propose a FW matrix selection method using a low base-layer bit-rate using a low base-layer bit-rate (e.g. R BL1 = 100 kbps for CIF sequences at 10 frames/sec) aimed at minimizing the visual quality gap between FGS and SL for the worst-case scenario (R t = R BL2 = 3 R BL1 = 300 kbps) aimed at minimizing the visual quality gap between FGS and SL for the worst-case scenario (R t = R BL2 = 3 R BL1 = 300 kbps)

Frequency Weighting Matrix Selection (Cont.) : the average DCT residual per frame : the average DCT residual per frame D fb (i, r) : the DCT residual of frame number f in block b at location i in the 8 x 8 DCT block at bit-rate r N f : the number of frames with the same scene characteristic N b : the number of blocks of each frame : : the residual differences at the two transmission bit-rates the larger residuals (at the lower transmission bit-rate) need to be compensated by the FGS EL  Denote  

Frequency Weighting Matrix Selection (Cont.)

Scene Characteristic Dependent Adaptive Frequency Weighting SC1: Scenes with lower brightness tend to have smaller amplitudes for DCT residuals. SC1: Scenes with lower brightness tend to have smaller amplitudes for DCT residuals. SC2: Scenes containing high frequency textures tend to have high DCT residuals SC2: Scenes containing high frequency textures tend to have high DCT residuals across most frequencies. across most frequencies. SC3: Scenes containing fast motion objects exhibit large DC and lower AC residuals. SC3: Scenes containing fast motion objects exhibit large DC and lower AC residuals. SC4: Scenes that do not fall in either one of the above mentioned classes, i.e., have SC4: Scenes that do not fall in either one of the above mentioned classes, i.e., have neither high frequency, fast motion texture or low brightness, and that have neither high frequency, fast motion texture or low brightness, and that have mostly low and moderate DCT residuals, were considered separately. mostly low and moderate DCT residuals, were considered separately.

Scene Characteristic Dependent Adaptive Frequency Weighting (Cont.)

Experimental Results Experimental cases: Experimental cases: NFWFGS: Non FW FGS NFWFGS: Non FW FGS do not use FW matrix do not use FW matrix SFWFGS: Single FW FGS (i.e., non-adaptive) SFWFGS: Single FW FGS (i.e., non-adaptive) SFWFGS1: use FW1 matrix SFWFGS1: use FW1 matrix SFWFGS2: use FW2 matrix SFWFGS2: use FW2 matrix SFWFGS3: use FW3 matrix SFWFGS3: use FW3 matrix AFWFGS: Adaptive FW FGS AFWFGS: Adaptive FW FGS do not use FW matrix do not use FW matrix use FW1 matrix use FW1 matrix use FW2 matrix use FW2 matrix use FW3 matrix use FW3 matrix Subjective assessment score: Subjective assessment score: Test coding artifacts including blockiness and flickering Test coding artifacts including blockiness and flickering very annoying: 1 very annoying: 1 annoying: 2 annoying: 2 slightly annoying: 3 slightly annoying: 3 perceptible but not annoying: 4 perceptible but not annoying: 4 imperceptible: 5 imperceptible: 5

Experimental Results (Cont.)

Conclusion This paper presents a novel scene-characteristic- dependent adaptive FW method aimed at improving the visual quality of FGS. This paper presents a novel scene-characteristic- dependent adaptive FW method aimed at improving the visual quality of FGS. Different FW should be used depending on the video sequence characteristics (texture, motion and brightness), each scene is classified in one of the predefined categories with a different FW matrix. Different FW should be used depending on the video sequence characteristics (texture, motion and brightness), each scene is classified in one of the predefined categories with a different FW matrix.