Communication & Multimedia C. -Y. Tsai 2005/8/17 1 MCTF in Current Scalable Video Coding Schemes Student: Chia-Yang Tsai Advisor: Prof. Hsueh-Ming Hang Institute of Electronics, NCTU

Communication & Multimedia C. -Y. Tsai 2005/8/17 2 Outline Overview Overview MCTF in Interframe Wavelet MCTF in Interframe Wavelet MCTF in JSVM MCTF in JSVM Comparison Comparison References References

Communication & Multimedia C. -Y. Tsai 2005/8/17 3 Outline Overview Overview Scalable Video Coding Scalable Video Coding MCTF in Interframe Wavelet MCTF in Interframe Wavelet MCTF in JSVM MCTF in JSVM Comparison Comparison References References

Communication & Multimedia C. -Y. Tsai 2005/8/17 4 Scalable Video Coding Ability to adjust Ability to adjust Different client requirements Different client requirements Scalabilities Scalabilities Rate/SNR Rate/SNR Spatial Spatial Temporal Temporal

Communication & Multimedia C. -Y. Tsai 2005/8/17 5 MCTF MCTF = Motion Compensated Temporal Filtering

Communication & Multimedia C. -Y. Tsai 2005/8/17 6 Rate/SNR Scalability Progressive approximation Progressive approximation GOP HeaderMotion Info.Image Data 300kbps PSNR=32.2 dB 500kbps PSNR=34.6 dB 1000kbps PSNR=38.2 dB

Communication & Multimedia C. -Y. Tsai 2005/8/17 7 Spatial Scalability Wavelet decomposition provides spatial scalability Wavelet decomposition provides spatial scalability Bit-plane Coder

Communication & Multimedia C. -Y. Tsai 2005/8/17 8 HHHH H2H2 H2H2 H3H3 HHH H1H1 H2H2 H2H2 H3H3 L H4H4 15Hz Video Sequence 7.5Hz Video Sequence 30Hz Video Sequence 3.25Hz Video Sequence Temporal Scalability

Communication & Multimedia C. -Y. Tsai 2005/8/17 9 Scalable Video Coding History History MSRA (wavelet) RPI (wavelet) UNSW (wavelet) HHI (AVC-based)  JSVM

Communication & Multimedia C. -Y. Tsai 2005/8/17 10 Approaches An AVC/H.264-based approach (also DCT-based)

Communication & Multimedia C. -Y. Tsai 2005/8/17 11 Approaches A wavelet-based approach with t+2D structure.

Communication & Multimedia C. -Y. Tsai 2005/8/17 12 Approaches A wavelet-based approach with 2D+t structure

Communication & Multimedia C. -Y. Tsai 2005/8/17 13 Lifting Scheme 5/3 lifting scheme 5/3 lifting scheme

Communication & Multimedia C. -Y. Tsai 2005/8/17 14 Outline Overview Overview MCTF in Interframe Wavelet MCTF in Interframe Wavelet Barbell lifting Barbell lifting In-band MCTF In-band MCTF Base-layer structure Base-layer structure MCTF in JSVM MCTF in JSVM Comparison Comparison References References

Communication & Multimedia C. -Y. Tsai 2005/8/17 15 Barbell Lifting Scheme Purpose: Purpose: Improve the accuracy of motion field. Improve the accuracy of motion field. Methods: Methods: Take (5,3) wavelet kernel. Take (5,3) wavelet kernel. Use “ barbell function ” to generate prediction /update values. Use “ barbell function ” to generate prediction /update values.

Communication & Multimedia C. -Y. Tsai 2005/8/17 16 Barbell Lifting Scheme

Communication & Multimedia C. -Y. Tsai 2005/8/17 17 Barbell Lifting Scheme Prediction Stage Update Stage

Communication & Multimedia C. -Y. Tsai 2005/8/17 18 In-Band MCTF Purpose: Purpose: Improve coding performance with spatial scalability Improve coding performance with spatial scalability Methods: Methods: Leaky motion compensation Leaky motion compensation Mode-based temporal filtering Mode-based temporal filtering

Communication & Multimedia C. -Y. Tsai 2005/8/17 19 In-Band MCTF The forming of different quality reference of LL The forming of different quality reference of LL Low quality reference as IP_DIR Low quality reference as IP_DIR High quality reference as IP_LBS High quality reference as IP_LBS

Communication & Multimedia C. -Y. Tsai 2005/8/17 20 In-Band MCTF Leaky motion compensation Leaky motion compensation leaky factor leaky factor Attenuate the prediction based on the unknown information at the decoder Attenuate the prediction based on the unknown information at the decoder make a good trade-off between drifting errors and coding efficiency make a good trade-off between drifting errors and coding efficiency

Communication & Multimedia C. -Y. Tsai 2005/8/17 21 In-Band MCTF Mode-based temporal filtering Mode-based temporal filtering Mode I: Low quality reference Mode I: Low quality reference Mode 2: High quality reference Mode 2: High quality reference Mode is selected by RD cost Mode is selected by RD cost

Communication & Multimedia C. -Y. Tsai 2005/8/17 22 Base-Layer Structure Purpose: Purpose: Coding efficiency improvement in low rates Coding efficiency improvement in low rates AVC compatible AVC compatible Methods: Methods: Insert AVC encoding module into MCTF Insert AVC encoding module into MCTF

Communication & Multimedia C. -Y. Tsai 2005/8/17 23 Base-Layer Structure Encoder Decoder

Communication & Multimedia C. -Y. Tsai 2005/8/17 24 Outline Overview Overview MCTF in Interframe Wavelet MCTF in Interframe Wavelet MCTF in JSVM MCTF in JSVM Base layer structure Base layer structure Inter-layer prediction Inter-layer prediction Adaptive prediction/update steps Adaptive prediction/update steps Comparison Comparison References References

Communication & Multimedia C. -Y. Tsai 2005/8/17 25 Base Layer Structure Purpose Purpose Coding efficiency improvement in low rates Coding efficiency improvement in low rates Compatibility to AVC Compatibility to AVC Methods Methods Unrestricted MCTF (UMCTF) Unrestricted MCTF (UMCTF) Hierarchical B pictures Hierarchical B pictures

Communication & Multimedia C. -Y. Tsai 2005/8/17 26 Base Layer Structure UMCTF UMCTF Update step is omitted. Update step is omitted. Hierarchical B pictures Hierarchical B pictures Fully compatible to AVC Main profile Fully compatible to AVC Main profile Non-dyadic decomposition is available Non-dyadic decomposition is available

Communication & Multimedia C. -Y. Tsai 2005/8/17 27 Non-Dyadic Decomposition

Communication & Multimedia C. -Y. Tsai 2005/8/17 28 Inter-Layer Prediction Purpose Purpose Reduce redundancy between layers Reduce redundancy between layers Methods Methods Inter-layer texture prediction Inter-layer texture prediction Inter-layer motion prediction Inter-layer motion prediction

Communication & Multimedia C. -Y. Tsai 2005/8/17 29 Inter-Layer Prediction

Communication & Multimedia C. -Y. Tsai 2005/8/17 30 Adaptive Prediction/Update Steps Purpose: Purpose: Delay (Memory)control Delay (Memory)control Method: Method: Sub-partitioning of GOP Sub-partitioning of GOP

Communication & Multimedia C. -Y. Tsai 2005/8/17 31 Adaptive Prediction/Update Steps

Communication & Multimedia C. -Y. Tsai 2005/8/17 32 Outline Overview Overview MCTF in Interframe Wavelet video MCTF in Interframe Wavelet video MCTF in JSVM MCTF in JSVM Comparison Comparison Cons and pros Cons and pros Experimental results Experimental results References References

Communication & Multimedia C. -Y. Tsai 2005/8/17 33 Wavelet Based SVC Key features Key features 3D wavelet decomposition 3D wavelet decomposition Open-loop prediction structure Open-loop prediction structure Spatial-temporal resolution scalability Spatial-temporal resolution scalability SNR scalability SNR scalability

Communication & Multimedia C. -Y. Tsai 2005/8/17 34 Wavelet Based SVC Advantages Advantages Nature for multi-resolution scalability Nature for multi-resolution scalability Open-loop prediction structure Open-loop prediction structure Provides elegant SNR scalability without impairing full exploitation of spatial-temporal correlation Provides elegant SNR scalability without impairing full exploitation of spatial-temporal correlation Simplifies the R-D model of the bitstreams. Simplifies the R-D model of the bitstreams. Facilitates the bitstream truncation Facilitates the bitstream truncation each subband is independent with other subbands each subband is independent with other subbands

Communication & Multimedia C. -Y. Tsai 2005/8/17 35 Wavelet Based SVC Disadvantages Disadvantages Decomposition modes (coding modes) selection Decomposition modes (coding modes) selection Texture & side information trade off Texture & side information trade off Intra-prediction Intra-prediction Badly-matched blocks Badly-matched blocks Downsampling filter problems Downsampling filter problems

Communication & Multimedia C. -Y. Tsai 2005/8/17 36 AVC Based SVC Key features Key features MCTF/Hierarchical B structure for temporal scalability MCTF/Hierarchical B structure for temporal scalability Hierarchical B structure with close-loop structure for base layer Hierarchical B structure with close-loop structure for base layer Multiple spatial layers for spatial scalability Multiple spatial layers for spatial scalability Multiple FGS layers at each spatial resolution for SNR scalability Multiple FGS layers at each spatial resolution for SNR scalability DCT coding of all the frames DCT coding of all the frames

Communication & Multimedia C. -Y. Tsai 2005/8/17 37 AVC Based SVC Advantages Advantages All the RDO and intra-prediction can be used. All the RDO and intra-prediction can be used. It guarantees the quality of the first testing point. It guarantees the quality of the first testing point. MPEG filter for low resolution video MPEG filter for low resolution video the target low resolution video is visually good. the target low resolution video is visually good.

Communication & Multimedia C. -Y. Tsai 2005/8/17 38 AVC Based SVC Disadvantages Disadvantages Redundancy between spatial layers Redundancy between spatial layers

Communication & Multimedia C. -Y. Tsai 2005/8/17 39 Experiments

Communication & Multimedia C. -Y. Tsai 2005/8/17 40 Experiments

Communication & Multimedia C. -Y. Tsai 2005/8/17 41 References [1] “ Draft of joint scalable video model (JSVM)3.0 reference encoding algorithm description ”, ISO/IEC JTC1/SC29/WG11, N7311, Poznan, July [2] D. Zhang, J. Xu, H. Xiong, and F. Wu, “ Improvement for in-band video coding with spatial scalability ”, ISO/IEC JTC1/SC29/WG11, M11681, HongKong, Jan [3] V. Bottreau, G. Pau, and J. Xu, “ Vidwav evaluation software manual ”, ISO/IEC JTC1/SC29/WG11, M12176, Poznan, July [4] X. Ji, J. Xu, D. Zhao, and F. Wu, “ Responses of CE1d: base- layer ”, ISO/IEC JTC1/SC29/WG11, M11127, Redmond, July [5] R. Xiong, J. Xu, and F. Wu, “ Coding performance comparison between MSRA wavelet video coding and JSVM ”, ISO/IEC JTC1/SC29/WG11, M11975, Busan, April [6] R. Xiong, J. Xu, and F. Wu, “ Response to VidWav EE1 ”, ISO/IEC JTC1/SC29/WG11, M12286, Poznan, July [7] J. Reichel, K. Hanke and B. Popescu, “ Scalable Video Model V1.0 ”, ISO/IEC JTC1/SC29/WG11, N6372, Munich, March 2004.