Project Proposal on Topic: Scalable Extension of HEVC UNDER THE GUIDANCE OF DR. K. R. RAO COURSE: EE5359 - MULTIMEDIA PROCESSING, SPRING 2015 Submitted By: Aanal Desai UT ARLINGTON ID: 1001103728 EMAIL ID: aanal.desai@mavs.uta.edu DEPARTMENT OF ELECTRICAL ENGINEERING UNIVERSITY OF TEXAS, ARLINGTON

Overview An increasing demand for video streaming to mobile devices such as smartphones, tablet computers, or notebooks and their broad variety of screen sizes and computing capabilities stimulate the need for a scalable extension. Modern video transmission systems using the Internet and mobile networks are typically characterized by a wide range of connection qualities, which are a result of the used adaptive resource sharing mechanisms. In such diverse environments with varying connection qualities and different receiving devices, a flexible adaptation of once-encoded content is necessary[2]. The objective of a scalable extension for a video coding standard is to allow the creation of a video bitstream that contains one or more sub-bitstreams, that can be decoded by themselves with a complexity and reconstruction quality comparable to that achieved using single-layer coding with the same quantity of data as that in the sub-bitstream[2].

Introduction SHVC provides a 50% bandwidth reduction for the same video quality when compared to the current H.264/AVC standard. SHVC further offers a scalable format that can be readily adapted to meet network conditions or terminal capabilities. Both bandwidth saving and scalability are highly desirable characteristics of adaptive video streaming applications in bandwidth-constrained, wireless networks[3]. The scalable extension to the current H.264/AVC [4] video coding standard (H.264/SVC) [8] provided resources of readily adapting encoded video stream to meet receiving terminal's resource constraints or prevailing network conditions. The JCT-VC is now developing the scalable extension (SHVC) [5] to HEVC in order to bring similar benefits in terms of terminal constraint and network resource matching as H.264/SVC does, but with a significantly reduced bandwidth requirement[3]

Types of Scalabilities Temporal, Spatial and SNR Scalabilities Spatial scalability and temporal scalability defines cases in which a sub-bitstream represents the source content with a reduced picture size (or spatial resolution) and frame rate (or temporal resolution), respectively[1]. Quality scalability, which is also referred to as signal-to-noise ratio (SNR) scalability or fidelity scalability, the sub-bitstream delivers the same spatial and temporal resolution as the complete bitstream, but with a lower reproduction quality and, thus, a lower bit rate[2].

High-Level Block Diagram of the Proposed Encoder Fig.1 [1]

Inter-layer Intra prediction A block of the enhancement layer is predicted using the reconstructed (and up-sampled) base layer signal.[2] Inter-layer motion prediction:- The motion data of a block are completely inferred using the (scaled) motion data of the co-located base layer blocks, or the (scaled) motion data of the base layer are used as an additional predictor for coding the enhancement layer motion. [2] Inter-layer residual prediction:- The reconstructed (and up-sampled) residual signal of the co-located base layer area is used for predicting the residual signal of an inter-picture coded block in the enhancement layer, while the motion compensation is applied using enhancement layer reference pictures[2].

Intra-BL prediction To utilize reconstructed base layer information, two Coding Unit (CU) level modes, namely intra-BL and intra-BL skip, are introduced[1]. The first scalable coding tool in which the enhancement layer prediction signal is formed by copying or up-sampling the reconstructed samples of the co-located area in the base layer is called Intra-BL prediction mode. [2] For an enhancement layer CU, the prediction signal is formed by copying or, for spatial scalable coding, up-sampling the co-located base layer reconstructed samples. Since the final reconstructed samples from the base layer are used, multi-loop decoding architecture is essential. [2] When a CU in the EL picture is coded by using the intra-BL mode, the pixels in the collocated block of the up-sampled BL are used as the prediction for the current CU. [1]

Fig.2 Intra BL mode [2]

Intra residual prediction In the intra residual prediction mode, the difference between the intra prediction reference samples in the EL and collocated pixels in the up-sampled BL is generally used to produce a prediction, denoted as difference prediction, based on the intra prediction mode. The generated difference prediction is further added to the collocated block in the up-sampled BL to form the final prediction.[1] Fig.3 [1]

Weighted Intra prediction In this mode, the (upsampled) base layer reconstructed signal constitutes one component for prediction. Another component is acquired by regular spatial intra prediction as in HEVC, by using the samples from the causal neighborhood of the current enhancement layer block. The base layer component is low pass filtered and the enhancement layer component is high pass filtered and the results are added to form the prediction.[2] The weights for the base layer signal are set such that the low frequency components are taken and the high frequency components are suppressed, and the weights for the enhancement layer signal are set vice versa. The weighted base and enhancement layer coefficients are added and an inverse DCT is computed to obtain the final prediction[2].

In our implementation, both low pass and high pass filtering happen in the DCT domain, as illustrated in Figure 8. First, the DCTs of the base and enhancement layer prediction signals are computed and the resulting coefficients are weighted according to spatial frequencies.[2] Fig.4 : Weighted intra prediction mode. [2]

Fig.5 Intra difference prediction mode. [2] Intra Prediction In the intra difference prediction, the (up-sampled) base layer reconstructed signal constitutes one component for the prediction. The intra prediction modes that are used for spatial intra prediction of the difference signal are coded using the regular HEVC syntax. [2] Fig.5 Intra difference prediction mode. [2]

Motion vector prediction Our scalable video extension of HEVC employs several methods to improve the coding of enhancement layer motion information by exploiting the availability of base layer motion information[2] In the offered scheme, collocated base layer MVs are used in both the merge mode and the AMVP mode for enhancement layer coding. The base layer MV is inserted as the first candidate in the merge candidate list and added after the temporal candidate in the AMVP candidate list. The MV at the center position of the collocated block in the base layer picture is used in both merge and AVMP modes[1]. In HEVC, the motion vectors are compressed after being coded and the compressed motion vectors are utilized in the TMVP derivation for pictures that are coded later. [1]

References [1] IEEE paper by Jianle Chen, Krishna Rapaka, Xiang Li, Vadim Seregin, Liwei Guo, Marta Karczewicz, Geert Van der Auwera, Joel Sole, Xianglin Wang, Chengjie Tu, Ying Chen, Rajan Joshi “ Scalable Video coding extension for HEVC”. Qualcomm Technology Inc, Data compression conference (DCC)2013, DOC 20-22 March 2013   [2] IEEE paper by, Philipp Helle, Haricharan Lakshman, Mischa Siekmann, Jan Stegemann, Tobias Hinz, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand Fraunhofer Institute for Telecommunications – Heinrich Hertz Institute, Berlin, Germany. “ScalableVideo coding extension of HEVC” Data compression conference (DCC)2013, DOC 20-22 March 2013 { T. Hinz et al, “An HEVC Extension for Spatial and Quality Scalable Video Coding”, Proceedings of SPIE, vol. 8666, pp. 866605-1 to 866605-16, Feb. 2013. } [3] IEEE paper “Scalable HEVC (SHVC)-Based Video Stream Adaptation in Wireless Networks” by James Nightingale, Qi Wang, Christos Grecos Centre for Audio Visual Communications & Networks (AVCN). 2013 IEEE 24th International Symposium on Personal, Indoor and Mobile Radio Communications: Services, Applications and Business Track [4] T. Weingand et al, "Overview of the H.264/AVC video coding standard," IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 560-576, July 2003. [5] T. Hinz et al, "An HEVC extension for spatial and quality scalable video coding," Proc. SPIE Visual Information Processing and Communication IV, Feb. 2013. [6] B. Oztas et al, "A study on the HEVC performance over lossy networks," Proc. 19th IEEE International Conference on Electronics, Circuits and Systems (ICECS), pp.785-788, Dec. 2012.

[7] J. Nightingale et al, "HEVStream: a framework for streaming and evaluation of high efficiency video coding (HEVC) content in loss-prone networks," IEEE Trans. Consum. Electron., vol.58, no.2, pp.404-412, May 2012. [8] H.Schwarz et al, “Overview of the scalable extension of the H.264/AVC standard,”IEEE Trans Circuits Syst Video Technology, vol.17, pp.1103-1120,Sept 2007 [9] J. Nightingale et al, "Priority-based methods for reducing the impact of packet loss on HEVC encoded video streams," Proc. SPIE Real-Time Image and Video Processing 2013, Feb. 2013. [10] T.Schierl et al, “Mobile Video Transmission coding”, IEEE Trans. Circuits Syst. Video Technol., vol. 1217, Sept 2007. [11] J. Chen, K. Rapaka, X. Li, V. Seregin, L. Guo, M. Karczewicz, G. Van der Auwera, J. Sole, X. Wang, C. J. Tu, Y. Chen, “Description of scalable video coding technology proposal by Qualcomm (configuration 2)”, Joint Collaborative Team on Video Coding, doc. JCTVC- K0036, Shanghai, China, Oct. 2012. [12] ISO/IEC JTC1/SC29/WG11 and ITU-T SG 16, “Joint Call for Proposals on Scalable Video Coding Extensions of High Efficiency Video Coding (HEVC)”, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. N12957 or ITU-T SG 16 Doc. VCEG-AS90, Stockholm, Sweden, Jul. 2012. [13] A. Segall, “BoG report on HEVC scalable extensions”, Joint Collaborative Team on Video Coding, doc. JCTVC-K0354, Shanghai, China, Oct. 2012 . [14] H. Schwarz, D. Marpe, T. Wiegand, “Overview of the Scalable Video Coding Extension of the H.264/AVC Standard”, IEEE Trans. Circuits and Syst. Video Technol., vol. 17, no. 9, pp. 1103􏰃1120, 2007.

[15] D. Hong, W. Jang, J. Boyce, A. Abbas, “Scalability Support in HEVC”, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JCTVC-F290, Torino, Italy, Jul. 2011. [16] G. J. Sullivan, J.-R. Ohm, W.-J. Han, T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Trans. Circuits and Syst. Video Technol., to be published. [17] J. Boyce, D. Hong, W. Jang, A. Abbas, “Information for HEVC scalability extension,” Joint Collaborative Team on Video Coding, doc. JCTVC-G078, Nov. 2011. [18] G.J. Sullivan et al, “Standardized extensions of High Efficiency Video Coding (HEVC)”, IEEE J-STSP, vol. 7, no. 6, pp. 1001 – 1016, Dec. 2013. (H.265/HEVC) Tutorial by Madhukar Budagavi m.budagavi@samsung.com http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/budagaviiscas2014ppt.pdf