Fast Intra Coding Based on Reference Samples Similarity in HEVC Presented By Advising Professor: Dr. K. R. Rao Nikita Thakur (1001102923)

Outline HEVC Picture Partitioning Intra Prediction Intra Mode Coding Angular Intra Prediction Proposed Algorithm Test sequences Future Work References

List of Acronyms and Abbreviations AVC: Advanced Video Coding BD-PSNR: Bjøntegaard Delta Peak Signal to Noise Ratio  BD-bitrate: Bjøntegaard Delta bit rate CABAC: Context Adaptive Binary Arithmetic Coding CTU: Coding Tree Unit CU: Coding Unit VCIP: Visual Communications and Image Processing DVD: Digital Video Disk EUSIPCO: European Signal Processing Conference HD: High Definition HDR: High Dynamic Range HEVC: High Efficiency Video Coding HM: HEVC Test Model ICICS: International Conference on Information, Communications and Signal Processing JCT: Joint Collaborative Team

JCT-VC: Joint Collaborative Team on Video Coding JPEG: Joint Photographic Experts Group MPEG: Moving Picture Experts Group PB: Prediction Block PU: Prediction Unit RD: Rate Distortion RDO: Rate-distortion Optimization SATD: Sum of Absolute Transformed Differences TB: Transform Block TU: Transform Unit UHD: Ultra High Definition

Overview of HEVC HEVC is latest video standard developed by Joint Collaborative Team on Video Coding (JCT-VC) in January, 2013 [1][2][3][4] . HEVC has been designed to address essentially all existing applications of H.264/MPEG-4 AVC [14] and to particularly focus on two key issues: increased video resolution and increased use of parallel processing architectures. The major achievements of the HEVC standard in comparison with the H.264 [1] standard are flexible prediction modes, larger transform block sizes, better partitioning options, improved interpolation and deblocking filters, prediction, signaling of modes and motion vectors and support efficient parallel processing [1].

Fig 1:Typical video coding

Picture Partitioning HEVC supports highly flexible partitioning of a video sequence. Each frame of the sequence is split up into rectangular or square regions (units or blocks) [12], each of which is predicted from previously coded data. After prediction, any residual information is transformed and entropy encoded. Each coded video frame, or picture, is partitioned into tiles and/or slices, which are further partitioned into coding tree units (CTUs). A coding tree unit can be subdivided into square regions known as coding units (CUs) using a quad-tree structure. Each CU is predicted using inter or intra prediction and transformed using one or more transform units. Figure 2.1: Picture, Slice, Coding Tree Unit (CTU), Coding Unit (CU) [12]  

Figure 2.2: Prediction Block for Intra Prediction [19] Prediction Unit (PU) is the basic unit for prediction An intra-predicted CU can be split into PBs only in two modes: either the PB is the same as the CB size, or the CB is split onto four smaller PBs. To predict a new prediction block (PB), intra-picture prediction uses the previously decoded boundary samples from spatially neighboring image data (in the same picture). Figure 2.2: Prediction Block for Intra Prediction [19]

Figure 2.3 shows the partitioning of a 32x32 CU intoPUs and TUs. A transform unit (TU) is the basic unit for the transform and quantization processes. The size andt he shape of the TU depend on the size of the PU Figure 2.3 shows the partitioning of a 32x32 CU intoPUs and TUs. Figure 2.3:Partitioning of 32x32 CU into PUs and TUs [14]

Table 1: Mode allocation [7] Intra Prediction HEVC employs 35 different intra modes to predict a PB: 33 Angular modes, Planar mode, and DC mode. The next table shows the mode name with their corresponding intra prediction mode index as by the convention used throughout the standard [7]. Table 1: Mode allocation [7] At the end the video encoder will choose the intra prediction mode that provides the best Rate-Distortion performance.

Intra mode coding Intra coding can be depicted in Fig. 2. In the rough mode decision (RMD) stage, the current block is predicted with mode 0 firstly. Then the differences between the original block and the prediction are obtained. Afterward Hadamard transform is applied to the difference block, and finally the sum of the absolute transformed difference (SATD) is calculated. After the process is repeated for all the 35 intra modes, the N modes with the minimum SATD are selected to be candidates for RDO. N is determined according to PU sizes as eight for PUs of size 4×4 and 8×8, and three for PUs of other sizes. The MPM will also be added in the N modes if they are not included yet.

Intra mode coding Fig 3:Process of Intra coding

after the substitution process Reference Samples Generation and Substitution Intra sample prediction is performed by extrapolating sample values from the reconstructed reference samples depending on the prediction mode Figure 3.1:An example of reference sample substitution process. Non-available reference samples are marked as grey: (a) reference samples before the substitution process (b) reference samples after the substitution process

Contd... There are three cases in which reference samples may not be available. The first case is when the reference samples at picture or slice boundaries are unavailable; in this case a nominal average sample value is used as a replacement for the reference samples depending on the sample bit-depth; for an 8-bit data, this is 128. The second case is when just the left boundary references are missing; these reference samples are generated copying the samples from the closest available reference samples above. The third case is when just the top boundary references are missing; these reference samples are generated copying the samples from the closest available reference samples on the left [10].

Angular Intra Prediction Figure 4: Angle definitions of angular intra prediction in HEVC numbered from 2 to 34 and the associated displacement parameters. H and V are used to indicate the horizontal and vertical directionalities, respectively, while the numeric part of the identifier refers to the sample position displacements in 1/32 fractions of sample grid positions[11]

Figure 4.1: Example of diagonal orientation [13]

Contd... Two arrays of reference samples are used: - the row of samples lying above the current PU to be predicted - the column of samples lying to the left of the same PU One of the reference arrays is defined to be the main array and the other array the side array: - vertical prediction: the reference row above the PU is called the main array and the reference column to the left of the same PU is called the side array. - horizontal prediction: the reference column to the left of the PU is called the main array and the reference row above the PU is called the side array. Predicted pixels are computed by using the linear interpolation of the reference top or left samples. Only the main array samples are used for prediction when the intra prediction angle is positive.

Contd... The prediction process when the angle value is negative, - The main array is extended by projecting samples from the side array onto it according to the prediction direction. - This projection entails copying a subset of the samples in the side array into the main array. -Only the extended main array is used for the prediction thus

Sample Prediction for Angular Prediction Mode Predicted sample values p[x][y] are obtained by projecting the location of the sample p[x][y] to the reference sample array applying the selected prediction direction and interpolating a value for the sample at 1/32 sample position accuracy[8]. Prediction for horizontal modes (modes 2-17) is given by: And Sample Prediction for vertical modes (modes 18–34) is given by: where i is the projected integer displacement on row y (for vertical modes) or column x (for horizontal modes) and calculated as a function of angular parameter A as follows: f represents the fractional part of the projected displacement on the same row or column and is calculated as:

Proposed Algorithm[33] Fig 5:An example of sample prediction for angular prediction Modes[33].

It can be seen from Fig. 5 that not all the samples in the reference array are needed for the prediction. The array of needed reference samples (ANRS) varies with different angular prediction modes. If the ANRS are similar, then they would generate similar predictions. So, we don’t have to apply the RMD and RDO to all the modes which havesimilar ANRSs. Only one of them is needed. Even though other modes are discarded, it will not result in too much degradation of quality and BD-rate. Fig. 5.1 shows an original block and the reference samples. Fig. 5.2 shows the ANRSs for the block. It can be seen that the ANRSs in the red rectangle are similar. These similar ANRSs would generate similar prediction. Only one of the angular modes needs to be evaluated in the RMD stage. In the proposed algorithm, only the first one (intra prediction mode 25) is remained and other are discarded.

Fig. 5.1. A block and the reference samples[33] Fig.5.2. The ANRSs of one block. The numbers indicates the corresponding intra modes in HEVC[33].

Mean and variance are used to judge whether two[33] ANRSs are similar. If the means of two ANRSs are close enough and the variances of both ANRSs are small enough, the two ANRSs are similar. The condition is defined as follows[33]: Mean(ANRS[i]) – Mean(ANRS[j])| < MeanThreshold && Variance(ANRS[i]) < VarianceThreshold (11) && Variance(ANRS[i]) < VarianceThreshold where ANRS[i] means the ANRS for intra mode i (i > 1and i < 35).

Fig. 5.3. The samples used in DC mode Fig. 5.4. The samples used in planar mode. DC and Planar Prediction[33] The cases of DC and planar mode are different from angular modes.For DC mode, the ANRS used in prediction is shownin the red rectangles in Fig.5.3. Only the samples which are on the left and above of the block are calculated for mean value. The variance always equals 0 in this case, since the prediction is determined by mean only. For the planar mode, the ANRS used in prediction is shown in the red rectangles in Fig. 5.4. The samples in the bottom row are copies of the bottom-left reference sample and the samples in the right column are copies of top-right reference sample. The mean and variance are calculated for the samples in the red rectangles.

The Process of Skipping Intra Modes[33] As is shown in Fig. 5.5, the proposed algorithm is based on rough mode decision (RMD). The algorithm processes each mode from mode 0 to mode 34. Before predicting a block using a certain mode, the mean and variance of the ANRS are calculated. If the ANRS of the current mode is similar to that of a preceding mode, the current mode is discarded. So, the Hadamard cost is calculated only for a subset of all the intra modes. The encoding time is there by reduced. Fig. 5.5. The flowchart of the proposed algorithm[33]

Future Work Similar algorithms can be developed for fast inter-prediction in which the RD cost of the different modes in inter-prediction are explored, and depending upon the adaptive threshold [34], mode decision can be terminated resulting in less encoding time and reduced complexity combining with the above proposed algorithm. There are a numerous number of Algorithms been developed for fast intra mode coding [14][15[16[17[18][19] .We can compare the Performance of each of these algorithms and compare them based on experimental results.

Test sequences BasketBallDrillText

Racehorses

Kimono

References [1] G. J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Transactions on circuits and systems for video technology, vol. 22, no.12, pp. 1649 – 1668, Dec .2012. [2] K. R. Rao, D. N. Kim and J. J. Hwang, “Video Coding Standards: AVS China, H.264/MPEG-4 Part10, HEVC, VP6, DIRAC and VC-1”, Springer, 2014. [3] V. Sze, M. Budagavi, and G. J. Sullivan, "High Efficiency Video Coding (HEVC): Algorithms and Architectures", Springer, 2014. [4] M. Wien, "High Efficiency Video Coding: Coding Tools and Specification”, Springer, 2014. [5] JVT Draft ITU-T recommendation and final draft international standard of joint video specification (ITU-T Rec. H.264-ISO/IEC 14496-10 AVC), March 2003, JVT-G050 available on http://ip.hhi.de/imagecom_G1/assets/pdfs/JVT-G050.pdf. [6] I.E.G.Richardson, “H.264 and MPEG-4 Video Compression Video Coding for Next generation Multimedia”, New York, Wiley, 2003 [7] G.J. Sullivan and T.Wiegand, “Rate-Distortion Optimization for Video Compression”. IEEE Signal Processing Magazine, vol. 15, pp.74-90, Nov 1998. [8] J.-R. Ohm et al, “Comparison of the coding efficiency of video coding standards – including high efficiency video coding (HEVC)”, IEEE Transactions on circuits and systems for video technology, vol. 22, pp.1669-1684, Dec. 2012. Software and data for reproducing selected results can be found at ftp://ftp.hhi.de/ieee-tcsvt/2012.

[9] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T & ISO/IEC JCTVC J0292r1, July 2012. [10] I. E. G. Richardson, “The H.264 Advanced Video Compression Standard,” II Edition, Wiley,2010. [11] J. Lainema et al, “Intra coding of the HEVC standard”, IEEE Transactions on circuits and systems for video technology, vol. 22, pp.1792- 1801, Dec. 2012. [12] P. Helle et al, “Block merging for quadtree-based partitioning in HEVC”, IEEE Transactions on circuits and systems for video technology, vol. 22, pp.1720-1731, Dec. 2012. [13] G. Van Wallendael et al , "3D video compression based on high efficiency video coding," IEEE Trans. Consumer Electronics, vol. 58, no.1, pp.137-145, Feb. 2012 [14] Y. Piao, J. Min and J. Chen, "Encoder improvement of unified intra prediction," Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, Document: JCTVCC207,Guangzhou, Oct. 2010. [15] L. Zhao et al, "Fast Mode Decision Algorithm for Intra Prediction in HEVC," IEEE Visual Communications and Image Processing (VCIP), pp. 1-4, Nov. 2011. [16] J. Kim, J. Yang ,H. Lee and B. Jeon, "Fast intra mode decision of HEVC based on hierarchical structure," 8th International Conference on Information, Communications and Signal Processing (ICICS), pp. 1-4, Dec. 2011.

[17] T. L. da Silva, L. V. Agostini and L. A  [17] T. L. da Silva, L. V. Agostini and L. A. da Silva Cruz, "Fast HEVC intra prediction mode decision based on EDGE direction information," Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pp.1214-1218, Aug. 2012. [18] W. Jiang and H. Ma, Y. Chen, "Gradient based fast mode decision algorithm for intra predicion in HEVC," 2nd International Conference on Consumer Electronics, Communications and Networks (CECNet), pp. 1836-1840, Apr. 2012. [19]L. Shen, Z. Zhang and P. An, “Fast CU size decision and mode decision algorithm for HEVC intra coding’, IEEE Trans. on consumer electronics, vol.59, pp.207-213, Feb. 2013 [20]G J. Sullivan et al, “Standardized Extensions of HEVC”, IEEE Journal of Selected topics in Signal Processing, Vol.7, No.6, pp.1001-1016, Dec. 2013. [21] HM C++ Code: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ [22] HM software manual: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/trunk/doc/software-manual.pdf [23] H. Zhang and Z. Ma, “Fast intra mode decision for high-efficiency video coding”, IEEE Transactions on circuits and systems for video technology, Vol.24, pp.660-668, Apr. 2014. [24]JCT-VC documents can be accessed.Available: http://phenix.intevry.fr/jct/doc_end_user/current_meeting.php?id_meeting=154&type_order=&sql_type=docume nt_number [25] VCEG & JCT documents available from http://wftp3.itu.int/av-arch in the video-site and jvt-site folders [26] HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.html

[27]R. Li et al, ”A new three-step search algorithm for block motion estimation”, IEEE Transactions on Circuits and Systems for Video Technology, vol.4, pp.438-442, Aug.1994. [28]X. Li et al, “Rate-Complexity-Distortion evaluation for hybrid video coding”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 21, pp. 957 - 970, July 2011. [29] P. Helle et al, “Block merging for quadtree-based partitioning in HEVC”, IEEE Trans. CSVT, vol. 22, pp.1720-1731, Dec. 2012. [30] Information on developments in HEVC NGVC- Next generation video coding http://bisqwit.iki.fi/story/howto/openmp/ [31]M. Wien, “High Efficiency Video Coding: Coding Tools and Specification”, Springer, 2014. [32]I.E. Richardson, “Coding video: A practical guide to HEVC and beyond”, Wiley, 11 April 2016. [33] Q.Jiang and J.Jeong,. "Fast Intra Coding Based on Reference Samples Similarity in HEVC”, International Conference on Signal-Image Technology & Internet-Based Systems,September 2015 [34] Website for downloading test sequence for research purposes http://media.xiph.org/video/derf/ [34] Video test sequences - http://forum.doom9.org/archive/index.php/t-135034.html [35]Tutorials: Tut1. N. Ling, “High efficiency video coding and its 3D extension: A research perspective,” Keynote Speech, ICIEA, Singapore, July 2012. Tut2. X. Wang et al, “Paralleling variable block size motion estimation of HEVC on CPU plus GPU platform”, IEEE ICME workshop, 2013. Tut3. H.R. Tohidpour, M.T. Pourazad and P. Nasiopoulos, “Content adaptive complexity reduction scheme for quality/fidelity scalable HEVC”, IEEE ICASSP 2013, pp. 1744-1748, June 2013. Tut4. M. Wien, “HEVC – coding tools and specifications”, Tutorial, IEEE ICME, San Jose, CA, July 2013.

Tut5. D. Grois, B. Bross and D. Marpe, “HEVC/H Tut5. D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial) Monday 29 June 2015 11.30am-3:00pm , IEEE ICME 2015, Torino, Italy, 29 June – 3 July, 2015.  Tut6. D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial), IEEE ICCE , Berlin, Germany, 6 – 9 Sept. 2015. Tut7. D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial) Sunday 27 Sept 2015, 9:00 am to 12:30 pm), IEEE ICIP, Quebec City, Canada, 27 – 30 Sept. 2015.  The tutorial below is for personal use only. Password: a2FazmgNK https://datacloud.hhi.fraunhofer.de/owncloud/public.php?service=files&t=8edc97d26d46d4458a9c1a17964bf881  Tut8. Please find the links to YouTube videos on the tutorial - HEVC/H.265 Video Coding Standard including the Range Extensions Scalable Extensions and Multiview Extensions below: https://www.youtube.com/watch?v=TLNkK5C1KN8 

Thank You!