1. HEVC INTER PREDICTION PRESENTERS: Divya Nityanand (1001112716) Rajath Shivananda (1001096626) Rakhee Barkur (1001096946) 1 EE 5359 Multimedia Processing Group Project Group-2

2. ACRONYMS AMVP- Advanced Motion Vector Prediction AVC – Advanced Video Coding BMA – Block Matching Algorithm CABAC – Context Adaptive Binary Arithmetic Coding CB – Coding Block CTU – Coding Tree Unit CTB – Coding Tree Block CU – Coding Unit HEVC – High Efficiency Video Coding HM – HEVC Test Model IEC – International Electrotechnical Commission ISO – International Standards Organization ITU – International Telecommunication Union 2

3. ACRONYMS CONTD.. JCT-VC - Joint Collaborative Team on Video Coding MC – Motion Compensation ME – Motion Estimation MV – Motion Vector MPEG – Moving Picture Experts Group PB – Prediction Block PU – Prediction Unit SAD- Sum of absolute Difference SSD -Sum Of Squared Differences TSS – Three Step Search TU – Transform Unit TZS- Test Zone Search VCEG – Video Coding Experts Group WP – Weighted Prediction 3

4. TABLE OF CONTENTS Motivation For Video Compression Video Coding Standards – Timeline Features Of HEVC Block Diagram Of HEVC Encoder Inter Prediction Block Diagram Of HEVC Inter-picture Prediction Inter-Picture Prediction Block Merging Fractional Sample Interpolation Weighted Sample Prediction Block Matching Algorithm Block Matching: Search Algorithms Diamond Search SAD (SUM OF ABSOLUTE DIFFERENCE) SSD ( SUM OF SQUARED DIFFERENCE) TZ Algorithm HM Source Code: Prime Functions For TZ BMA References 4

5. MOTIVATION FOR VIDEO COMPRESSION ● Nowadays videos constitute majority of internet traffic(almost 86%). ● A digital video signal requires enormous storage space and transmission bandwidth. ● Video Compression shrinks a file and makes it more practical to store and share. ● Compression works by removing redundant information and effectively summarizes the contents of a file so as to preserve its originality. 5 FIGURE 1[11]

6. VIDEO CODING STANDARDS – TIMELINE 6 HEVC standard is the most recent joint video project of the ITU-T VCEG and the ISO/IEC MPEG standardization organizations, working together in a partnership known as JCT-VC[2]. FIGURE 2[31]

7. FEATURES OF HEVC[8] Flexible frame partitioning. Consists of 33 Directional modes +1 Planar +1 DC prediction mode for better image encoding. Improved functionality of the Interpolation and Deblocking filters. Supports Parallel processing. Aims at doubling the coding efficiency at the same data rate compared to H.264/AVC. Uses the quad tree structure for the partitioning of the CTBs into smaller CUs and PUs for prediction. 7

8. BLOCK DIAGRAM OF HEVC ENCODER 8 FIGURE 3[2]

9. INTER PREDICTION[3] It takes advantage of the temporal redundancies that exist among successive frames and derives a motion- compensated prediction (MCP) for a block of image samples. Inter Prediction comprises of Motion Estimation and Motion Compensation. 9 FIGURE 4[3]

10. 10 MOTION ESTIMATION FIGURE 5[13]

11. BLOCK DIAGRAM OF HEVC INTER-PICTURE PREDICTION 11 Grey parts represent the bi-prediction path FIGURE 6[3]

12. INTER-PICTURE PREDICTION BLOCK MERGING 12 The HEVC quad-tree partitioning has a disadvantage of over-segmenting the image. This leads to redundant signaling and ineffective borders. This drawback can be overcome using Block merging technique. Fig a, b, c shows the image of a pendulum, using quad-tree partitioning and block merging respectively[3]

13. FRACTIONAL SAMPLE INTERPOLATION Interpolation is applied on the reference pictures to derive the prediction signal when the corresponding motion vector has fractional sample accuracy. HEVC supports motion vectors with quarter-pixel accuracy for the luma component and one-eighth pixel accuracy for chroma components. 13 FIGURE 7[11]

14. WEIGHTED SAMPLE PREDICTION[3] 14

15. BLOCK MATCHING ALGORITHM[12] Divide the current frame into blocks. Find one displacement vector for each block. Within a search range, find a “best match” that minimizes an error measure. Intelligent search strategies can reduce computation 15 FIGURE 8[13]

16. BLOCK MATCHING ALGORITHM [CONT.][12] 16 FIGURE 9[12]

17. BLOCK MATCHING: SEARCH ALGORITHMS 17 Full Search Four Step Search Three Step Search Diamond Search Test Zone Search Exhaustive Search The various BMAs that are used to determine the best matched MV FIGURE 10[18]

18. DIAMOND SEARCH[12] 18 FIGURE 11[12]

19. SAD (SUM OF ABSOLUTE DIFFERENCE)[19] SAD architecture is used to calculate the minimum cost between the current block and reference block. Since Motion Estimation(ME) is the most time consuming portion in HEVC encoder it is required to reduce the residual information that is sent to the transform block. The rate-distortion cost J is shown in equation(1), where λ-> Lagrange multiplier R-> Bits required to encode the motion vector difference D-> Distortion function. One of the most widely used distortion function for motion estimation is SAD that can be defined as shown in equation (2), where CB-> Current block pixels RB-> Reference block pixels MxN is the size of the current PU block. Minimum of J is considered. 19

20. SSD ( SUM OF SQUARED DIFFERENCE)[20] SSD between ‘b’ and ‘p’ corresponds to squared Euclidian norm as shown in the equation 1. (1) b represents current block p represents reference block. The measure corresponds to square of the error between b and p. Considering p as an approximation of b, the difference signal can be interpreted as noise that has been added to the original signal block. 20

21. TEST ZONE(TZ) SEARCH ALGORITHM[9] 21 HM Code uses the TZ search algorithm to determine the MVs. Steps are as follows- 1.Establish few search centres and select the one that has least matching error. 1.Determine the Search Range and Search Pattern. 1.If distance between Optimal Point = Search center ; Stop the search Optimal Point > iRaster; perform Raster search 4. Raster refinement performs 8 point diamond or square search with the best distance decreasing according to the exponential of 2 and updating every step of the current starting point location until the best distance is 0. FIGURE 12[9]

22. HM SOURCE CODE: PRIME FUNCTIONS FOR TZ BMA[14] TLibEncoder Source file : TEncSearch.cpp Main Class: TEncSearch Important Functions used : xTZ2PointSearch xTZ8PointSquareSearch xTZ8PointDiamondSearch bUseAdaptiveRaster // bool if dst > iRaster perform Raster search bFirstSearchDiamond // bool 1= Diamond; 0=Square bStarRefinementEnable // bool enable either Star/Raster refinement iRaster=5; // stride length 22

23. REFERENCES 23 [1] 4K Ultra HD TV Broadcast Standards---> http://www.4kshooters.net/2014/07/13/4k-ultra-hd-tv- broadcast-standards-move-forward-with-10bit-hevc-up-to-60fps/http://www.4kshooters.net/2014/07/13/4k-ultra-hd-tv- broadcast-standards-move-forward-with-10bit-hevc-up-to-60fps/ [2] G.J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Trans. CSVT, vol. 22,no.12, pp. 1649 – 1668, Dec 2012. [3] V. Sze, M. Budagavi and G.J. Sullivan (editors) “High efficiency video coding (HEVC): algorithms and architectures”, Springer, 2014. [4] Open source article, “H.264/MPEG-4 AVC,” Wikipedia Foundation, http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC [5] A. Asghar, et al, “Motion Estimation and Inter Prediction Mode Selection in HEVC,” Recent Researches in Telecommunications, Informatics, Electronics and Signal Processing, March 2013. [6] Basics of video coding---> http://lea.hamradio.si/~s51kq/V-BAS.HTM http://lea.hamradio.si/~s51kq/V-BAS.HTM [7] S. Lui et al,“Video Prediction Block Structure and the Emerging High Efficiency Video Coding Standard”, IEEE proceedings on Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, pp. 1-4, 2012. [8] Introduction to HEVC - Slideshare---> http://www.slideshare.net/EnricoMilani/introduction-to-high- efficiency-video-codinghttp://www.slideshare.net/EnricoMilani/introduction-to-high- efficiency-video-coding

24. REFERENCES [CONT.] [9] L. Xufeng, et al, ”Fast motion estimation for HEVC.” Broadband Multimedia and Broadcasting (BMSB), 2014 IEEE International Symposium on, IEEE, 2014. [10] HEVC tutorial 2014 ISCAS ---> http://www.rle.mit.edu/eems/wp-content/uploads/2014/06/H.265-HEVC-Tutorial-2014-ISCAS.pdf [11] Video Compression Image---> http://s1323.photobucket.com/user/UMG- Mark/media/Product%20Support/Funnels_zps553503b7.png.htmlhttp://s1323.photobucket.com/user/UMG- Mark/media/Product%20Support/Funnels_zps553503b7.png.html [12] Motion Estimation for Video Coding---> http://web.stanford.edu/class/ee398a/handouts/lectures/EE398a_MotionEstimation_2012.pdf [13] Slideshare---> http://www.slideshare.net/mmv-lab-univalle/slides7-ccc-v8http://www.slideshare.net/mmv-lab-univalle/slides7-ccc-v8 [14] HM Source Code--- > https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/trunk/HM Source Code--- > https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/trunk/ [15] Introduction to Motion estimation and Motion compensation---> http://www.cmlab.csie.ntu.edu.tw/cml/dsp/training/coding/motion/me1.html [16] M. A. B. Ayed, et al, “TZ Search pattern search improvement for HEVC motion estimation modules,” Advanced Technologies for Signal and Image Processing (ATSIP), 2014 1st International Conference on, March 2014. [17] N. Ling, et al,“Improved and simplified fast motion estimation for JM,” JVT meeting, Poznan, Poland, Tech. Rep.JVT-P021, July 2005. 24

25. REFERENCES [CONT.] [18] Different block matching algorithms---> https://en.wikipedia.org/wiki/Block-matching_algorithmhttps://en.wikipedia.org/wiki/Block-matching_algorithm [19] P. Nalluri et al, “High Speed SAD Architectures for variable block size estimation in HEVC video coding”, IEEE International Conference on Image Processing (ICIP), Oct 2014. [20] M. Wein, “High Efficiency Video Coding: Coding Tools and Specification”, Springer, 2014. [21] HM Software Manual - https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ [22] Visual studio: http://www.dreamspark.comhttp://www.dreamspark.com [23] Tortoise SVN: http://tortoisesvn.net/downloads.htmlhttp://tortoisesvn.net/downloads.html [24] Tutorials---> N. Ling, “High efficiency video coding and its 3D extension: A research perspective,” Keynote Speech, ICIEA, Singapore, July 2012. [25] Tutorials---> X. Wang et al, “Paralleling variable block size motion estimation of HEVC on CPU plus GPU platform”, IEEE ICME workshop, 2013. [26] Tutorials---> H.R. Tohidpour, et al, “Content adaptive complexity reduction scheme for quality/fidelity scalable HEVC”, IEEE ICASSP 2013, pp. 1744-1748, June 2013. [27] K.R. Rao, D.N. Kim and J.J. Hwang, “Video coding standards: AVS China, H.264/MPEG-4 Part10, HEVC, VP6, DIRAC and VC-1”, Springer, 2014. [28] M. Kavanagh, “H.265/HEVC Overview and Comparison with H.264/AVC,” Sold by: Amazon Digital Services, Inc. August, 2015. [29] Test sequences---> http://media.xiph.org/video/derf/http://media.xiph.org/video/derf/ [30] Test sequences---> http://trace.eas.asu.edu/yuv/http://trace.eas.asu.edu/yuv/ [31] http://www.uta.edu/faculty/krrao/dip/ - “Evolution of video coding standard figure”.http://www.uta.edu/faculty/krrao/dip/ 25

26. THANK YOU 26