1. E ARLY TERMINATION FOR TZ SEARCH IN HEVC MOTION ESTIMATION PRESENTED BY: Rajath Shivananda (1001096626) 1 EE 5359 Multimedia Processing Individual Project

2. List of Acronyms and Abbreviations AVC- Advanced Video Coding. AMVP- Advance motion vector prediction. AP- Above Predictor. ARP- Above Right Prdictor. B-frame- Bi-predictive frame. BMA - Block Matching Algorithm. CABAC- Context Adaptive Binary Arithmetic Coding. CTB- Coding Tree Block. CTU- Coding Tree Unit. CU- Coding Unit. CB- Coding Block DCT- Discrete Cosine Transform. HDTV- High Definition Television. HEVC- High Efficiency Video Coding. HM- HEVC Test Model. I-frame- Intra-coded frame. ICASSP- International Conference on Acoustics, Speech and Signal Processing. JCT- Joint Collaborative Team. JCT-VC- Joint Collaborative Team on Video Coding. JM- H.264 Test Model.

3. JPEG- Joint Photographic Experts Group. KBPS - Kilo Bits Per Second. LCU - Large Coding Unit. LDSP- Large Diamond Search Pattern LP – Left Preditor. MV- Motion Vector. MP - Median Predictor. MC - Motion Compensation. ME - Motion Estimation. MPEG - Motion Picture Experts Group. P-frame: Predicted frame. PC- Prediction Chunking. PU- Prediction Unit. PB- Prediction Block. PSNR- Peak Signal to Noise Ratio. SAD- Sum of absolute Difference. SCU- Small coding unit. SSD -Sum of Squared Differences. QP: Quantization Parameter.

4. RD: Rate Distortion TB: Transform Block. TU: Transform Unit. TZSearch: Test Zone Search

5. Abstract [14] The TZSearch algorithm was adopted in the high efficiency video coding reference software HM as a fast Motion Estimation (ME) algorithm for its excellent performance in reducing ME time and maintaining a comparable Rate Distortion (RD) performance. However, the multiple initial search point decision and the hybrid block matching search contribute a relatively high computational complexity to TZSearch. Based on the statistical analysis of the probability of median predictor to be selected as the final best point in the large Coding Units (CUs) (64x64, 32x32) and small CUs (16x16, 8x8) as well as the center-biased characteristic of the final best search point in ME process two early terminations for TZSearch are proposed. Experimental results shows that 38.96% encoding time is saved, while the RD performance degradation is quite acceptable [16].

6. Objective of the project In this project, TZSearch is used as the best block matching algorithm compared to full search algorithm. Proposed algorithm is followed to terminate TZSearch algorithm to reduce the computational time by 38.96 % [16]. Median predictors are used as the best initial search point which is about 67.41% (Average). Further Experiment is conducted using 3 configuration profiles which are random access, low delay and custom configuration profile. The results for these different configuration profiles are compared and the best configuration profile is selected. Lastly, different video sequences are used and their PSNR and Bitrate are compared.

7. Basic Concepts of Video Coding Color Spaces : RGB color space – Each pixel is represented by three numbers indicating the relative proportions of red, green and blue colors YCrCb color space – Y is the luminance component, a monochrome version of color image. Y is a weighted average of R, G and B: Y = kr R + kg G + kb B where k are the weighting factors. 7

8. The popular patterns of sampling [4] are: 4:4:4 – The three components Y: Cr: Cb have the same resolution, which is for every 4 luminance samples there are 4 Cr and 4 Cb samples. 4:2:2 – For every 4 luminance samples in the horizontal direction, there are 2 Cr and 2 Cb samples. This representation is used for high quality video color reproduction. 4:2:0 – The Cr and Cb each have half the horizontal and vertical resolution of Y. This is popularly used in applications such as video conferencing, digital television and DVD storage. 8

9. 9 Figure 1. 4:2:0 sub-sampling pattern [4]. Figure 2. 4:2:2 and 4:4:4 sub-sampling and sampling patterns [4].

10. H.265 / High Efficiency Video Coding High Efficiency Video Coding (HEVC) [5] is an international standard for video compression developed by a working group of ISO/IEC MPEG (Moving Picture Experts Group) and ITU-T( International Telecommunication Union ) VCEG (Video Coding Experts Group). The main goal of HEVC standard is to significantly improve compression performance compared to existing standards (such as H.264/Advanced Video Coding [6]) in the range of 50% bit rate reduction at similar visual quality[7]. The macroblocks and blocks in H.264 are replaced by CTU,CU,TU and PU in H.265/HEVC. 10

11. Figure 3. Block Diagram of HEVC CODEC [7]. 11

12. HEVC Encoder and Decoder Figure 4. Block Diagram of the HEVC Encoder [2]. 12

13. Figure 5. Block Diagram of the HEVC Decoder [8]. 13

14. Each picture is split into block-shaped regions, with the exact block partitioning being conveyed to the decoder. The first picture of a video sequence is coded using only intra-picture prediction. The encoder and decoder generate identical inter-picture prediction signals by applying motion compensation (MC) using the MV and mode decision data, which are transmitted as side information. The transform coefficients are then scaled, quantized, entropy coded, and transmitted together with the prediction information. The quantized transform coefficients are constructed by inverse scaling and are then inverse transformed to duplicate the decoded approximation of the residual signal. The residual is then added to the prediction, and the result of that addition may then be fed into one or two loop filters to smooth out artifacts induced by block-wise processing and quantization. The duplicate of the output of the decoder is stored in a decoded picture buffer to be used for the prediction of subsequent pictures. 14

15. Coding tree units and coding tree block (CTB) structure: Figure 6. 64*64 CTBs split into CBs [9] 15

16. Coding units (CUs) and coding blocks (CBs): One Luma CB and ordinarily two Chroma CBs, together with associated syntax, form a coding unit (CU) as shown in Figure.7. A CTB may contain only one CU or may be split to form multiple CUs, and each CU has an associated partitioning into prediction units (PUs) and a tree of transform units (TUs). Figure 7. CU’s split into CB’s [9] 16

17. Prediction units and prediction blocks (PBs) : Figure 8. Partitioning of Prediction Blocks from Coding Blocks [9] 17

18. TUs and transform blocks : Figure 9. Partitioning of Transform Blocks from Coding Blocks [9] 18

19. INTER PREDICTION[3] It takes advantage of the temporal redundancies that exist among successive frames and derives a motion- compensated prediction (MCP) for a block of image samples. Inter Prediction comprises of Motion Estimation and Motion Compensation. 19 Figure 10. Prediction from reference blocks [48].

20. 20 MOTION ESTIMATION Figure 11. Motion Vectors [47]

21. BLOCK DIAGRAM OF HEVC INTER-PICTURE PREDICTION 21 Grey parts represent the bi-prediction path Figure 12. HEVC Inter – picture prediction [48]

22. INTER-PICTURE PREDICTION BLOCK MERGING 22 The HEVC quad-tree partitioning has a disadvantage of over-segmenting the image. This leads to redundant signaling and ineffective borders. This drawback can be overcome using Block merging technique. Figure 13. Shows the image of a pendulum, using quad-tree partitioning and block merging respectively [48].

23. FRACTIONAL SAMPLE INTERPOLATION Interpolation is applied on the reference pictures to derive the prediction signal when the corresponding motion vector has fractional sample accuracy. HEVC supports motion vectors with quarter-pixel accuracy for the luma component and one-eighth pixel accuracy for chroma components. 23 Figure 14. Interpolation[45]

24. BLOCK MATCHING ALGORITHM[12] Divide the current frame into blocks. Find one displacement vector for each block. Within a search range, find a “best match” that minimizes an error measure. Intelligent search strategies can reduce computation 24 Figure 15. Block matching algorithm [44].

25. BLOCK MATCHING ALGORITHM [CONT.][12] 25 Figure 16. Block matching algorithm [46]

26. I- Frame Figure 17. I-Frame from BQMall video sequence.

27. B-Frame Figure 18. B-Frame from BQMall video sequence

28. B-Frame (Continued) Figure 19. B-Frame from BQMall video sequence.

29. B-Frame (Continued) Figure 20. B-Frame from BQMall video sequence.

30. Residual from I-Frame Figure 20. Residual of I-Frame of BQMall video sequence

31. Residual from B-Frame Figure 21. Residual of B-Frame of BQMall video sequence

32. BLOCK MATCHING: SEARCH ALGORITHMS 32 Full Search Four Step Search Three Step Search Diamond Search Test Zone Search Exhaustive Search The various BMAs that are used to determine the best matched MV Figure 22. Search Algorithm [18].

33. DIAMOND SEARCH[12] 33 Figure 23. Diamond Search [12].

34. SAD (SUM OF ABSOLUTE DIFFERENCE)[19] SAD architecture is used to calculate the minimum cost between the current block and reference block. Since Motion Estimation(ME) is the most time consuming portion in HEVC encoder it is required to reduce the residual information that is sent to the transform block. The rate-distortion cost J is shown in equation(1), where λ-> Lagrange multiplier R-> Bits required to encode the motion vector difference D-> Distortion function. One of the most widely used distortion function for motion estimation is SAD that can be defined as shown in equation (2), where CB-> Current block pixels RB-> Reference block pixels MxN is the size of the current PU block. Minimum of J is considered. 34

35. Median Predictors Flexible size representation technique contribute the largest proportion of encoding time to HEVC encoder. The search procedure of TZSearch includes two steps, initial search point decision and block matching search, respectively. The first step is to determine the initial search point by using a set of predictors which includes Median Predictor (MP) [10], Left Predictor (LP), Above Predictor (AP), Above-Right Predictor (ARP) and (0, 0). LP, AP and ARP are corresponding to the Motion Vector (MV) of left, top, top right block of the current block, respectively. After the initial search point is determined, the hybrid block matching search, including multiple diamond/square search and raster search, are used to locate the best matching block which is with the minimum RD cost. However, the computational complexity of multiple initial search point decision and hybrid block matching search is still relatively high. If these two processes can be simplified, much more encoding time will be saved. Figure 24. Median Predictors [30]

36. Proposed Early termination Algorithm [1] 36

37. Test Sequences Figure 25. A frame from Mobisode2 video sequence. Resolution – 416x240 Figure 26. A frame from BQMall video sequence. Resolution – 832x480.

38. Test Sequences (Continued) Figure 27. A frame from Johnny video sequence. Resolution – 1280x720. Figure 28. A frame from Park Scene video sequence. Resolution – 1920x1080.

39. Experimental Results Table 1 shows the no of time ( in percentage) median predictors are selected as the best initial search point. It can be seen from table 1 that about 60.33 % on an average median predictors are selected. Table 1. Possibility of selecting Median Predictors as the final best search point, Unit(%). QP Sequence24283236Average BQMall58.7460.9960.6662.7060.77 Johnny61.4464.1567.3570.8465.94 Modisode265.5568.1570.5570.2468.62 ParkScene42.2246.8445.2449.6645.99 Average56.987560.032560.9563.3660.33

40. Experimental Results (Continued) Test Conditions Table 2. Test conditions. Test Conditions Frame Rate30 Total Number of Frames60 GOP8 Search Range64 CU Size / Depth64/4 Inter frames intervals32 QP24, 28, 32, 36

41. Experimental Results (Continued) Comparison of low delay, random access and random access early for different QP values and different video sequences. Table 3. PSNR for low delay, random access and random access early profile. Table 4. PSNR for low delay, random access and random access early profile. Table 5. PSNR for low delay, random access and random access early profile. PSNR in DB for Different QP Video SequenceProfile24283236 Modisode2Low_delay44.087241.822939.807137.9595 Random_access44.668842.496940.556538.7318 Random_access_early44.642942.499740.5335386617 Bitrate in kbps for Different QP Video SequenceProfile24283236 Modisode2Low_delay154.052088.056053.408034.7000 Random_access131.436077.476046.840030.4360 Random_access_early134.720078.748047.588030.3840 Encoding Time in Seconds for Different QP Video SequenceProfile24283236 Modisode2Low_delay213.243154.464179.762138.741 Random_access 123.502115.912104.691113.527 Random_access_early 78.85972.57563.30366.9806 Encoding Time saved (%).36.1475937.3878539.5334841.00029

42. Experimental Results (Continued) Comparison of low delay, random access and random access early for different QP values and different video sequences. Table 6. PSNR for low delay, random access and random access early profile. Table 7. PSNR for low delay, random access and random access early profile. Table 8. PSNR for low delay, random access and random access early profile. PSNR in DB for Different QP Video SequenceProfile24283236 BQMallLow_delay39.329736.963634.641232.4062 Random_access39.475037.301635.099332.9294 Random_access_early39.480437.294735.034032.8468 Bitrate in kbps for Different QP Video SequenceProfile24283236 BQMallLow_delay2049.26001118.9640634.7920371.7720 Random_access1877.14801070.1760631.3680383.0400 Random_access_early1906.90801083.3560632.6000382.9560 Encoding Time in Seconds for Different QP Video SequenceProfile24283236 BQMallLow_delay1115.699852.405767.365703.206 Random_access 628.543538.078475.976424.812 Random_access_early 392.471335.318294.552254.845 Encoding Time saved (%).37.5586137.6822738.1162140.00993

43. Experimental Results (Continued) Comparison of low delay, random access and random access early for different QP values and different video sequences. Table 9. PSNR for low delay, random access and random access early profile. Table 10. PSNR for low delay, random access and random access early profile. Table 11. PSNR for low delay, random access and random access early profile. PSNR in DB for Different QP Video SequenceProfile24283236 JohnnyLow_delay39.329736.963634.641232.4062 Random_access43.429942.187240.753139.0332 Random_access_early43.418942.177040.736239.0224 Bitrate in kbps for Different QP Video SequenceProfile24283236 JohnnyLow_delay2049.26001118.9640634.7920371.7720 Random_access700.2360346.9480201.4520125.3560 Random_access_early713.6160348.2760201.6160125.0160 Encoding Time in Seconds for Different QP Video SequenceProfile24283236 JohnnyLow_delay1115.699852.405767.365703.206 Random_access 900.794856.603813.568872.784 Random_access_early 601.5399563.924487.1408513.1097 Encoding Time saved (%). 33.2211434.167440.1229241.21

44. Experimental Results (Continued) Comparison of low delay, random access and random access early for different QP values and different video sequences. Table 12. PSNR for low delay, random access and random access early profile. Table 13. PSNR for low delay, random access and random access early profile. Table 14. PSNR for low delay, random access and random access early profile. PSNR in DB for Different QP Video SequenceProfile24283236 ParkSceneLow_delay39.456737.303535.291033.4236 Random_access39.719237.774535.856934.0347 Random_access_early39.719637.759535.835534.0067 Bitrate in kbps for Different QP Video SequenceProfile242832 ParkSceneLow_delay5042.2524590.5474266.833 Random_access2871.6842722.5412267.504 Random_access_early1782.3621700.2261360.5024 Encoding Time in Seconds for Different QP Video SequenceProfile24283236 ParkSceneLow_delay5042.2524590.5474266.8333280.501 Random_access 2871.6842722.5412267.5042324.282 Random_access_early 1782.3621700.2261358.5021324.68 Encoding Time saved (%). 37.9332137.5500340.088243.00692

45. Conclusion Hence from the results shown in table 1 it can be said that about 60% of the motion vectors are median predictors. And these median predictors are terminated using TZSearch which saves encoding time by 38%. The results can be seen from table 3 through table 14.

46. References [1] HEVC overview http://www.apsipa2013.org/wpcontent/uploads/2013/09/Tutorial_8_NextGenerationVideoCoding_Part_2.pdf [2] D. Marpe et al, “The H.264/MPEG4 advanced video coding standard and its applications”, IEEE Communications Magazine, Vol. 44, pp. 134-143, Aug. 2006. [3] B. Bross et al, “High Efficiency Video Coding (HEVC) Text Specification Draft 10”, Document JCTVC-L1003, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC),Mar.2013. [4] I.E.G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002 [5] HEVC white paper - Ateme: http://www.ateme.com/an-introduction-to-uhdtv-and-hevc [6] G.J. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, Vol. 7, No. 6, pp. 1001-1016, Dec. 2013. [7] HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.html [8] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T / ISO-IEC Document: JCTVC J0292r1, July 2012. [9] U.S.M. Dayananda, “Study and Performance comparison of HEVC and H.264 video codecs” Final project report, EE Dept., UTA, Arlington, TX, Dec. 2011 available on http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/index_tem.html [10] HM Software Manual - https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ [11] Visual studio: http://www.dreamspark.comhttp://www.dreamspark.com [12] Tortoise SVN: http://tortoisesvn.net/downloads.htmlhttp://tortoisesvn.net/downloads.html [13] Multimedia processing course website: http://www.uta.edu/faculty/krrao/dip/http://www.uta.edu/faculty/krrao/dip/ [14] C. E. Rhee et al, “A Survey of Fast Mode Decision Algorithms for Inter-Prediction and Their Applications to High Efficiency Video Coding”, IEEE Transactions on Consumer Electronics, vol 58, no. 4, pp 1375-1383, Dec. 2012. [15] R. Li, B. Zeng, and M.L. Lio, “A new three-step search algorithm for block motion estimation” IEEE Trans. Circuits and Systems for Video Technology, vol. 4, no. 4, pp. 438-443,August 1994. [16] Z. Pan et al, “Early termination for TZSearch in HEVC motion estimation”, IEEE ICASSP 2013, pp. 1389-1392, June 2013.

47. References (Continued) [17] X. Zhang, S. Wang, S. Ma, “Early termination of coding unit splitting for HEVC”, Signal & Information Processing Association Annual Summit and Conference. Page(s):1-4, December 2012. [18] Ahmad asghar, Muhammad atiq, Rai Ammad khan, Nadeem a. khan, “Motion Estimation and Inter Prediction Mode Selection in HEVC”, Recent Researches in Telecommunications, Informatics, Electronics and Signal Processing, page(s): 351 – 357, December 2013. [19] G. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp 1649-1668, December 2012. [20]HEVC tutorial  http://www.apsipa2013.org/wpcontent/uploads/2013/09/Tutorial_8_NextGenerationVideoCoding_Part_2.pdf.http://www.apsipa2013.org/wpcontent/uploads/2013/09/Tutorial_8_NextGenerationVideoCoding_Part_2.pdf [21] V. Sze, M. Budagavi and G.J. Sullivan (editors), “High Efficiency Video Coding (HEVC) Algorithms and Architectures” Springer, 2014. [22] L.C.Manikandan et al, “A new survey on Block Matching Algorithms in Video Coding” in International Journal of Engineering Research, vol. 3, pp.121-125, February 2014. [23] M. Jakubowski and G. Pastuszak, “Block-based motion estimation algorithms-a survey,” Journal of Opto-Electronics Review, vol. 21, pp 86-102, March 2013. [24] Maria Santamaria and Maria Trujilo,”A comparision of block-matching motion estimation algorithms” on October 4th 2012. [25] L. Xufeng, et al,”Fast motion estimation for HEVC.” 2014 IEEE International Symposium on Broadband Multimedia and Broadcasting (BMSB), IEEE, December 2014. [26] N. Purnachand, et al, “Fast motion estimation algorithm for HEVC,” in Consumer Electronics - Berlin (ICCE-Berlin), 2012 IEEE second International Conference on Consumer Electronics - Berlin (ICCE-Berlin), 2012 IEEE International Conference on. Berlin: pp. 34–37, IEEE, March 2012. [27] X.-l. Tang, et al, “An analysis of TZ search algorithm in JMVC,” in Green Circuits and Systems (ICGCS), 2010 International Conference on, ser. Green Circuits and Systems (ICGCS), 2010 International Conference on. Shanghai: pp. 516–520, IEEE, September 2010. [28] L.N.A. Alves, and A. Navarro, " Fast Motion Estimation Algorithm for HEVC ", Proc IEEE International Conf. on Consumer Electronics - vol.11, pp. 11 - 14, ICCE Berlin, Germany, September 2012 [29] M. Jakubowski and G. Pastuszak, “Block-based motion estimation algorithms-a survey,” Journal of Opto-Electronics Review, vol. 21, pp 86-102, March 2013.

48. References (Continued) [30] Z. Pan, S. Kwong, L. Xu, Y. Zhang, T. Zhao, “Predictive and distribution-oriented fast motion estimation for H.264/AVC” Journal of Real-Time Image Processing, vol. 9, page(s): 597 – 607, December 2014. [31] P. Nalluri et al, “High Speed SAD Architectures for variable block size estimation in HEVC video coding”, IEEE International Conference on Image Processing (ICIP). Page(s): 1233 – 1237, October 2014. [32] M. A. B. Ayed, et al, “TZ Search pattern search improvement for HEVC motion estimation modules,” Advanced Technologies for Signal and Image Processing (ATSIP). Page(s): 95 – 99, March 2014. 2014. [33] Introduction to Motion estimation and Motion compensation---> http://www.cmlab.csie.ntu.edu.tw/cml/dsp/training/coding/motion/me1.html [34] HM Software Manual - https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ [35] Visual studio: http://www.dreamspark.comhttp://www.dreamspark.com [36] Tortoise SVN: http://tortoisesvn.net/downloads.htmlhttp://tortoisesvn.net/downloads.html [37] Tutorials---> N. Ling, “High efficiency video coding and its 3D extension: A research perspective,” Keynote Speech, IEEE Conference on Industrial Electronics and Applications, Singapore, July 2012. [38] Tutorials---> X. Wang et al, “Paralleling variable block size motion estimation of HEVC on CPU plus GPU platform”, IEEE International Conference on Multimedia and Expo workshop, 2013. [39] Tutorials---> H.R. Tohidpour, et al, “Content adaptive complexity reduction scheme for quality/fidelity scalable HEVC”, IEEE International Conference on Image Processing, pp. 1744-1748, June 2013. [40] HEVC tutorial 2014 ISCAS ---> http://www.rle.mit.edu/eems/wp-content/uploads/2014/06/H.265-HEVC-Tutorial-2014-ISCAS.pdf [41] Video Lecture on Digital Voice and Picture Communication by Prof.S. Sengupta, Department of Electronics and Electrical Communication Engineering IIT Kharagpur -> https://www.youtube.com/watch?v=Tm4C2ZFd3zE.https://www.youtube.com/watch?v=Tm4C2ZFd3zE [42] Lecture on video coding standards ->http://nptel.iitm.ac.inhttp://nptel.iitm.ac.in [43] YUV format figures -> http://www.dvxuser.com/V6/showthread.php?336009-Request-4K-10-bit-4-2-2-internal-for-the-DVX200!/page6.http://www.dvxuser.com/V6/showthread.php?336009-Request-4K-10-bit-4-2-2-internal-for-the-DVX200!/page6 [44] Slideshare---> http://www.slideshare.net/mmv-lab-univalle/slides7-ccc-v8

49. References (Continued) [45] Video Compression Image---> http://s1323.photobucket.com/user/UMG-Mark/media/Product%20Support/Funnels_zps553503b7.png.html [46] Motion Estimation for Video Coding---> http://web.stanford.edu/class/ee398a/handouts/lectures/EE398a_MotionEstimation_2012.pdf [47] Different block matching algorithms---> https://en.wikipedia.org/wiki/Block-matching_algorithmhttps://en.wikipedia.org/wiki/Block-matching_algorithm [48] V. Sze, M. Budagavi and G.J. Sullivan (editors) “High efficiency video coding (HEVC): algorithms and architectures”, Springer, 2014.