Fast Block Based Motion Estimation Algorithms in HEVC Srikanth Vasireddy 1001101538 Srikanth.Vasireddy@mavs.uta.edu Multimedia Processing Lab,UTA

Contents Growing demand for Video HEVC Encoder and Decoder Features of Moving pictures Block based Motion Estimation Different Motion Estimation Algorithms Test Sequences Simulation Results Work done and In Progress Acronyms References Multimedia Processing Lab,UTA

Growing Demand for Video Increase in Applications Higher efficiency Broadcast Quality Higher throughput Increase in Mobile data traffic Low power Multimedia Processing Lab,UTA

HEVC Encoder [5] Fig .1: HEVC Encoder[5] ME has 84% coding complexity and time to encode [1] [5] Multimedia Processing Lab,UTA

HEVC Decoder [1] Fig .2: HEVC Decoder[1] Multimedia Processing Lab,UTA

Features of Moving Pictures Moving images contain significant temporal redundancy successive frames are very similar Multimedia Processing Lab,UTA

Intra frame and Inter frame coding Video coding algorithms usually contain two coding schemes : Intraframe coding : Intraframe coding does not exploit the correlation among adjacent frames; Intraframe coding therefore is similar to the still image coding. Interframe coding :The interframe coding should include motion estimation/compensation process to remove temporal redundancy. “The amount of data can be reduced significantly if the previous frame is subtracted from the current frame.”[4] Fig.3: Motion Estimation and Motion Compensation [4] Multimedia Processing Lab,UTA

Temporal Redundancy Removal [1] M.J.Jakubowski and G.Pastuszak, “Block-based motion estimation algorithms – a survey ,” Opto-Electronic Review , Volume 21, pp 86-102,,March2013. Multimedia Processing Lab,UTA

Block based Motion Estimation [6][11] The MPEG and H.26X standards[4] use block based technique for motion estimation /compensation. In this technique, each current frame is divided into equal-size blocks, called source blocks. Each source block is associated with a search region in the reference frame. The objective of block-matching is to find a candidate block in the search region best matched to the source block. The relative distances between a source block and its candidate blocks are called motion vectors. Multimedia Processing Lab,UTA

X: Source block for block-matching Bx: Search area associated with X Video Sequence time X: Source block for block-matching Bx: Search area associated with X MV: Motion vector Reference frame current frame Fig.4: Block matching Scenario [6] Multimedia Processing Lab,UTA

Motion Vector and Search Area Source block Motion vector: (u, v) Search Area: Candidate block Multimedia Processing Lab,UTA

Motion Estimation Algorithms [11] Full Search Algorithm Three Step Search Algorithm Four Step Search Algorithm Diamond Search Algorithm Hexagonal Search Algorithm Multimedia Processing Lab,UTA

Full Search Algorithm u v Candidate Block Search Area If p=7, then there are (2p+1)(2p+1)=225 candidate blocks. Fig.5 : Full Search Scenario [6][11]

Computational complexity is of order n^2 for a block size of nxn In order to get the best match block in the reference frame, it is necessary to compare the current block with all the candidate blocks of the reference frames. Full search motion estimation calculates the sum of absolute difference (SAD) value at each possible location in the search window. Full search computes the all candidate blocks intensively for the large search window. Computational complexity is of order n^2 for a block size of nxn Multimedia Processing Lab,UTA

3SS Algorithm The first step involves block-matching based on 4-pel resolution at the nine location.(step size m).Now they check for minimum cost distance and shift center to the new point of minimum. The second step involves block-matching based on 2-pel resolution around the location determined by the first step.(step size is m/2) The third step repeats the process in the second step (but with resolution 1-pel). Fig.6: 3 Step Search Scenario [6] [11] The position with minimum cost will give us the motion vector and also position of the best match. Multimedia Processing Lab,UTA

4SS Algorithm 4SS algorithm utilizes a center-biased search pattern with nine checking points on a 5 x 5 window in the instead of a 9 x 9 window in the 3SS. This algorithm helps in reducing the number of search points compared to the 3SS and hence is more robust. Block distortion method (BDM) point is used Fig.7: 4Step Search Scenario [6] [11] Multimedia Processing Lab,UTA

Diamond Search Algorithm The DS algorithm employs two search patterns. Large diamond search pattern(LDSP) comprises 9 checking points from which eight points surround the center one to compose a diamond shape. Small diamond search pattern (SDSP) consisting of 5 checking points forms a small diamond shape. LDSP is repeatedly used until the minimum block distortion (MBD) occurs at the center point. Fig.8 : Diamond Search Scenario for ME [7] [11] Multimedia Processing Lab,UTA

Hexagonal Search Algorithm In block motion estimation, a search pattern with a different shape or size has a very important impact on search speed and distortion performance. HEXBS algorithm can find a same motion vector with fewer search points than the DS algorithm. (Calculate the minimum cost at 6 corner points of Hexagon) Generally speaking, the larger the motion vector, the more search points the HEXBS algorithm can save. Fig.9:Hexagonal Search Scenario for ME [7][11] Multimedia Processing Lab,UTA

Test Sequences[23] RaceHorses_416x240_30.yuv sequence BQMall_832x480_60.yuv sequence KirstenAndSara_1280x720_60.yuv sequence ParkScene_1920x1080_24.yuv sequence Multimedia Processing Lab,UTA

Simulation Results RaceHorses_416x240_30.yuv, Number of frames encoded = 20 Random Access profile (FAST SEARCH) Random Access profile (FULL SEARCH) QP PSNR in dB Bit rate in kbps Encoding time in seconds 22 39.5858 1504.992 120.810 39.5969 1494.096 1399.795 27 35.9841 769.740 101.093 35.9926 762.0240 1508.792 32 32.7600 391.116 88.600 32.7769 389.296 1306.499 37 30.1072 202.572 71.437 30.1423 202.3080 1231.534 Table 1: Results for RaceHorses_416x240_30.yuv sequence in Random Access Configuration   BQMall_832x480_60.yuv , Number of frames encoded = 20 Random Access profile (FAST SEARCH) Random Access profile (FULL SEARCH) QP PSNR in dB Bit rate kbps Encoding time in seconds Bit rate in kbps 22 40.6653 4538.160 308.840 40.7583 4528.440 4875.343 27 38.2347 2257.968 259.764 38.2446 2252.592 5214.150 32 35.4849 1200.864 244.682 35.4965 1196.352 4837.578 37 32.7343 656.088 224.326 32.7370 655.2240 5248.942 Table 2: Results for BQMall_832x480_60.yuv sequence in Random Access Configuration Multimedia Processing Lab,UTA

Testing Platform: KristenAndSara_1280x720_60.yuv   KristenAndSara_1280x720_60.yuv Number of fames encoded =20 ParkScene_1920x1080_24.yuv Random Access profile (FAST SEARCH) QP PSNR in dB Bit rate in kbps Encoding time in seconds 22 44.3166 2676.456 533.038 40.7117 8421.072 1808.863 27 42.5646 1232.856 570.403 38.3703 3673.3632 1356.223 32 40.4650 667.344 481.128 36.0006 1697.606 1211.177 37 38.0715 381.720 444.487 33.7621 791.712 1107.963 Table 3: Results for KirstenAndSara_1280x720_60.yuv & ParkScene_1920x1080_24.yuv sequences in Random Access Configuration Testing Platform: Processor Intel Core(TM) i5 CPU 4210U 2.40 GHz Number of cores 2 Memory 8GB Operating System 64 bit (x64-based processor),Windows 8.1 Multimedia Processing Lab,UTA

Figure 10: Snapshot of encoder_randomaccess_main.cfg[15] Multimedia Processing Lab,UTA

Fig 11: Bit rate vs QP for all test sequences Fig 12: PSNR vs QP for all test sequences Multimedia Processing Lab,UTA

Fig 13: R-D plot for RaceHorses_416x240_30.yuv sequence Fig 14: R-D plot for BQMall_832x480_60.yuv sequence Multimedia Processing Lab,UTA

Fig 15: R-D plot for KirstenAndSara_1280x720_60.yuv sequence Fig 16: R-D plot for ParkScene_1920x1080_24.yuv sequence Multimedia Processing Lab,UTA

Fig 17: Encoding time comparison for RaceHorses_416x240_30 Fig 17: Encoding time comparison for RaceHorses_416x240_30.yuv sequence Multimedia Processing Lab,UTA

Fig 18: Encoding time comparison for RaceHorses_416x240_30 Fig 18: Encoding time comparison for RaceHorses_416x240_30.yuv sequence Multimedia Processing Lab,UTA

Fig 19: Fast Search Encoding time comparison for all test sequences Multimedia Processing Lab,UTA

Fig 20: Full Search Encoding time comparison for RaceHorses & BQMall Multimedia Processing Lab,UTA

Work done and In Progress Analyzed Full Search,3SS and Diamond Search. Developed functions in MATLAB like To find PSNR w.r.t reference image To compute motion compensated image To find minimum distance among macro blocks To compute Mean Absolute difference Developing functions for Full Search ,3SS and Diamond Search Multimedia Processing Lab,UTA

Acronyms BBME : Block Based Motion Estimation BD-BR: Bjontegaard Delta Bitrate. BD-PSNR: Bjontegaard Delta Peak Signal to Noise Ratio. CABAC: Context Adaptive Binary Arithmetic Coding. CTB: Coding Tree Block. CTU: Coding Tree Unit. CU: Coding Unit. DBF: De-blocking Filter. DCT: Discrete Cosine Transform. fps: frames per second. HEVC: High Efficiency Video Coding. HM: HEVC Test Model. ISO: International Organization for Standardization. ITU-T: International Telecommunication Union- Telecommunication Standardization Sector. JCT-VC: Joint Collaborative Team on Video Coding. MAD: Mean Absolute Difference MC: Motion Compensation. ME: Motion Estimation. MPEG: Moving Picture Experts Group. MSE: Mean Square Error. PB: Prediction Block. PSNR: Peak Signal to Noise Ratio. QP: Quantization Parameter SAO: Sample Adaptive Offset. TB: Transform Block. TU: Transform Unit. VCEG: Visual Coding Experts Group. Multimedia Processing Lab,UTA

REFERENCES [1] V. Sze and M. Budagavi, “Design and Implementation of Next Generation Video Coding Systems (H.265/HEVC Tutorial)”, IEEE International Symposium on Circuits and Systems (ISCAS), Melbourne, Australia, June 2014, available on http://www.rle.mit.edu/eems/publications/tutorials/ [2] HEVC tutorials http://www.vcodex.com/h265.html [3] G.J. Sullivan; J. Ohm; W.J, Han and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Trans. on Circuits and Systems for Video Technology, Volume: 22, Issue: 12, pp. 1649-1668, Dec. 2012. [4] I.E. Richardson “Video Codec Design : Developing Image and Video compression systems”,Wiley,2002. [5] G. J. Sullivan et al “Standardized Extensions of High Efficiency Video Coding (HEVC).”IEEE Journal of selected topics in Signal Processing” vol. 7, pp.1001-1016, Dec. 2013 [6] L.C.Manikandan et al “A new survey on Block Matching Algorithms in Video Coding” in International Journal of Engineering Research ,Volume 3,pp.121-125,Feb 2014. [7] ] L.N.A. Alves, and A. Navarro, " Fast Motion Estimation Algorithm for HEVC ", Proc IEEE International Conf. on Consumer Electronics -ICCE Berlin , Germany , vol.11 , pp. 11 - 14 , Sep. , 2012 [8] F. Bossen, et al, “HEVC complexity and implementation analysis”, IEEE Trans. on Circuits and Systems for Video Technology, Volume: 22, Issue: 12, pp. 1685 - 1696, Dec. 2012. [9] J. Ohm, et al, “Comparison of the Coding Efficiency of Video Coding Standards –Including High Efficiency Video Coding (HEVC)”, IEEE Trans. on Circuits and Systems for Video Technology, volume: 22, Issue: 12, pp. 1669 - 1684, Dec. 2012. [10] K. Kim, et al, “Block partitioning structure in the HEVC standard,” IEEE Trans. on circuits and systems for video technology, vol. 22, pp.1697-1706, Dec. 2012. [11] M. Jakubowski and G. Pastuszak, “Block-based motion estimation algorithms-a survey,” Journal of Opto-Electronics Review, vol. 21, pp 86-102, Mar. 2013. http://link.springer.com/article/10.2478%2Fs11772-013-0071-0#page-1 [12] A. Abdelazim, W. Masri and B. Noaman "Motion estimation optimization tools for the emerging high efficiency video coding (HEVC)", SPIE vol. 9029, Visual Information Processing and Communication V, 902905, Feb. 17, 2014, doi:10.1117/12.2041166 Multimedia Processing Lab,UTA

[13] Software repository for HEVC - https://hevc. hhi. fraunhofer [14] HEVC white paper –Ittiam systems - http://www.Ittiam.com/Downloads/en/documentation.aspx [15] HM Software Manual - https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ [16] G. Bjontegaard, "Calculation of average PSNR difference between RD curves", VCEG-M33,ITU-T SG 16/Q 6,Austin, TX, April 2001. [17] Multimedia Processing Lab at UTA: http://www.uta.edu/faculty/krrao/dip/ Analysis of Motion Estimation (ME) Algorithms. By Tuan Phan Minh Ho (Spring 2014) Comparative study of Motion Estimation (ME) Algorithms by Khyati Mistry (Spring 2008) [18] http://www.h265.net has info on developments in HEVC NGVC – Next generation video coding. [19] Detailed Overview of HEVC/H.265 by Shevach Riabtsev : https://app.box.com/s/rxxxzr1a1lnh7709yvih [20] W. Hong, “Coherent Block-Based Motion Estimation for Motion-Compensated Frame Rate Up-Conversion", IEEE International Conference on Consumer Electronics, pp. 165-166, Jan.2010.  [21] N.Purnachand and L.N. Alves, A. Navarro “Improvements to TZ search motion estimation algorithm for multiview video coding” 19th International Conference on Systems, Signals and Image Processing IWSSIP, pp. 388 -391, 2012. [22] Video test sequences - http://forum.doom9.org/archive/index.php/t-135034.html or http://media.xiph.org/video/derf/ or ftp://ftp.kw.bbc.co.uk/hevc/hm-11.0-anchors/bitstreams/ or http://forum.doom9.org/archive/index.php/t-135034.html [23] M. Wien, “High efficiency video coding: Tools and specification”, Springer, 2015. [24] I.E. Richardson, “Coding video: A practical guide to HEVC and beyond”, Wiley, 11 May 2015 [25] V.Sze ,M.Budagavi and G.J.Sullivan “ High Efficiency Video Coding(HEVC) –Algorithms and Architectures”, Springer, 2014. [26] X. Li et al, “Rate-complexity-distortion evaluation for hybrid video coding”, IEEE International Conference on Multimedia and Expo (ICME), pp. 685-690, July. 2010. [27] G. Correa et al, “Performance and computational complexity assessment of high efficiency video encoders”, IEEE Trans. on Circuits and Systems for Video Technology, Vol.22, pp.1899-1909, Dec.2012. Multimedia Processing Lab,UTA