Srikanth Vasireddy Multimedia Processing Lab,UTA1

 Growing demand for Video  Need for Compression  Key Steps in Video Coding  Overview of HEVC  Features of Moving pictures  Block Matching  Motion Estimation Algorithms  Project Proposal  Software Tools  Acronyms  References Multimedia Processing Lab,UTA2

3 Increase in Applications (86% by 2016)[1] Need Higher Coding efficiency HD and Ultra HD broadcast Need Higher throughput Increase in Mobile data traffic Need Low Power Video is must on all the portable devices

A video is nothing but a sequence of images. Attributes: Figure 1 represents the attributes of the video. -Height - Width - Frame Rate - Pixel Values Multimedia Processing Lab,UTA4 Fig.1 : Attributes of Video [1]

Multimedia Processing Lab,UTA5 Higher bit rates required for uncompressed Video.( HD video, Blu-Ray DVD,TV Broadcast) Compression is achieved by removing the redundant information from video sequence by maintaining an ‘acceptable’ level of video quality. Information TypeCompression Tool Spatial RedundancyIntra prediction Perceptual RedundancyQuantization Statistical RedundancyEntropy Coding Temporal RedundancyInter prediction Fig.2 : Need for Compression[1]

 Intra Prediction  Inter Prediction (Motion Vectors are taken into account)  Transform & Quantization (many pixels to few coefficients)  Entropy Coding Multimedia Processing Lab,UTA6 Fig.3 : Intra Prediction & Inter Prediction[1]

 New standard for Video Compression which has better performance than previous standards.  The HEVC standard is designed to achieve multiple goals, including improved coding efficiency, ease of transport system integration and data loss resilience, as well as implementation ability using parallel processing architectures. (50% bit-rate reduction )[1]  Possible to store or transmit video more efficiently than with earlier technologies such as H.264[2] At the same picture size and quality, an HEVC video sequence should occupy less storage or transmission capacity than H.264 video sequence as shown in Fig.4 At the same storage or transmission BW, HEVC video sequence should be of higher quality and/or resolution than H.264 video sequence as shown in Fig.4 Multimedia Processing Lab,UTA7 Fig.4 : HEVC (vs) H.264 [2]

Multimedia Processing Lab,UTA8 Fig.5: HEVC Working [2]

Multimedia Processing Lab,UTA9 ME has 84% coding complexity and time to encode [1] [5] Fig.6: HEVC Encoder[5]

Multimedia Processing Lab,UTA10 Fig.7: HEVC Decoder[1]

 Moving images contain significant temporal redundancy successive frames are very similar Multimedia Processing Lab,UTA11

 Video coding algorithms usually contains two coding schemes : Intraframe coding : Intraframe coding does not exploit the correlation among adjacent frames; Intraframe coding therefore is similar to the still image coding. Interframe coding :The interframe coding should include motion estimation/compensation process to remove temporal redundancy. Multimedia Processing Lab,UTA12 “The amount of data can be reduced significantly if the previous frame is subtracted from the current frame.”[4] Fig.8: Motion Estimation and Motion Compensation [4]

Multimedia Processing Lab,UTA13 M.J.Jakubowski and G.Pastuszak, “Block-based motion estimation algorithms – a survey,” Opto-Electronic Review, Volume 21, pp ,,March2013.

Multimedia Processing Lab,UTA14  The MPEG and H.26X standards[4] use block-matching technique for motion estimation /compensation.  In the block-matching technique, each current frame is divided into equal-size blocks, called source blocks.  Each source block is associated with a search region in the reference frame.  The objective of block-matching is to find a candidate block in the search region best matched to the source block.  The relative distances between a source block and its candidate blocks are called motion vectors.

Multimedia Processing Lab,UTA15 Video Sequence X: Source block for block-matching B x : Search area associated with X MV: Motion vector current frame Recon. Reference frame Fig.9: Block Matching Scenario [6]

Multimedia Processing Lab,UTA16 Search Area Source block Candidate block Search Area: Motion vector: (u, v)

Multimedia Processing Lab,UTA17 Full Search Algorithm Three Step Search Algorithm Four Step Search Algorithm Diamond Search Algorithm Hexagonal Search Algorithm

If p=7, then there are (2p+1)  (2p+1)=225 candidate blocks. u v Search Area Candidate Block Full Search Algorithm Fig.10 : Full Search Scenario [6][11]

Multimedia Processing Lab,UTA19 In order to get the best match block in the reference frame, it is necessary to compare the current block with all the candidate blocks of the reference frames. Full search motion estimation calculates the sum absolute difference (SAD) value at each possible location in the search window. Full search computes the all candidate blocks intensively for the large search window.

Multimedia Processing Lab,UTA20 The first step involves block-matching based on 4-pel resolution at the nine location.(step size p).Now they check for minimum cost distance and shift centre to the new point of minimum. The second step involves block- matching based on 2-pel resolution around the location determined by the first step.(step size is p/2) The third step repeats the process in the second step (but with resolution 1-pel). 3SS Algorithm Fig.11: 3 Step Search Scenario [6] [11] The position with minimum cost will give us the motion vector and also position of best match.

Multimedia Processing Lab,UTA21 4SS algorithm utilizes a center-biased search pattern with nine checking points on a 5 x 5 window in the instead of a 9 x 9 window in the 3SS. This algorithm helps in reducing the number of search points compared to the 3SS and hence is more robust. Block distortion method (BDM) point is used 4SS Algorithm Fig.12: 4Step Search Scenario [6] [11]

Multimedia Processing Lab,UTA22 Diamond Search Algorithm The DS algorithm employs two search patterns. Large diamond search pattern(LDSP) comprises nine checking points from which eight points surround the center one to compose a diamond shape. Small diamond search pattern (SDSP) consisting of five checking points forms a small diamond shape. LDSP is repeatedly used until the minimum block distortion (MBD) occurs at the center point. Fig.13 : Diamond Search Scenario for ME [7] [11]

Multimedia Processing Lab,UTA23 Hexagonal Search Algorithm Fig.14:Hexagonal Search Scenario for ME [7][11] In block motion estimation, a search pattern with a different shape or size has a very important impact on search speed and distortion performance. HEXBS algorithm can find a same motion vector with fewer search points than the DS algorithm. (Calculate the minimum cost at 6 corner points of Hexagon) Generally speaking, the larger the motion vector, the more search points the HEXBS algorithm can save.

 This project aims at understanding the various block matching motion estimation algorithms for HEVC and analyze how fast the best match is selected and implement some of these algorithms in HM 16.0 software.  Also planning to implement some of the block based motion estimation algorithms in MATLAB. Multimedia Processing Lab,UTA24

 Building HM 16.0 Software[13]  Understand the Motion Estimation Algorithms[7] and how to do fast search for finding the best match.(Diamond and Hexagonal Search)  Implementing the code in HM16.0 and obtaining the results  Implement some algorithms in MATLAB  Optimize the algorithms if time permits Multimedia Processing Lab,UTA25

 The simulation will be conducted using HM Software 16.0 [13] with different video sequences [14], search range, block sizes.  PSNR (dB), Rate distortion RD-plots [16] and BD (Bjontegaard Delta) [16] results will be calculated for different algorithms using various search patterns. Also computation time [28] for different algorithms will be calculated.  MATLAB will be used to implement some algorithms and PSNR variation between different algorithms will be plotted. Multimedia Processing Lab,UTA26

Multimedia Processing Lab,UTA27 BBME : Block Based Motion Estimation BD-BR: Bjontegaard Delta Bitrate. BD-PSNR: Bjontegaard Delta Peak Signal to Noise Ratio. CABAC: Context Adaptive Binary Arithmetic Coding. CTB: Coding Tree Block. CTU: Coding Tree Unit. CU: Coding Unit. DBF: De-blocking Filter. DCT: Discrete Cosine Transform. fps: frames per second. HEVC: High Efficiency Video Coding. HM: HEVC Test Model. ISO: International Organization for Standardization. ITU-T: International Telecommunication Union- Telecommunication Standardization Sector. JCT-VC: Joint Collaborative Team on Video Coding. MAD: Mean Absolute Difference MC: Motion Compensation. ME: Motion Estimation. MPEG: Moving Picture Experts Group. MSE: Mean Square Error. PB: Prediction Block. PSNR: Peak Signal to Noise Ratio. QP: Quantization Parameter SAO: Sample Adaptive Offset. TB: Transform Block. TU: Transform Unit. VCEG: Video Coding Experts Group.

[1] V. Sze and M. Budagavi, “Design and Implementation of Next Generation Video Coding Systems (H.265/HEVC Tutorial)”, IEEE International Symposium on Circuits and Systems (ISCAS), Melbourne, Australia, June 2014, available on [2] HEVC tutorials [3] G.J. Sullivan; J. Ohm; Woo-Jin Han and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Trans. on Circuits and Systems for Video Technology, Volume: 22, Issue: 12, pp , Dec Overview of the High Efficiency Video Coding (HEVC) Standard [4] Ian Richardson “Video Codec Design : Developing Image and Video compression systems”,Wiley,2002. [5] G. J. Sullivan et al “Standardized Extensions of High Efficiency Video Coding (HEVC).”IEEE Journal of selected topics in Signal Processing” vol. 7, pp , Dec [6] L.C.Manikandan et.al “A new survey on Block Matching Algorithms in Video Coding” in International Journal of Engineering Research,Volume 3,pp ,Feb [7] ] L.N.A. Alves, and A. Navarro, " Fast Motion Estimation Algorithm for HEVC ", Proc IEEE International Conf. on Consumer Electronics -ICCE Berlin, Germany, vol.11, pp , Sep., 2012 [8] F. Bossen, et al, “HEVC complexity and implementation analysis”, IEEE Trans. on Circuits and Systems for Video Technology, Volume: 22, Issue: 12, pp , Dec [9] J. Ohm, et al, “Comparison of the Coding Efficiency of Video Coding Standards –Including High Efficiency Video Coding (HEVC)”, IEEE Trans. on Circuits and Systems for Video Technology, volume: 22, Issue: 12, pp , Dec [10] K. Kim, et al, “Block partitioning structure in the HEVC standard,” IEEE Trans. on circuits and systems for video technology, vol. 22, pp , Dec [11] M. Jakubowski and G. Pastuszak, “Block-based motion estimation algorithms-a survey,” Journal of Opto-Electronics Review, vol. 21, pp , Mar [12] A. Abdelazim, W. Masri and B. Noaman "Motion estimation optimization tools for the emerging high efficiency video coding (HEVC)", SPIE vol. 9029, Visual Information Processing and Communication V, , Feb. 17, 2014, doi: / Multimedia Processing Lab,UTA28

[13] Software repository for HEVC - [14] Video test sequences - or [15] HM Software Manual - [16] G. Bjontegaard, "Calculation of average PSNR difference between RD curves", VCEG-M33,ITU-T SG 16/Q 6,Austin, TX, April [17] Multimedia Processing Lab at UTA: Analysis of Motion Estimation (ME) Algorithms. By Tuan Phan Minh Ho (Spring 2014) Comparative study of Motion Estimation (ME) Algorithms by Khyati Mistry (Spring 2008) [18] has info on developments in HEVC NGVC – Next generation video coding. [19] Detailed Overview of HEVC/H.265 by Shevach Riabtsev : [20] W. Hong, “Coherent Block-Based Motion Estimation for Motion-Compensated Frame Rate Up-Conversion", IEEE International Conference on Consumer Electronics, pp , Jan [21] L.N.A. Alves and A. Navarro, " Fast Motion Estimation Algorithm for HEVC ", Proc. IEEE International Conf. on Consumer Electronics - ICCE Berlin, Germany, vol.11, pp , Sep., [22] A. Abdelazim, W. Masri and B. Noaman "Motion estimation optimization tools for the emerging high efficiency video coding (HEVC)", SPIE vol. 9029, Visual Information Processing and Communication V, , Feb. 17, [23] Video test sequences - or [24] M. Wien, “High efficiency video coding: Tools and specification”, Springer, [25] I.E. Richardson, “Coding video: A practical guide to HEVC and beyond”, Wiley, 11 May 2015 [26] V.Sze,M.Budagavi and G.J.Sullivan “ High Efficiency Video Coding(HEVC) –Algorithms and Architectures”,Springer,2014. [27 X. Li et al, “Rate-complexity-distortion evaluation for hybrid video coding”, IEEE International Conference on Multimedia and Expo (ICME), pp , July [28] G. Correa et al, “Performance and computational complexity assessment of high efficiency video encoders”, IEEE Trans. on Circuits and Systems for Video Technology, Vol.22, pp , Dec Multimedia Processing Lab,UTA29