PERFORMANCE COMAPRISON OF HEVC,H.264 and VP9 A PROJECT PROPOSAL UNDER THE GUIDANCE OF DR. K. R. RAO COURSE: EE5359 - MULTIMEDIA PROCESSING, SPRING 2015 By DEEPIKA SREENIVASULU PAGALA deepika.pagala@mavs.uta.edu 1001112646

OBJECTIVE: The objective of this project is to study video coding standards HEVC [1][34][35][36], H.264 [2] [34] and VP9 [3][4] and understand various techniques in video coding such as prediction, transform, quantization and coding. A performance comparison of these video codecs based on various metrics such as computational time, PSNR [25], SSIM [5][20][31], BD-Bitrate [6] and BD-PSNR [7] will be carried out. The HM 16.0 [26][33], JM 18.6 [27][32] and VPX encoder [28] from The WebM Project test models for HEVC, H.264 and VP9 respectively will be used for this purpose.

EVOLUTION OF VIDEO CODING STANDARDS [8]:

GROWING DEMAND FOR VIDEO [30]: Video exceeds half of internet traffic and will grow to 86 percent by 2016 [30]. Increase in applications, content, fidelity, etc. -Need higher coding efficiency! [30]. Ultra-HD 4K broadcast expected for Japan in 2014. London Olympics Opening and Closing Ceremonies shot in Ultra-HD 8K. - Need higher throughput! [30]. 25x increase in mobile data traffic over next five years. Video is a “must have” on portable devices. - Need lower power! [30].

Fundamental Concepts in Video Coding [10]: Color Spaces RGB color space – Each pixel is represented by three numbers indicating the relative proportions of red, green and blue colors. YCrCb color space – Y is the luminance component and Cr ,Cb - two color difference or chrominance components. As the human visual system is less sensitive to color than the luminance component, YCrCb has advantages over RGB space.

H.264/MPEG-4 [2][9]: H.264/Advanced Video Coding (AVC) is video coding standard of the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group [2]. Most popular video standard- 80% of video is encoded on internet with H.264/AVC [9]. Developed for multimedia applications [9]. Adopted advanced coding techniques such as multiple-reference frame prediction, and context-based adaptive binary arithmetic coding (CABAC).

ENCODER IN H.264 [11]:

DECODER IN H.264 [11]:

FEATURES OF H.264: Prediction [12]: Fig 5: Inter prediction in H.264 [12] Fig 4: Intra prediction in H.264 [12]

FEATURES OF H.264(cotd..): Transform and Quantization [13]: A block of residual samples is transformed using a 4x4 or 8x8 integer transform, an approximate form of the Discrete Cosine Transform (DCT) . The transform outputs a set of coefficients, each of which is a weighting value for a standard basis pattern. The output of the transform, a block of transform coefficients, is quantized, i.e. each coefficient is divided by an integer value . Entropy Coding [15]: H.264 uses CABAC(Context Adaptive Binary Arithmetic Coding ) or CAVLC(Context Adaptive Variable Length Coding) for entropy coding[14].

HEVC/H.265 [1]: High Efficiency Video Coding (HEVC) is an international standard for video compression developed by a working group of ISO/IEC MPEG (Moving Picture Experts Group) and ITU-T VCEG (Video Coding Experts Group). High Efficiency Video Coding (HEVC) is the latest Video Coding format . It challenges the state-of-the-art H.264/AVC Video Coding standard which is in current use in the industry by being able to reduce the bit rate by 50% and retaining the same video quality [1].

Fig 6 : Block Diagram of HEVC Encoder [17] Encoder in H.265 [19][20]: Fig 6 : Block Diagram of HEVC Encoder [17]

Fig 7 : Block Diagram of HEVC Decoder [21] DECODER IN H.265 [19][21]: Fig 7 : Block Diagram of HEVC Decoder [21]

FEATURES OF HEVC: Partitioning [19]: Fig 8: Picture, Slice, Coding Tree Unit (CTU), Coding Unit (CU) [19]

FEATURES OF HEVC(cotd..): Prediction [1] : - Intra prediction: Each CU is predicted from neighboring image data in the same picture, using DC prediction, planar prediction or directional prediction. Inter Prediction :Each PU is predicted from image data in one or two reference pictures, using motion compensated prediction. Transform and Quantization [19]: Any residual data remaining after prediction, is transformed using a block transform based on the Discrete Cosine Transform (DCT). The transformed data is quantized. One or more block transforms of size 32x32, 16x16, 8x8 and 4x4 are applied to residual data in each CU. Entropy Coding: - HEVC uses Context Adaptive Binary Arithmetic Coding (CABAC) [14] for Entropy coding.

Vp9 [3][4]: VP9 is an open and royalty free video compression standard being developed by Google VP9 is a successor to VP8. One of the goals of VP9 is to reduce the bit rate by 50% compared to VP8 while having the same video quality [22]. VP9 has many design improvements compared to VP8. VP9 supports the use of superblocks of 64x64 pixels.

ENCODER INVP9 [24]: Fig 9: Encoder block diagram for VP9 [24]

DECODER IN VP9 [24]: Fig 10: Decoder block diagram for VP9 [24]

Fig 11: Example partitioning of a 64x64 Super-block [4] [23] FEATURES OF VP9: Prediction block sizes [4] [23]: Fig 11: Example partitioning of a 64x64 Super-block [4] [23]

FEATURES OF VP9(cotd..) Prediction [4]: - Intra Prediction : VP9 supports a set of 10 Intra prediction modes for block sizes ranging from 4x4 up to 32x32 - Inter Prediction : VP9 supports a set of 4 inter prediction modes for block sizes ranging from 4x4 up to 64x64 pixels Transform and Quantization [4]: The residuals after subtraction of predicted pixel values are subjected to transformation and quantization . Transform blocks can be 32x32, 16x16, 8x8 or 4x4 pixels. Entropy coding [4]: - VP9 uses 8-bit arithmetic coding engine from VP8 known as bool-coder

COMPARISON METRICS: Peak Signal to Noise Ratio (PSNR) [25] Structural Similarity Index (SSIM) [5][20][31] BD-Bitrate and BD-PSNR [6] Computational time as a measure of implementation complexity

PROFILES USED FOR COMPARISON: The HM 16.0 [26][33] ,JM 18.6 [27] [32] and VPX encoder [28] from The WebM Project test models for HEVC , H.264 and VP9 respectively will be used for comparison in this project.

Fig.12 akiyo_qcif.yuv(176x144) [38] TEST SEQUENCES : Fig.12 akiyo_qcif.yuv(176x144) [38]  

TEST SEQUENCES (Cotd..): Fig.13 waterfall_cif.yuv(352x288) [38]

TEST SEQUENCES (Cotd..): Fig.14 BasketballDrill_832x480.yuv [39]

TEST SEQUENCES (Cotd..): Fig 16 :Jockey_1920x1080.yuv [29]

TEST SEQUENCES (Cotd..): Fig 17: PeopleOnStreet_2560_1600_30_crop.yuv [29]

List of Acronyms and Abbreviations: ADST: Asymmetric Discrete Sine Transform JCT-VC: Joint Collaborative Team on Video Coding. AVC: Advanced Video Coding. JM: H.264 Test Model. BD-BR: Bjontegaard Delta Bitrate. JPEG: Joint Photographic Experts Group. BD-PSNR: Bjontegaard Delta Peak Signal to Noise Ratio. KTA: Key Technical Areas (H.264 based exploration software of VCEG) CABAC: Context Adaptive Binary Arithmetic Coding. MC: Motion Compensation. CAVLC: Context Adaptive Variable Length Coding. ME: Motion Estimation. CTB: Coding Tree Block. MPEG: Moving Picture Experts Group. CTU: Coding Tree Unit. MSE: Mean Square Error. CU: Coding Unit. NGOV: Next Generation open Video DBF: De-blocking Filter. PB: Prediction Block. DCT: Discrete Cosine Transform. PCS : Picture Coding Symposium DST :Discrete Sine Transform PSNR: Peak Signal to Noise Ratio. DPB :Decoded Picture Buffer PU: Prediction Unit HD: High Definition QP: Quantization Parameter HEVC: High Efficiency Video Coding. RD: Rate Distortion HM: HEVC Test Model. SAO: Sample Adaptive Offset. IEC: International Electro-technical Commission. SSIM: Structural Similarity Index. ISO: International Organization for Standardization. TB: Transform Block. ITU-T: International Telecommunication Union- Telecommunication Standardization Sector. TU: Transform Unit. VCEG: Visual Coding Experts Group. JCT: Joint Collaborative Team.

REFERENCES: [1] G. J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Transactions on circuits and systems for video technology, vol. 22, no.12, pp. 1649 – 1668, Dec 2012. [2] JVT Draft ITU-T recommendation and final draft international standard of joint video specification (ITU-T Rec. H.264-ISO/IEC 14496-10 AVC), March 2003, JVT-G050 available on http://ip.hhi.de/imagecom_G1/assets/pdfs/JVT-G050.pdf [3] D. Grois et al, “Performance Comparison of H.265/ MPEG-HEVC, VP9, and H.264/MPEG-AVC Encoders”, IEEE PCS 2013, pp 394-397, San José, CA, USA, Dec 8-11, 2013 [4] D. Mukherjee et al, “The latest open-source video codec VP9–An overview and preliminary results”, Google Inc., United States [5] Z. Wang et al, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004 [6] G. Bjøntegaard, “Calculation of average PSNR differences between RD-curves”, ITU-T Q.6/SG16 VCEG 13th Meeting, Document VCEG-M33, Austin, USA, Apr. 2001

REFERENCES(Cotd..): [7] X. Li et al, “Rate-complexity-distortion evaluation for hybrid video coding”, IEEE International Conference on Multimedia and Expo (ICME), pp. 685-690, July. 2010 [8] N. Ling, “High efficiency video coding and its 3D extension: A research perspective,” Keynote Speech, ICIEA, pp 2150-2155, Singapore, July 2012 - http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6361087 [9] V. sze , M. Budagavi , " Design and Implementation of Next Generation Video Coding Systems (H.265/HEVC Tutorial) " , IEEE ISCAS Tutorial 2014 , Melbourne , Australia , June.2014 - filehttp://www.rle.mit.edu/eems/wp-content/uploads/2014/06/H.265-HEVC-Tutorial-2014-ISCAS.pdf [10] I. E. G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002 [11] A. Puri et al, “Video coding using the H.264/MPEG-4 AVC compression standard”, Signal Processing: Image Communication, vol. 19, pp. 793-849, Oct. 2004. [12] H.264 tutorial by I.E.G. Richardson: https://www.vcodex.com/h264.html [13] N. Ahmed , T. Natarajan and K. R. Rao, “Discrete Cosine Transform”, IEEE Transactions on Computers, Vol. C-23, pp. 90-93, Jan. 1974.

REFERENCES(Cotd..): [14] D. Marpe, H. Schwarz, and T. Wiegand, “Context-based adaptive binary arithmetic coding in the H.264/AVC video compression standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, pp. 620–636, Jul. 2003. [15] J . Ostermann, et al, " Video coding with H.264/AVC tools, performance , and complexity", IEEE Circuits and Systems Magazine , Vol.4 , pp.7-28, Aug.2004. [16] J. Ohm , et al, "Comparison of the Coding Efficiency of Video Coding Standards - including High Efficiency Video Coding (HEVC) ", IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, Issue: 12 , pp. 1669 -1684 , Dec.2012. [17] G. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, Vol. 7, No. 6, pp. 1001-1016, Dec. 2013. [18] HEVC white paper - http://www.ateme.com/an-introduction-to-uhdtv-and-hevc [19] HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.html [20] W. Malpica and A. Bovik , "Range image quality assessment by structural similarity", IEEE ICASSP 2009, 19-24 Apr. 2009.

REFERENCES(Cotd..): [21] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T / ISO-IEC Document: JCTVC J0292r1, July 2012. [22] "VP-Next Overview and Progress Update" (PDF). WebM Project (Google). Retrieved 2012-12-29. Available on: http://downloads.webmproject.org/ngov2012/pdf/04-ngov-project-update.pdf [23] M. P. Sharabayko et al, "Intra Compression Efficiency in VP9 and HEVC" Applied Mathematical Sciences, Vol. 7, no. 137, pp.6803 – 6824, Hikari Ltd, 2013 [24] J. Padia, “Complexity reduction for VP6 to H.264 transcoder using motion vector reuse,” M.S. Thesis, EE Dept., UTA, Arlington, TX, 2010. Available on: http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/index_tem.html [25] White paper on PSNR-NI - http://www.ni.com/white-paper/13306/en/ [26] Access to HM Reference Software: http://hevc.hhi.fraunhofer.de [27] Access to JM 18.6 Reference Software: http://iphome.hhi.de/suehring/tml/

REFERENCES(Cotd..): [28] Chromium® open-source browser project, VP9 source code, Online: http://git.chromium.org/gitweb/?p=webm/libvpx.git;a=tree;f=vp9;hb=aaf61dfbcab414bfacc3171501be17d191ff8506 [29] http://ultravideo.cs.tut.fi/#testsequences - Video test sequences (4kx2k) [30] Cisco Visual Networking Index - http://www.cisco.com/c/en/us/solutions/service-provider/visual-networking-index-vni/index.html [31] J. Wang et al, "Fractal image coding using SSIM", IEEE 18th International Conference on Image Processing, Brussels, pp.241-244, 11-14 sept. 2011. [32] H.264/AVC Software Reference Manual: http://iphome.hhi.de/suehring/tml/JM%20Reference%20Software%20Manual%20(JVT-AE010).pdf [33] HEVC Software Reference Manual : https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/branches/HM-9.2-dev/doc/software-manual.pdf

REFERENCES(Cotd..): [34] K. R. Rao, D. N. Kim and J. J. Hwang, “VideoCodingStandards:AVSChina,H.264/MPEG-4Part10,HEVC,VP6,DIRACandVC-1”, Springer, 2014. [35] V. Sze , M. Budagavi , G. J.Sullivan "High Efficiency Video Coding (HEVC) : Algorithms and Architectures", Springer, 2014. [36] M. Wien, "High Efficiency Video Coding : Coding Tools and Specification" , Springer , 2014. [37] I. E. Richardson , "Coding Video : A practical guide to HEVC and beyond " , Wiley , 11 May 2015. [38] https://media.xiph.org/video/derf/ - test sequences [39] ftp://ftp.kw.bbc.co.uk/hevc/hm-11.0-anchors/bitstreams/ - test sequences [40] G. Correa , et al , " Fast HEVC Encoding Decisions Using Data Mining " , IEEE Transactions on Circuits and Systems for Video Technology , Vol . 25 , No. 4 , pp. 660 - 673, April 2015. [41] D. K. Kwon and M.Budagavi , " Combined scalable and mutiview extension of High Efficiency Video Coding (HEVC) " , IEEE Picture Coding Symposium , pp. 414 - 417 , Dec . 2013 .