INTERIM REPORT ON PERFORMANCE COMAPRISON OF HEVC,H.264 and VP9 A PROJECT UNDER THE GUIDANCE OF DR. K. R. RAO COURSE: EE MULTIMEDIA PROCESSING, SPRING 2015 By DEEPIKA SREENIVASULU PAGALA

OBJECTIVE: The objective of this project is to study video coding standards HEVC [1][34][35][36], H.264 [2] [34] and VP9 [3][4] and understand various techniques in video coding such as prediction, transform, quantization and coding. A performance comparison of these video codecs based on various metrics such as computational time, PSNR [25], SSIM [5][20][31], BD-Bitrate [6] and BD- PSNR [7] will be carried out. The HM 16.0 [26][33], JM 18.6 [27][32] and VPX encoder [28] from The WebM Project test models for HEVC, H.264 and VP9 respectively will be used for this purpose.

EVOLUTION OF VIDEO CODING STANDARDS [8]:

GROWING DEMAND FOR VIDEO [30]: Video exceeds half of internet traffic and will grow to 86 percent by 2016 [30]. Increase in applications, content, fidelity, etc. - Need higher coding efficiency! [30]. Ultra-HD 4K broadcast expected for Japan in London Olympics Opening and Closing Ceremonies shot in Ultra-HD 8K. - Need higher throughput! [30]. 25x increase in mobile data traffic over next five years. Video is a “must have” on portable devices. - Need lower power! [30].

FUNDAMENTAL CONCEPTS IN VIDEO CODING [10]: Color Spaces RGB color space – Each pixel is represented by three numbers indicating the relative proportions of red, green and blue colors. YC r C b color space – Y is the luminance component and C r, C b - two color difference or chrominance components. As the human visual system is less sensitive to color than the luminance component, YC r C b has advantages over RGB space.

H.264/MPEG-4 [2][9]: H.264/Advanced Video Coding (AVC) is video coding standard of the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group [2]. Most popular video standard- 80% of video is encoded on internet with H.264/AVC [9]. Developed for multimedia applications [9]. Adopted advanced coding techniques such as multiple-reference frame prediction, and context- based adaptive binary arithmetic coding (CABAC).

ENCODER IN H.264 [11]:

DECODER IN H.264 [11]:

FEATURES OF H.264: Prediction [12]: Fig 4: Intra prediction in H.264 [12] Fig 5: Inter prediction in H.264 [12]

FEATURES OF H.264(cotd..): Transform and Quantization [13]: A block of residual samples is transformed using a 4x4 or 8x8 integer transform, an approximate form of the Discrete Cosine Transform (DCT). The transform outputs a set of coefficients, each of which is a weighting value for a standard basis pattern. The output of the transform, a block of transform coefficients, is quantized, i.e. each coefficient is divided by an integer value. Entropy Coding [15]: H.264 uses CABAC(Context Adaptive Binary Arithmetic Coding ) or CAVLC(Context Adaptive Variable Length Coding) for entropy coding[14].

HEVC/H.265 [1]: High Efficiency Video Coding (HEVC) is an international standard for video compression developed by a working group of ISO/IEC MPEG (Moving Picture Experts Group) and ITU-T VCEG (Visual Coding Experts Group). High Efficiency Video Coding (HEVC) is the latest Video Coding format. It challenges the state-of-the-art H.264/AVC Video Coding standard which is in current use in the industry by being able to reduce the bit rate by 50% and retaining the same video quality [1].

Encoder in H.265 [19][20]: Fig 6 : Block Diagram of HEVC Encoder [17]

DECODER IN H.265 [19][21]: Fig 7 : Block Diagram of HEVC Decoder [21]

FEATURES OF HEVC: Partitioning [19]: Fig 8: Picture, Slice, Coding Tree Unit (CTU), Coding Unit (CU) [19]

FEATURES OF HEVC(cotd..): Prediction [1] : - Intra prediction: Each CU is predicted from neighboring image data in the same picture, using DC prediction, planar prediction or directional prediction. -Inter Prediction :Each PU is predicted from image data in one or two reference pictures, using motion compensated prediction. Transform and Quantization [19]: -Any residual data remaining after prediction, is transformed using a block transform based on the Discrete Cosine Transform (DCT). The transformed data is quantized. One or more block transforms of size 32x32, 16x16, 8x8 and 4x4 are applied to residual data in each CU. Entropy Coding: - HEVC uses Context Adaptive Binary Arithmetic Coding (CABAC) [14] for Entropy coding.

Vp9 [3][4]: VP9 is an open and royalty free video compression standard being developed by Google VP9 is a successor to VP8. One of the goals of VP9 is to reduce the bit rate by 50% compared to VP8 while having the same video quality [22]. VP9 has many design improvements compared to VP8. VP9 supports the use of superblocks of 64x64 pixels.

ENCODER INVP9 [24]: Fig 9: Encoder block diagram for VP9 [24]

DECODER IN VP9 [24]: Fig 10: Decoder block diagram for VP9 [24]

FEATURES OF VP9: Prediction block sizes [4] [23]: Fig 11: Example partitioning of a 64x64 Super-block [4] [23]

FEATURES OF VP9(cotd..) Prediction [4]: - Intra Prediction : VP9 supports a set of 10 Intra prediction modes for block sizes ranging from 4x4 up to 32x32 - Inter Prediction : VP9 supports a set of 4 inter prediction modes for block sizes ranging from 4x4 up to 64x64 pixels Transform and Quantization [4]: -The residuals after subtraction of predicted pixel values are subjected to transformation and quantization. Transform blocks can be 32x32, 16x16, 8x8 or 4x4 pixels. Entropy coding [4]: - VP9 uses 8-bit arithmetic coding engine from VP8 known as bool-coder

COMPARISON METRICS: Peak Signal to Noise Ratio (PSNR) [25] Structural Similarity Index (SSIM) [5][20][31] BD-Bitrate and BD-PSNR [6] Computational time as a measure of implementation complexity

PROFILES USED FOR COMPARISON: The HM 16.2 [26][33],JM 18.6 [27] [32] and VPX encoder [28] from The WebM Project test models for HEVC, H.264 and VP9 respectively will be used for comparison in this project.

TEST SEQUENCES : Fig.12 akiyo_qcif.yuv(176x144) [38]

TEST SEQUENCES (Cotd..): Fig.13 waterfall_cif.yuv(352x288) [38]

TEST SEQUENCES (Cotd..): Fig.14 BasketballDrill_832x480.yuv [39]

TEST SEQUENCES (Cotd..): Fig 15 :Jockey_1920x1080.yuv [29]

CONFIGURATION OF HM 16.2 Main all-intra profile settings : IntraPeriod : 1 # Period of I-Frame ( -1 = only first) GOPSize : 1 # GOP Size (number of B slice = GOPSize-1) QP : 22 # Quantization parameter(0-51) (22, 27, 32 or 37 is used at a time) Command line parameters for using HM 16.2 encoder: TAppEncoder [-h] [-c config.cfg] [--parameter=value] Options: -h Prints parameter usage -c Defines configuration file to use. Multiple configuration files may be used with repeated –c options. --parameter=value Assigns value to a given parameter. Sample command line parameters for HM 16.2 encoder: C:\HEVC\bin\vc10\Win32\Release>TAppEncoder.exe -c encoder_intra_main.cfg - wdt hgt fr 30 -f 20 -i C:\HEVC\bin\vc10\Win32\testsequences >> C:\HEVC\bin\vc10\Win32\final_results\ PeopleOnStreet_2560x1600_30_crop_qp37.txt

CONFIGURATION OF JM 18.6 High all-intra profile settings : FramesToBeEncoded : 20 # Number of frames to be coded FrameRate : 50.0 # Frame Rate per second ( ) ProfileIDC : 100 # Profile IDC (66=baseline, 77=main, 88=extended; FREXT Profiles: 100=High, 110=High 10, 122=High 4:2:2, 244=High 4:4:4, 44=CAVLC 4:4:4 Intra, 118=Multiview High Profile, 128=Stereo High Profile) IntraProfile : 1 # Activate Intra Profile for FRExt (0: false, 1: true) LevelIDC : 40 # Level IDC (e.g. 20 = level 2.0) IntraPeriod : 1 # Period of I-pictures (0=only first) IDRPeriod : 1 # Period of IDR pictures (0=only first) QPISlice : 22 # Quant. param for I Slices (0-51) (22, 27, 32 or 37 is used at a time)

CONFIGURATION OF JM 18.6 (COTD..) Command line parameters for using JM 18.6 encoder: Options: -h Prints parameter usage. -d Use as default file for parameter initializations. If not used then file defaults to “encoder.cfg” in local directory. -f Read for resetting selected encoder parameters. Multiple files could be used that set different parameters. -p Set parameter to. The entry for is case insensitive. Sample command line parameters for JM 18.6 encoder: C:\jm 18.6\JM\bin>lencod.exe –f encoder.cfg -p InputFile=" C:\jm 18.6\JM\bin\ Jockey_1920x1080.yuv" –p SourceWidth=832 -p SourceHeight=480 >> C:\jm 18.6\JM\bin\final_results\ jockey_37.txt

RESULTS :

RESULTS(COTD..)

LIST OF ACRONYMS AND ABBREVIATIONS: ADST: Asymmetric Discrete Sine Transform. AHG: Ad Hoc Group. AVC: Advanced Video Coding. BD-BR: Bjontegaard Delta Bitrate. BD-PSNR: Bjontegaard Delta Peak Signal to Noise Ratio. CABAC: Context Adaptive Binary Arithmetic Coding. CAVLC: Context Adaptive Variable Length Coding. CTB: Coding Tree Block. CTU: Coding Tree Unit. CU: Coding Unit. DBF: De-blocking Filter. DCT: Discrete Cosine Transform. DST :Discrete Sine Transform. DPB :Decoded Picture Buffer. DVD: Digital Video Disk. HD: High Definition. HDR :High Dynamic Range. HEVC: High Efficiency Video Coding. HM: HEVC Test Model. ICME: International Conference on Multimedia and Expo. IEC: International Electro-technical Commission. ISCAS : International Symposium on Circuits and Systems. ISO: International Organization for Standardization. ITU-T: International Telecommunication Union- Telecommunication Standardization Sector. JCT: Joint Collaborative Team. JCT-VC: Joint Collaborative Team on Video Coding. JM: H.264 Test Model. JCT-VC: Joint Collaborative Team on Video Coding. JPEG: Joint Photographic Experts Group. KTA: Key Technical Areas (H.264 based exploration software of VCEG) MC: Motion Compensation. ME: Motion Estimation. MPEG: Moving Picture Experts Group. MSE: Mean Square Error. MVC : Multiview Video Coding.. NGOV: Next Generation open Video PB: Prediction Block. PCS : Picture Coding Symposium PSNR: Peak Signal to Noise Ratio. PU: Prediction Unit QP: Quantization Parameter RD: Rate Distortion SAO: Sample Adaptive Offset. SCC: Screen Content Coding. SSIM: Structural Similarity Index. TB: Transform Block. TU: Transform Unit. VCEG: Visual Coding Experts Group. WCG: Wide Color Gamut.

REFERENCES: [1] G. J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Transactions on circuits and systems for video technology, vol. 22, no.12, pp – 1668, Dec [2] JVT Draft ITU-T recommendation and final draft international standard of joint video specification (ITU-T Rec. H.264-ISO/IEC AVC), March 2003, JVT-G050 available on [3] D. Grois et al, “Performance Comparison of H.265/ MPEG-HEVC, VP9, and H.264/MPEG- AVC Encoders”, IEEE PCS 2013, pp , San José, CA, USA, Dec 8-11, 2013 [4] D. Mukherjee et al, “The latest open-source video codec VP9–An overview and preliminary results”, Google Inc., United States [5] Z. Wang et al, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp , Apr [6] G. Bjøntegaard, “Calculation of average PSNR differences between RD-curves”, ITU-T Q.6/SG16 VCEG 13th Meeting, Document VCEG-M33, Austin, USA, Apr. 2001

REFERENCES(Cotd..): [7] X. Li et al, “Rate-complexity-distortion evaluation for hybrid video coding”, IEEE International Conference on Multimedia and Expo (ICME), pp , July [8] N. Ling, “High efficiency video coding and its 3D extension: A research perspective,” Keynote Speech, ICIEA, pp , Singapore, July [9] V. sze, M. Budagavi, " Design and Implementation of Next Generation Video Coding Systems (H.265/HEVC Tutorial) ", IEEE ISCAS Tutorial 2014, Melbourne, Australia, June filehttp:// ISCAS.pdf [10] I. E. G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002 [11] A. Puri, X.Chen and A.Luthara, “Video coding using the H.264/MPEG-4 AVC compression standard”, Signal Processing: Image Communication, vol. 19, pp , Oct [12] H.264 tutorial by I.E.G. Richardson: [13] N. Ahmed, T. Natarajan and K. R. Rao, “Discrete Cosine Transform”, IEEE Transactions on Computers, Vol. C-23, pp , Jan

REFERENCES(Cotd..): [14] D. Marpe, H. Schwarz, and T. Wiegand, “Context-based adaptive binary arithmetic coding in the H.264/AVC video compression standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, pp. 620–636, Jul [15] J. Ostermann, et al, " Video coding with H.264/AVC tools, performance, and complexity", IEEE Circuits and Systems Magazine, Vol.4, pp.7-28, Aug [16] J. Ohm, et al, "Comparison of the Coding Efficiency of Video Coding Standards - including High Efficiency Video Coding (HEVC) ", IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, Issue: 12, pp , Dec [17] G. J. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, Vol. 7, No. 6, pp , Dec [18] HEVC white paper - [19] HEVC tutorial by I.E.G. Richardson: [20] W. Malpica and A. Bovik, "Range image quality assessment by structural similarity", IEEE ICASSP 2009, Apr

REFERENCES(Cotd..): [21] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T / ISO-IEC Document: JCTVC J0292r1, July [22] "VP-Next Overview and Progress Update" (PDF). WebM Project (Google). Retrieved Available on: project-update.pdf [23] M. P. Sharabayko et al, "Intra Compression Efficiency in VP9 and HEVC" Applied Mathematical Sciences, Vol. 7, no. 137, pp.6803 – 6824, Hikari Ltd, 2013 [24] J. Padia, “Complexity reduction for VP6 to H.264 transcoder using motion vector reuse,” M.S. Thesis, EE Dept., UTA, Arlington, TX, Available on: [25] White paper on PSNR-NI - [26] Access to HM Reference Software: [27] Access to JM 18.6 Reference Software:

REFERENCES(Cotd..): [28] Chromium® open-source browser project, VP9 source code, Online: 1be17d191ff8506 [29] - Video test sequences (4kx2k) [30] Cisco Visual Networking Index - provider/visual-networking-index-vni/index.html [31] J. Wang et al, "Fractal image coding using SSIM", IEEE 18th International Conference on Image Processing, pp , Brussels, Belgium, sept [32] H.264/AVC Software Reference Manual: [33] HEVC Software Reference Manual : manual.pdf

REFERENCES(Cotd..): [34] K. R. Rao, D. N. Kim and J. J. Hwang, “VideoCodingStandards:AVSChina,H.264/MPEG-4 Part10, HEVC, VP6, DIRAC and VC-1 ”, Springer, [35] V. Sze, M. Budagavi,and G. J.Sullivan "High Efficiency Video Coding (HEVC) : Algorithms and Architectures", Springer, [36] M. Wien, "High Efficiency Video Coding : Coding Tools and Specification", Springer, [37] I. E. Richardson, "Coding Video : A practical guide to HEVC and beyond ", Wiley, 11 May [38] - test sequences [39] ftp://ftp.kw.bbc.co.uk/hevc/hm-11.0-anchors/bitstreams/ - test sequences [40] G. Correa, et al, " Fast HEVC Encoding Decisions Using Data Mining ", IEEE Transactions on Circuits and Systems for Video Technology, Vol. 25, No. 4, pp , April [41] D. K. Kwon and M. Budagavi, " Combined scalable and multiview extension of High Efficiency Video Coding (HEVC) ", IEEE Picture Coding Symposium, pp , Dec [42] Encoding Time Evaluation Intel VTune Amplilfier XE Software profiler Available