1. Performance and Computational Complexity Assessment of High- Efficiency Video Encoders Proposal on: Presented by: MANU RAJENDRA SHEELVANT Under the guidance of: Dr. K. R. RAO

2. ACRONYNMS ALF : Adaptive Loop Filter. AMP : Asymmetric Motion estimator. AVC : Advanced Video Coding. CABAC: Context Adaptive Binary Arithmetic Coding CTB: Coding Tree Block. CTU: Coding Tree Unit. CU: Coding Unit. CB : Coding Block DCT : Discrete Cosine Transform. DBF: De-blocking Filter. FDM : Fast Merge Decision. HEVC: High Efficiency Video Coding. HM: HEVC Test Model. HP : Hierarchical Prediction. JCT: Joint Collaborative Team. JCT-VC: Joint Collaborative Team on Video Coding. JM: H.264 Test Model. JPEG: Joint Photographic Experts Group. MV : Motion Vector. MC: Motion Compensation. ME: Motion Estimation. MPEG: Motion Picture Experts Group. PC : Prediction Chunking. PU : Prediction Unit. PB: Prediction Block. QP: Quantization Parameter. SAO: Sample Adaptive Offset. TB: Transform Block. TU: Transform Unit. VCEG: Video Coding Experts Group.

3. MOTIVATION Study about the computational complexity and encoding performance of the HEVC encoder. Different cases of encoding configurations will be tested on wide variety of video contents and test sequences. The relation/trade-off between HEVC complexity and coding efficiency cost will be studied. Different types of coding tools will be combined and/or left out and then check the how the HEVC complexity and coding efficiency cost will be effected. P. 5.43. Correa et al [2] have investigated the coding efficiency and computational complexity of HEVC encoders. Implement this analysis by considering the set of 16 different encoding configurations.

4. Figure 1: Block Diagram of HEVC CODEC[10] 4 HEVC VIDEO CODEC

5. HEVC ENCODER AND DECODER Figure 2: Block Diagram of the HEVC Encoder[7] 5

6. HEVC ENCODER Six mandatory configurations for normalized experiments have been proposed by JCT-VC [37]. These are combinations of two possible complexity or efficiency settings, defined as high efficiency 10 (HE10) [36] and Main, and three basic coding or access settings, named as  Intra Only  Random Access  Low Delay.

7. POSSIBLE MANDATORY CONFIGURATIONS Complexity/Efficiency Settings Coding/Access Settings 1.High efficiency 10 (HE10)Intra Only 2.High efficiency 10 (HE10)Random Access 3.High efficiency 10 (HE10)Low Delay. 4.MainIntra Only 5.MainRandom Access 6.MainLow Delay. Table 1: Possible Encoding Configurations and Combinations

8. COMPLEXITY/EFFICIENCY SETTINGS High Efficiency 10:- The HE10 setting is used for high coding efficiency (and high complexity). Main:- Main defines a coding configuration with low computational complexity, which does not provide the best coding efficiency.

9. CODING/ACCESS SETTINGS The Intra Only mode uses no temporal prediction. Random Access allows the use of future frames as reference in temporal prediction. Low Delay uses only previous frames as reference temporal prediction.

10. DIFFERENT CONFIGURATION TOOLS Hadamard ME :- Enable(1) or Disable(0) Hadamard Transform for fractional Motion Estimator(ME). DCT is used for fractional Motion Estimation. Hadamard ME reduces the complexity also giving away a little Coding Efficiency. Deblocking Filter Disable :- Disable(0) or Enable(1) Deblocking Filter during encoding process. AMP (Asymmetric Motion Partitioning) :- Enable(1) or Disable(0) AMP. AMP improves the coding efficiency, since irregular image patterns, which otherwise would be constrained to being represented by a smaller symmetric partition, can now be more efficiently represented without requiring further splitting.

11. DIFFERENT CONFIGURATION TOOLS Search Range :- Specifies the search range used for motion estimation. (Default = 96) (To change we use --sr) (If sr = 0 full frame search). Bi- Prediction Refinement (BiPredSearchRange) :- Specifies the search range used for bi-prediction refinement in motion estimation. (Default = 4).

12. DIFFERENT CONFIGURATION TOOLS Fast Encoding (FEN) :- Enable(1) or Disable(0) fast encoding. When enabled, the following occurs: In the SAD computation for blocks having size larger than 8, only the lines of even rows in the block are considered. The number of iterations used in the bi-directional motion vector refinement in the motion estimation process is reduced from 4 to 1. Fast Merge Decision (FDM) :- Enable(1) or disable(0) the use of fast encoder decisions for 2Nx2N merge mode.

13. DIFFERENT CONFIGURATION TOOLS Sample Adaptive Offset (SAO) :- Enable(1) or Disable(0) Sample Adaptive Offset Filter. It reduces sample distortion by first classifying reconstructed samples into different categories, obtaining an offset for each category, and then adding the offset to each sample of the category. Adaptive Loop Filter (ALF) :- Enable(1) or Disable(0) ALF. It is used minimize the mean square error between original pixels and decoded pixels using Wiener-based adaptive filter coefficients.

14. PROPOSED EXECUTION Table 1: Encoder Configurations Used for Complexity and Performance Analysis [2]

15. NEW CONFIGURATION TOOLS HM7.0 -> HM.16.8 Transform Skip :- Enable(1) or Disable(0) transform-skipping mode decision for 4x4 TUs. Transform Skip Fast :- Enable(1) or Disable(0) reduced testing of the transform-skipping mode decision for chroma TUs. When enabled, no RDO search is performed for chroma TUs, instead they are transform-skipped if the four corresponding luma TUs are also skipped. This option has no effect if Transform Skip is disabled.

16. COMPLEXITY ANALYSIS Encode different test sequences and compare their YUV-PSNR, Bit rates, and Time taken to encode the video sequence. Normalized plot (w.r.t Configuration case 16) of time taken in different cases. Normalized plot (w.r.t Configuration case 16) of YUV-PSNR in different cases. YUV-PSNR = [6(PSNR) Y + (PSNR) U + (PSNR) V ]/8 Normalized plot (w.r.t Configuration case 16) of Bit rates in different cases. Plot BD-BitRate v Time Taken. Plot BD-PSNR v Time Taken.

17. REFERENCES [1] G.J. Sullivan et al, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Transactions on Circuits and Systems for Video Technology (CSVT), Vol. 22, No. 12, pp. 1649-1668, Dec. 2012. [2] G. Correa et al, “Performance and computational complexity assessment of high efficiency video encoders”, IEEE Trans. CSVT, vol. 22, pp.1899-1909, Dec. 2012. [3] Video test sequences - https://media.xiph.org/video/derf/.https://media.xiph.org/video/derf/. [4] HM 16.8 software - https://hevc.hhi.fraunhofer.de/trac/hevc/milestone/HM-16.8.https://hevc.hhi.fraunhofer.de/trac/hevc/milestone/HM-16.8. [5] I.E.G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002. [6] B. Bross et al, “High Efficiency Video Coding (HEVC) Text Specification Draft 10”, Document JCTVC- L1003, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC), Mar.2013. [7] D. Marpe et al, “The H.264/MPEG4 advanced video coding standard and its applications”, IEEE Communications Magazine, Vol. 44, pp. 134-143, Aug. 2006.

18. REFERENCES (Contd..) [8] HEVC white paper-Ateme: http://www.ateme.com/an-introduction-to-uhdtv-and-hevchttp://www.ateme.com/an-introduction-to-uhdtv-and-hevc [9] G.J. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, Vol. 7, No. 6, pp. 1001-1016, Dec. 2013. [10] HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.htmlhttp://www.vcodex.com/h265.html [11] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T / ISO-IEC Document: JCTVC J0292r1, July 2012. [12] T. Wiegand et al, “Overview of the H.264/AVC Video Coding Standard”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 13, No. 7, pp. 560-576, July 2003. [13] U.S.M. Dayananda, “Study and Performance comparison of HEVC and H.264 video codecs” M.S. Thesis, EE Dept., UTA, Arlington, TX, Dec. 2011 available on http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/index_tem.html http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/index_tem.html [14] M. Wein, “ High Efficiency Video Coding: Coding Tools and Specification”, Springer, 2014.

19. REFERENCES (Contd..) [15] T. Vermeir, “Use cases and requirements for lossless and screen content coding”, JCTVC-M0172, 13 th JCT-VC meeting, Incheon, KR, Apr. 2013. [16] J. Sole et al, “Requirements for wireless display applications”, JCTVC-M0315, 13 th JCT-VC meeting, Incheon, Korea, Apr. 2013. [17] A. Gabriellini et al, “Combined Intra-Prediction for High-Efficiency Video Coding”, IEEE Journal of selected topics in Signal Processing. Vol. 5, no. 7, pp. 1282-1289, Nov. 2011. [18] L. Younghoon, K. Jungsoo, and K. Chong-Min, “Energy-aware video encoding for image quality improvement in battery-operated surveillance camera,” IEEE Trans. Very Large Scale Integration (VLSI) Syst., vol. 20, no. 2, pp. 310–318, Feb. 2012. [19] W. Kim, J. You, and J. Jeong, “Complexity control strategy for real-time H.264/AVC encoder,” IEEE Trans. Consumer Electronics, vol. 56, no. 2, pp. 1137–1143, May 2010.

20. REFERENCES (Contd..) [20] D. N. Kwon, et al, “Performance and computational complexity optimization in configurable hybrid video coding system,” IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 1, pp. 31–42, Jan. 2006. [21] O. Lehtoranta and T. D. Hamalainen, “Complexity analysis of spatially scalable MPEG-4 encoder,” in Proc. Int. Symp. System-on-Chip, 2003, pp. 57–60. [22] J. Ostermann, et al, “Video coding with H.264/AVC: Tools, performance, and complexity,” IEEE Circuits Syst. Mag., vol. 4, no. 1, pp. 7–28, Jan.–Mar. 2004. [23] S. Saponara, et al, “Performance and complexity co-evaluation of the advanced video coding standard for cost-effective multimedia communications,” EURASIP J. Appl. Signal Process., pp. 220– 235, Jan. 2004. [24] P. Li, Y. Chen, and W. Ji, “Rate-distortion-complexity analysis on AVS encoder,” presented at the 11th Pacific Rim Conf. Multimedia, Shanghai, China, 2011.

21. REFERENCES (Contd..) [25] A. Hallapuro, V. Lappalainen, and T. D. Hamalainen, “Performance analysis of low bit rate H.26L video encoder,” presented at the Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 2. 2001. [26] M. Horowitz, et al, “H.264/AVC baseline profile decoder complexity analysis,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 704–716, July 2003. [27] L. Szu-Wei and C. C. J. Kuo, “Complexity modeling of spatial and temporal compensations in H.264/AVC decoding,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 5, pp. 706–720, May 2010. [28] M. Zhan, H. Hao, and W. Yao, “On complexity modeling of H.264/AVC video decoding and its application for energy efficient decoding,” IEEE Trans. Multimedia, vol. 13, no. 6, pp. 1240–1255, Dec. 2011. [29] V. Lappalainen, A. Hallapuro, and T. D. Hamalainen, “Complexity of optimized H.26L video decoder implementation,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 717–725, July 2003.

22. REFERENCES (Contd..) [30] On Software Complexity, JCTVC-G757, ISO/IEC-JCT1/SC29/WG11, Geneva, Switzerland, 2011. [31] HM Decoder Complexity Assessment on ARM, JCTVC-G262, ISO/IECJCT1/SC29/WG11, Geneva, Switzerland, 2011. [32] Z. Chang-Guo et al, “MPEG video decoding with the UltraSPARC visual instruction set,” in Proc. Compcon ’95: Technol. Inform. Superhighway Dig. Papers, pp. 470–477, 1995. [33] L. Xiang, M. Wien, and J. R. Ohm, “Rate-complexity-distortion optimization for hybrid video coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 7, pp. 957–970, Jul. 2011. [34] PTscalar V1.0 [Online]. Available: http://eda.ee.ucla.edu/PTscalar/http://eda.ee.ucla.edu/PTscalar/ [35] OProfile: A System Profiler for Linux [Online]. Available: http://oprofile.sourceforge.net/news/http://oprofile.sourceforge.net/news/

23. REFERENCES(Contd..) [36] Common Test Conditions and Software Reference Configurations, JCTVC-I1100, ISO/IEC- JCT1/SC29/WG11, Geneva, Switzerland, 2012. [37] G. Correa, et al, “Complexity control of high efficiency video encoders for power-constrained devices,” IEEE Trans. Consumer Electron., vol. 57, no. 4, pp. 1866– 1874, Nov. 2011. [38] G. Laroche, J. Jung, and B. Pesquet-Popescu, “RD optimized coding for motion vector predictor selection,” IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 9, pp. 1247–1257, Sep. 2008. [39] S. Oudin et al, “Block merging for quadtree-based video coding,” in Proc. IEEE Int. Conf. Multimedia Expo, Jul. 2011, pp. 1–6. [40] V. Sze and M. Budagavi, “Parallelization of CABAC transform coefficient coding for HEVC,” in Proc. PCS, 2012, pp. 509–512. [41] HM Software Manual: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/trunk/doc/software-manual.pdf. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/trunk/doc/software-manual.pdf

24. REFERENCES(Contd..) [42] N. Ling, “High efficiency video coding and its 3D extension: A research perspective,” Keynote Speech, ICIEA, Singapore, July 2012. [43] Tut2. X. Wang et al, “Paralleling variable block size motion estimation of HEVC on CPU plus GPU platform”, IEEE ICME workshop, 2013. [44] Tut3. H.R. Tohidpour, M.T. Pourazad and P. Nasiopoulos, “Content adaptive complexity reduction scheme for quality/fidelity scalable HEVC”, IEEE ICASSP 2013, pp. 1744-1748, June 2013. [45] Tut4. M. Wien, “HEVC – coding tools and specifications”, Tutorial, IEEE ICME, San Jose, CA, July 2013. [46] Tut5. D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial) Monday 29 June 2015 11.30am-3:00pm, IEEE ICME 2015, Torino, Italy, 29 June – 3 July, 2015. [47] Tut6. D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial), IEEE ICCE, Berlin, Germany, 6 – 9 Sept. 2015. [48] Tut7. D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial) Sunday 27 Sept 2015, 9:00 am to 12:30 pm), IEEE ICIP, Quebec City, Canada, 27 – 30 Sept. 2015. The tutorial below is for personal use only. Password: a2FazmgNK https://datacloud.hhi.fraunhofer.de/owncloud/public.php?service=files&t=8edc97d26d46d4458a9c1a17964bf 881https://datacloud.hhi.fraunhofer.de/owncloud/public.php?service=files&t=8edc97d26d46d4458a9c1a17964bf 881.

25. Any Suggestions/Questions are welcome. Thank You.