1. EE5359 Multimedia Processing Interim Presentation SPRING 2015 ADVISOR: Dr. K.R.Rao EE5359 Multimedia Processing1 BY: BHARGAV VELLALAM SRIKANTESWAR Email-bhargav.vellalamsrikanteswa@mavs.uta.edu UTA ID – 1001048654 Submission date – 21 st April 2015

2.  3D - Three Dimension  AVC - Advanced Video Coding  AMVP - Advanced motion vector prediction  CTU - Coding Tree  CU - Coding Unit  CABAC - Context Adaptive Binary Arithmetic Coding  DCT- Discrete Cosine Transform  FPS – Frames Per Second  GOP - Group Of Pictures  HD - High Definition EE5359 Multimedia Processing2

3.  HEVC - High Efficiency Video Coding  ISO - International Organization for Standardization  MPEG - Moving Pictures Experts Group  MSE - Mean Square Error  MV - Motion Vector  POC - Phase-Only Correlation  PB – Prediction Block  PU - Prediction Unit  QP - Quantization Parameter EE5359 Multimedia Processing3

4.  RD – Rate Distortion  SAO - Sample Adaptive Offset  SSIM – Structural Similarity Index Metrics  TB - Transform Block  TU- Transform Unit  VCEG - Video Coding Experts Group EE5359 Multimedia Processing4

5.  Introduction to HEVC  About Block Artifacts  Proposed Implementation  Comparison metrics  Profile used and testing platform  Configuration of HM 16.4  Test sequences  Results  Conclusion and Future work  References EE5359 Multimedia Processing5

6.  HEVC is the latest video coding standard jointly presented by ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group[1].  The best performance improvement of HEVC over H.264 is ~50% bit rate reduction for equal perceptual video quality[1].  The growing popularity of HD video, and the emergence of beyond-HD formats (e.g.4kx2k or 8kx4k resolution) are creating even stronger needs for coding efficiency superior to H.264/MPEG-4 [4] AVC's capabilities[4]. EE5359 Multimedia Processing6

7. 7 Figure 1: Comparison of bit rates in HEVC over previous standards[14]

8. EE5359 Multimedia Processing 8 Figure 2: Block diagram of HEVC Encoder[9]

9.  Partitioning each picture into multiple units  Predicting each unit using inter or intra prediction, and subtracting the prediction from the unit  Transforming and quantizing the residual (the difference between the original picture unit and the prediction)  Entropy encoding transform output, prediction information, mode information and headers[17] EE5359 Multimedia Processing9

10. 10 Figure 3: HEVC Decoder[10]

11.  Entropy decoding and extracting the elements of the coded sequence  Rescaling and inverting the transform stage  Predicting each unit and adding the prediction to the output of the inverse transform  Reconstructing a decoded video image [17] EE5359 Multimedia Processing11

12.  A distortion that appears in compressed video material as abnormally large pixel blocks[6].  Occurs when the encoder cannot keep up with the allocated bandwidth[6].  Visible with fast motion sequences or quick scene changes[6]. EE5359 Multimedia Processing12

13. EE5359 Multimedia Processing13 Figure 4 : example showing macro blocks in a image[15]

14.  In a coding scheme that uses block-based prediction and transform coding, discontinuities can occur in the reconstructed signal at the block boundaries [17].  Visible discontinuities at the block boundaries are blocking artifacts [17]. EE5359 Multimedia Processing14

15. EE5359 Multimedia Processing15 Figure 5 : Block boundary with blocking artifact[17]

16.  The HEVC de-blocking filter significantly improves the subjective quality of coded video sequences at lower bitrates[6].  Reference software encoder may produce visible block artifacts on some sequences with content that shows chaotic motion, such as water or fire[6]. EE5359 Multimedia Processing16

17.  Analyzed the reasons for blocking artifacts in various sequences  Two simple encoder-side methods are implemented that improve the subjective quality.  0 to 1% increase in bit rate. EE5359 Multimedia Processing17

18. In order to attenuate block artifacts in a picture with higher depth, HEVC can be configured to signal the deblocking filter offsets at the slice/picture level. Higher offsets are sent for the frames, which are at higher depth in the coding hierarchy. The proposed approach relaxes the deblocking decisions thresholds and clipping values for the pictures at higher depth [6]. EE5359 Multimedia Processing18

19.  When the rate-distortion optimization chooses 32 × 32 intra-predicted CUs at higher depth, the prediction is often coarse.  It might be difficult to conceal a blocking artifact by just applying the deblocking filtering  By limiting the maximum TU size to 16 × 16 samples for coding of intra CUs in inter-predicted slices.  This will restrict the maximum intra-predicted block size and will increase the bit rate [6]. EE5359 Multimedia Processing19

20.  PSNR – Peak Signal to Noise Ratio  Computational time  BD- Bit rate  BD - PSNR EE5359 Multimedia Processing20

21.  The HM 16.4 main profile [8], This profile allows for a bit depth of 8-bits per sample with 4:2:0 chroma sampling, which is the most common type of video used with consumer devices.  TESTING PLATFORM Processor : Intel(R) core(TM) i5-4200,2.30 GHz Memory : 6 GB Operating system : 64 bit windows 8 EE5359 Multimedia Processing21

22.  The project is implemented in Random Access configuration of HM 16.4  Profile : main  IntraPeriod : 1 # Period of I-Frame ( -1 = only first)  GOPSize : 8 # GOP Size (number of B slice = GOPSize-1)  QP : 32 # Quantization parameter(0-51)  FastSearch 1 # 0:Full search 1:TZ search  SearchRange : 64 # (0: Search range is a Full frame) EE5359 Multimedia Processing22

23. EE5359 Multimedia Processing23 The following test sequences are used for analyzing the project [11]

24. EE5359 Multimedia Processing24 Figure 6: RaceHorses_416x240_30.yuv [11] Figure 7: ParkScene_1920x1080_24.yuv [11] Figure 8: Kimono_1920x1080_24.yuv [11]

25. EE5359 Multimedia Processing25 Figure 9: football30_cif_90.yuv [11]Figure 10: BasketballDrill_832x480_50.yuv [11

26. EE5359 Multimedia Processing26 Figure 11: ParkScene_1920x1080_24.yuv Without reduction of block artifacts Figure 12: ParkScene_1920x1080_24.yuv with reduction of block artifacts Figure 11 and Figure 12 are encoded sequences for a QP of 42 without and with reduction of block artifacts respectively.

27. EE5359 Multimedia Processing27

28. EE5359 Multimedia Processing28 Figure 13 : RD- Plot for RaceHorses_416x240_30.yuv

29. EE5359 Multimedia Processing29 Figure 14 : QP vs PSNR comparison plot for RaceHorses_416x240_30.yuv

30. EE5359 Multimedia Processing30 Figure 15 : QP vs Encoding time comparison Plot for RaceHorses_416x240_30.yuv

31. EE5359 Multimedia Processing31

32. EE5359 Multimedia Processing32 Figure 16 : RD- Plot for ParkScene_1920x1080_24.yuv

33. EE5359 Multimedia Processing33 Figure 17: QP vs PSNR comparison Plot for ParkScene_1920x1080_24.yuv

34. EE5359 Multimedia Processing34 Figure 18: QP vs Encoding time comparison Plot for ParkScene_1920x1080_24.yuv

35. EE5359 Multimedia Processing35

36. EE5359 Multimedia Processing36 Figure 19: RD-Plot for Kimono_1920x1080_24.yuv

37. EE5359 Multimedia Processing37 Figure 20: QP vs PSNR comparison plot for Kimono_1920x1080_24.yuv

38. EE5359 Multimedia Processing38 Figure 21: QP vs Encoding time comparison plot for Kimono_1920x1080_24.yuv

39. EE5359 Multimedia Processing39

40. EE5359 Multimedia Processing40 Figure 22: RD-Plot for football30_cif_90.yuv

41. EE5359 Multimedia Processing41 Figure 23: QP vs PSNR comparison plot for football30_cif_90.yuv

42. EE5359 Multimedia Processing42 Figure 24: QP vs Encoding time comparison Plot for football30_cif_90.yuv

43. EE5359 Multimedia Processing43

44. EE5359 Multimedia Processing44 Figure 25: RD-Plot for BasketballDrill_832x480_50.yuv

45. EE5359 Multimedia Processing45 Figure 26: QP vs PSNR comparison plot for BasketballDrill_832x480_50.yuv

46. EE5359 Multimedia Processing46 Figure 27: QP vs Encoding time comparison Plot for BasketballDrill_832x480_50.yuv

47. EE5359 Multimedia Processing47

48. EE5359 Multimedia Processing48 Figure 28: BD-PSNR Comparison with and without blocking artifacts

49. EE5359 Multimedia Processing49 Figure 29: %BD-Bit rate comparison with and without blocking artifacts

50.  RD plots infer that after the blocking artifacts are reduced in a sequence, PSNR increases where as Bit rate also increases. This is a drawback.  As the Quantization Parameter is increased the encoding time decreases.  BD-Bit rate plot indicates that there is 0.578% increase in average BD-bit rate with reduction of blocking artifacts.  BD-PSNR indicates that there is an average of 1.67dB increase in BD-PSNR with reduction of blocking artifacts.  Future work has to be done to decrease the bit rate and encoding time. EE5359 Multimedia Processing50

51. 1. G.J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Transactions on CSVT, vol. 22, pp.1649-1668, Dec. 2012. 2. P. Hanhart et al, “Subjective quality evaluation of the upcoming HEVC video compression standard” SPIE Applications of digital image processing XXXV, vol.8499, pp.8499-30, Aug. 2012. 3. G J. Sullivan et al,” Standardized Extensions of HEVC”, IEEE Journal of Selected topics in Signal Processing, Vol.7, no.6, pp.1001-1016, Dec. 2013. 4. F. Bossen et al, ” HEVC Complexity and Implementation Analysis”, IEEE Transactions on CSVT, vol.22, pp.1685-1696, Dec. 2012. 5. HEVC white paper-Ateme: http://www.ateme.com/an-introduction-to-uhdtv-and-hevc.http://www.ateme.com/an-introduction-to-uhdtv-and-hevc 6. A. Norkin et al,"Two HEVC encoder methods for block artifact reduction ", IEEE International Conference on Visual Communications and Image Processing (VCIP) article no. 14028673,pp.1-6,Nov.2013 EE5359 Multimedia Processing51

52. 7. L. Zhao et al, “Fast mode decision algorithm for intra prediction in HEVC”, IEEE International Conference on Visual Communications and Image Processing Conference (VCIP) article no.6115979,6-9 Nov. 2011. 8. HEVC Reference Software HM16.4. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16. 9. D. Marpe,T.Weigand and G.J.Sullivan, “The H.264/MPEG4 advanced video coding standard and its applications”, IEEE Communications Magazine, Vol. 44, pp. 134-143, Aug. 2006. 10. C. Fogg, “Suggested figures for the HEVC specification”, ITU-T / ISO-IEC Document: JCTVC J0292r1, July 2012. 11. Required test sequences http://media.xiph.org/video/derf/http://media.xiph.org/video/derf/ 12. V. Sze and M. Budagavi, "High Throughput CABAC Entropy Coding in HEVC" (PDF). IEEE Transactions on Circuits and Systems for Video Technology, Vol.22, pp,1778-1791, Dec 2012."High Throughput CABAC Entropy Coding in HEVC" 13. T.Nguyen et al, "Transform Coding Techniques in HEVC" IEEE Journal of Selected Topics in Signal Processing,Vol.7, pp. 978–989, Dec. 2013. EE5359 Multimedia Processing52

53. 14. W.Y.Wei, "Deblocking Algorithms in Video and Image Compression Coding." Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan, ROC. 15. A. Norkin et al, “HEVC Deblocking Filter”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 22, No. 12, pp. 1746-1754, Dec. 2012. 16. W.Shen et al, “A high-throughput VLSI architecture for deblocking filter in HEVC” IEEE International Symposium on Circuits and Systems (ISCAS), pp 673-676, May 2013 17. Access to HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.htmlhttp://www.vcodex.com/h265.html 18. I. E.G. Richardson, “The H.264 Advanced Video Compression Standard”, 2nd Edition, Wiley 2010 19. N.Ahmed, T.Natarajan and K.R.Rao, " Discrete Cosine Transform ", IEEE Transactions on Computers, Vol 23, PP 90- 93, Jan.1974. EE5359 Multimedia Processing53

54. 20. Access to HM 16.4 software manual: http://hevc.kw.bbc.co.uk/svn/jctvc-a124/tags/HM-16.4/doc/software- manual.pdfhttp://hevc.kw.bbc.co.uk/svn/jctvc-a124/tags/HM-16.4/doc/software- manual.pdf 21. V.Sze, M,Budagavi and G.J.Sullivan,"High Efficiency Video Coding (HEVC): Algorithms and Architectures", 1st Edition, Springer 2014. 22. I.E.G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002. 23. M.Wein," High Efficiency Video Coding, Coding Tools and Specification”, Springer 2015. 24. X. Li et al, “Rate-complexity-distortion evaluation for hybrid video coding”, IEEE International Conference on Multimedia and Expo (ICME), pp. 685-690, July 2010. 25. G. Bjontegaard, “Calculation of Average PSNR Differences between RD Curves”, document VCEGM33, ITU-T SG 16/Q 6, Austin, TX, Apr. 2001. 26. I.E.G. Richardson, "Coding Video: A Practical Guide to HEVC and Beyond" Wiley, May 2015. EE5359 Multimedia Processing54

55. 27. J. Vanne et al, “Comparative Rate-Distortion-Complexity Analysis of HEVC and AVC Video Codecs”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 22, No. 12, pp. 1885-1898, Dec. 2012. 28. HM Encoder Description: http://mpeg.chiariglione.org/standards/mpeg-h/high-efficiency-video- coding/n14703-high-efficiency-video-coding-hevc-encoderhttp://mpeg.chiariglione.org/standards/mpeg-h/high-efficiency-video- coding/n14703-high-efficiency-video-coding-hevc-encoder 29. White paper on PSNR-NI: http://www.ni.com/white-paper/13306/en/http://www.ni.com/white-paper/13306/en/ 30. HEVC Tutorial by V.Sze and M.Budagavi,” Design and Implementation of Next Generation video coding systems”, IEEE International Symposium on Circuits and Systems (ISCAS), Melbourne,Australia,June 2014. 31. A.Hore and D.Ziou," Image Quality Metrics : PSNR vs SSIM",IEEE International Conference on Pattern Recognition (ICPR),pp. 2366 - 2369, Aug. 2010. 32. G. Correa et al, “Fast HEVC encoding decisions using data mining”, IEEE Transactions on CSVT, vol.25, pp. 660-673, Apr.2015. EE5359 Multimedia Processing55

56. 33. Intel VTune Amplifier XE Software profiler website : http://software.intel.comhttp://software.intel.com 34. H.Schwarz, D.Marpe and T.Weigand," Analysis of Hierarchical B pictures and MCTF", IEEE International Conference on Multimedia and Expo (ICME),pp. 9-12, July 2006. EE5359 Multimedia Processing56

57. Thank You. EE5359 Multimedia Processing57