Investigation of Image Quality of Dirac, H.264 and H.265 Biju Shrestha UTA ID: 1000113697 Email: biju.shrestha@mavs.uta.edu EE 5359: Multimedia Processing 9/16/2018

Overview Introduction Image quality assessment using SSIM and FSIM Dirac H.264 H.265 Image quality assessment using SSIM and FSIM SSIM – structural similarity metric FSIM – featured similarity index Conclusion References

Introduction Video codec compress/decompress digital video Types of video codecs Dirac H.264 H.265 Source: [2]

Dirac Developed by BBC (British Broadcasting Corporation) Research and is open source Powerful and flexible despite using less number of core tools [1] Features [1] Multi-resolution transforms Inter and intra frame coding Frame and field coding Dual syntax CBR (constant bit rate) and VBR (variable bit rate) operations Variable bit depths. Multiple chroma sampling formats Lossless and lossy coding Choice of wavelet filters Simple stream navigation Source: [1]

Dirac Encoder Architecture Source: [15]

Dirac Decoder Architecture Source: [18]

H.264 Also referred as AVC (advance video coding) is a standard for video compression [2] Joint development of video coding experts group (VCEG) of the ITU-T* and the moving picture experts group (MPEG) of ISO/IEC* [11] Enhanced coding efficiency Applications - video telephony, video conferencing, TV, storage, streaming video, digital video authoring, digital cinema, etc. [11] *ITU-T : international telecommunication union – telecommunication standardization sector *ISO : international organization for standardization *IEC : international engineering consortium Source: [2, 11]

H.264 Decoder Block Diagram Source: [2]

H.264 Encoder Block Diagram Source: [2]

H.264 Profiles Source: [12]

H.265 Also known as high efficiency video coding (HEVC) [3] Can deliver significant improvement relative to AVC (ITU-T* H.264 | ISO/IEC 14496-10) [10] Efficiency can be progressed by average of 37 % for hierarchical B structure and 36 % for IPPP structure [16] Video codec is composed of many processes including intra prediction and inter prediction, transforms, quantization, entropy coding, and filtering [17] *ITU-T: international telecommunication union – telecommunication standardization sector Source: [3, 10, 16, 17]

H.265 Encoder Block Diagram Grey boxes – Proposed tools White boxes – H.264/AVC tools *IST : integer sine transform Source: [17]

H.265 Decoder Block Diagram Grey boxes – Proposed tools White boxes – H.264/AVC tools Source: [27]

Image Quality Assessment Digital images and videos are prone to several distortion [5] Nonstructural distortion Structural distortion Phases when distortion occurs [5] Acquisition Processing Compressing Storage Transmission Reproduction Results of distortion – poor visual quality [5] Different metrics available to quantify visual quality [3, 8, 13, 14] Source: [3, 5, 8, 13, 14]

Nonstructural and Structural Distortions Source: [14]

Comparison Metrics FSIM – Featured Similarity index SSIM – Structural Similarity metric Bit rate PSNR – Peak Signal to Noise Ratio MSE – Mean Squared Error

FSIM/FSIMc Based on human visual system (HVS) FSIM is designed for gray-scale images FSIMc incorporates chrominance information Mathematical model of FSIM SL(x) = overall similarity between f1(x) and f2(x) Mathematical model of FSIMc λ > 0 is the parameter used to adjust the importance of the chrominance components. Source: [3]

FSIM/FSIMc Index Computation f1 is the reference image, and f2 is a distorted version of f1 [3]. Source: [3]

SSIM Based on degradation of structural information [5] HVS is adapted to extract structural information from an images [14] Figure: Block diagram of structural similarity (SSIM) measurement system [5] Source: [5, 14]

Mathematical Representation of SSIM General form of SSIM α, β, and γ are parameters that mediate the relative importance of the three components µx and µy = local sample means of x and y respectively σx and σy = local sample standard deviations of x and y respectively σxy = local sample correlation coefficient between x and y C1, C2, and C3 = constants that stabilize the computations when denominators become small Using α = β = γ = 1. We get, Source: [7]

PSNR and MSE Both are directly dependent on the intensity of an image [3] Both do not correlate with subjective fidelity ratings [3] Both cannot model human visual system very accurately [4] Source: [3, 4]

Example: SSIM and MSE *MSE is approximately same for all images but SSIM is different giving better comparison[13] Source: [13, 22]

Ranking of Image Quality Assessment Metrics Different metrics take different approach to quantify images Ranking of metrics by different databases   TID2008 CSIQ LIVE IVC MICT A57 FSIM 1 SSIM 2 PSNR 3 Table : Ranking of image quality assessment metrics performance on six databases[5] Source: [3]

Results Video Information QCIF sequence: foreman_qcif.yuv Frame height: 176 Frame width: 144 Frame rate: 30 frame/second Source: [28] 9/16/2018

Results from H.264 Footer Text 9/16/2018

Results from Dirac 9/16/2018

MSU Video Quality Measurement Tool 3.0 Open source video quality measurement tool Calculate different metrics like PSNR, SSIM, MSSSIM, and MSE 9/16/2018

Conclusion Measurement of metrics like PSNR, MSE, SSIM, and MS SSIM respectively for various bitrates for foreman QCIF sequence using H.264 encoder and Dirac encoder was performed Further measurement of metrics using H.265 is needed to do comparative analysis Comparative metrics like PSNR, MSE, SSIM, MS SSIM, and FSIM at various bitrates are being used Based on various test sequences, the performance of codec will be investigated Software and tools that will be used: MATLAB Visual studio JM software KTA software Dirac Pro MSU VQMT (Moscow State University video quality measurement tool) Source: [19 - 24]

References [1] Dirac Video (2008, September 23), “Dirac Specification” [Online]. Available: http://diracvideo.org/download/specification/dirac-spec-latest.pdf [2] I. Richardson (2011), “A Technical Introduction to H.264/AVC” [Online]. Available: http://www.vcodex.com/files/H.264_technical_introduction.pdf [3] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity index for image quality assessment,” IEEE Transactions on Image Processing, vol.20, no.8, pp.2378-2386, Aug. 2011. [4] Z.Li and A.M. Tourapis, “New video quality metrics in the H.264 reference software,” Input Document to JVT, Hannover, DE, 20-25 Jul. 2008. [5] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli,“Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, issue 4, pp. 600-612, Apr. 2004. [6] Z. Wang, E.P. Simoncelli, and A.C. Bovik, “Multiscale structural similarity for image quality assessment,” Conference Record of the Thirty-Seventh Asilomar Conference on Signals, Systems and Computers, 2003, vol.2, pp. 1398- 1402, 9-12 Nov. 2003. [7] C. Li, and A. C. Bovik, “Content-weighted video quality assessment using a three-component image model.” Journal of Electronic Imaging, vol.19, pp. 65-71, Mar. 2010. [8] X. Ran and N. Farvardin, “A perceptually-motivated three-component image model - part I: description of the model,” IEEE Transactions on Image Processing, vol.4, no.4, pp.401-415, Apr. 1995. [9] J. L. Li, G. Chen, and Z. R. Chi, “Image coding quality assessment using fuzzy integrals with a three-component image model,” IEEE Transactions on Fuzzy Systems, vol.12, no.1, pp. 99- 106, Feb. 2004. [10] G. J. Sullivan and J. Ohm, “Recent developments in standardization of high efficiency video coding (HEVC),” Proc. SPIE 7798, 77980V, 2010. [11] G. Sullivan, P. Topiwalla, and A. Luthra, “The H.264/AVC video coding standard: overview and introduction to the fidelity range extensions,” SPIE Conference on Applications of Digital Image Processing XXVII, vol. 5558, pp. 53-74, Aug. 2004. [12] A. Puri, X. Chen, and A. Luthra, “Video coding using the H.264/MPEG-4 AVC compression standard,” Signal Processing: Image Communication, vol. 19, pp. 793-849, Oct. 2004.

References [13] Z. Wang et al (2003, February), “The SSIM index for image quality assessment” [Online]. Available: https://ece.uwaterloo.ca/~z70wang/research/ssim/ [14] C. Chukka, “A universal image quality index and SSIM comparison” [Online]. Available: http://www-ee.uta.edu/Dip/Courses/EE5359/chaitanyaee5359d.pdf [15] BBC Research, “The technology behind Dirac” [Online]. Available: http://www.bbc.co.uk/rd/projects/dirac/technology.shtml [16] E. Alshina et al, “Technical considerations of new challenges in video coding standardization,” International Organization for Standardization Organization Internationale De Normalisation ISO/IEC JTC1/SC29/WG11 Coding of Moving Pictures and Audio, Oct. 2008. [17] S. Jeong et al, “Highly efficient video codec for entertainment quality,” ETRI Journal, vol.33, no. 2, pp. 145-154, Apr. 2011. [18] K. R. Rao and D. N. Kim, “Current video coding standards: H.264/AVC, Dirac, AVS China and VC-1,” 42nd Southeastern Symposium on System Theory (SSST), pp.1-8, Mar. 2010. [19] A. M. Tourapis (January 2009), “H.264/14496-10 AVC reference software manual” [Online]. Available: http://iphome.hhi.de/suehring/tml/JM%20Reference%20Software%20Manual%20%28JVT-AE010%29.pdf [20] F. Bossen, D. Flynn, and K. Sühring (July 2011), “HEVC reference software manual” [Online]. Available: http://phenix.int-evry.fr/jct/doc_end_user/documents/6_Torino/wg11/JCTVC-F634-v2.zip [21] DiracPRO software: http://dirac.kw.bbc.co.uk/download/ [22] D. T. Lee, “JPEG 2000: Retrospective and new developments,” Proc. IEEE, vol. 93, pp. 32-41, Jan. 2005. [23] KTA software: http://iphome.hhi.de/suehring/tml/download/KTA/ [24] H.264/AVC Reference Software: http://iphome.hhi.de/suehring/tml/download/ [25] A. Ravi, “Performance analysis and comparison of the Dirac video codec with H.264/MPEG-4 part 10 AVC,” M.S. thesis, Dept. Elect. Eng., Univ. of Texas at Arlington, 2009 [25] I.E.G. Richardson, “H.264 and MPEG-4 video compression: video coding for next generation multimedia,” Great Britain: Wiley, 2003, pp. 159-223 [26] MSU video quality measurement tool: http://compression.ru/video/quality_measure/video_measurement_tool_en.html [27] B. Bross et al, “High efficiency video coding (HEVC) text specification draft 6,” Joint collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, 7th Meeting: Geneva, CH, 21–30 Nov. 2011. [28] “YUV video sequences” [Online]. Available: http://trace.eas.asu.edu/yuv/