Visual Quality Analysis for Images Degraded by Different Types of Noise Nikolay Ponomarenkoa, Vladimir Lukina, Oleg I. Ieremeieva, Karen Egiazarianb, Jaakko T. Astolab a National Aerospace University, 61070, Kharkov, Ukraine; b Tampere University of Technology, Institute of Signal Processing, P.O. Box-553, FIN-33101, Tampere, Finland Nikolay Ponomarenko 22/04/2007

Visual Quality Metrics Classic metrics: MSE, SNR, PSNR. HVS-metrics: PSNR-HVS-M, MSSIM, VSNR, etc. Nikolay Ponomarenko 22/04/2007

Typical Metric Verification Approach Main characteristics Test image database TID2008 LIVE Database 1 Number of distorted images 1700 779 2 Number of different types of distortions 17 5 3 Number of experiments carried out Totally 838 (437 - Ukraine, 251 - Finland, 150 - Italy) 161 (all USA) 4 Methodology of visual quality evaluation Pair-wise sorting (choosing the best that visually differs less from original between two considered) Evaluation using five level scale (Excellent, Good, Fair, Poor, Bad) Number of elementary evaluations of image visual quality in experiments 256428 25000 6 Scale of obtained estimates of MOS 0..9 0..100 (stretched from the scale 1..5) 7 Variance of estimates of MOS 0.63 250 8 Normalized variance of estimates of MOS 0.031 0.083 Nikolay Ponomarenko 22/04/2007

Tampere Image Database 2008 (TID2008) 24 images are fragments of the Kodak images with resolution 512x384 pixels. 25th image is additionally created. Nikolay Ponomarenko 22/04/2007

Examples of Images with Signal-dependent Noise Raw optical images X-ray image SAR image Ultrasound image UV and IR images No one image database includes images distorted by signal-dependent noise Nikolay Ponomarenko 22/04/2007

Generation of Noisy Images Peculiarities of generating images with signal-dependent noise: 25 reference (noise-free) images from TID2008 were used; each reference image has been distorted by 3 types of noise: additive, Poisson, multiplicative; MSEs for distorted images are equal for each color component Dependence of noise variance on image true value in ij-th pixel can be described as Additive noise: zero mean, Gaussian and with constant variance Poisson noise: noise variance depends on the true value as Multiplicative noise: noise variance depends on the true value as Nikolay Ponomarenko 22/04/2007

Example of Test Images a b c d Test noise-free image #1 (a) and its fragments for multiplicative (b), Poisson (c) and additive (d) noise cases Nikolay Ponomarenko 22/04/2007

Subjective experiments In subjective experiments, participants compared 3 distorted images and gave them ratings ( distorted image with the best visual quality got 3 point, the worst – 1 point). More than 60 experiments were conducted. Nikolay Ponomarenko 22/04/2007

Mean Opinion Score The images ##1, 12, 15, 20 corrupted by multiplicative noise are perceived as having the highest visual quality. The images ## 4, 5, 18 with additive noise are perceived better than others distorted images. Poisson noisy images are perceived as having the best visual quality only for the test image #16 and for others cases mainly took the second (intermediate) place. For 17 of 25 test the minimal and maximal values of MOS are all within the limits from approximately 1.8 till 2.2. Visual quality of these images is practically the same, and different participants have chosen images with different types of noise as the best. Nikolay Ponomarenko 22/04/2007

Examples of Test Images a b c Test image # 4: noise-free image (a) and its fragment for additive (b) and multiplicative (c) noise cases Nikolay Ponomarenko 22/04/2007

Examples of Test Images a b c Test image # 12: noise-free image (a) and its fragment for additive (b) and multiplicative (c) noise cases Nikolay Ponomarenko 22/04/2007

Modification of the quality metric PSNR-HVS-M The metric PSNR-HVS-M is calculated according to the following two basic expressions: is determined in discrete cosine transform (DCT) domain in blocks of size 8х8 pixels taking into account weight matrix and masking effects, D is the dynamic range of values for a considered image. Nikolay Ponomarenko 22/04/2007

Modification of the quality metric PSNR-HVS-M Weber–Fechner law states that the relationship between perception and stimulus is logarithmic: The modified (proposed) metric PSNR-HVS-MW is calculated using local weight wmn to account for the Weber-Fechner law Med(I) denotes median value for entire image, Med(Imn) is median value for a current nm-th block of size 8х8 pixels, β is a stabilizing non-negative constant (to be optimized). Nikolay Ponomarenko

Distortion types and considered subsets of TID2008 № Type of distortion Noise Noise2 Noise3 Safe Hard Simple JPEG Exotic Exotic2 Exotic3 Actual Full 1 Additive Gaussian noise + - 2 Different additive noise in color components 3 Spatially correlated noise 4 Masked noise 5 High frequency noise 6 Impulse noise 7 Quantization noise 8 Gaussian blur 9 Image denoising 10 JPEG compression 11 JPEG2000 compression 12 JPEG transmission errors 13 JPEG2000 transmission errors 14 Non eccentricity pattern noise 15 Local block-wise distortions of different intensity 16 Mean shift (intensity shift) 17 Contrast change Karen Egiazarian 22/01/2013 22/04/2007

Optimization of the metric PSNR-HVS-MW SROCC values for different β Metrics Noise1 Noise2 Noise3 Safe Hard Simple JPEG Exotic Exotic2 Exotic3 Actual Full MSSIM 0. 813 0.850 0.830 0.849 0. 874 0.898 0.957 0. 728 0.811 0.673 0.868 0.853 PSNRHVSM 0.918 0.930 0.922 0.936 0.783 0.942 0.971 0.274 0.287 0.518 0.929 0.559 PSNR-HVS-MW calculated for different β 0.4 0.9216 0.9333 0.9224 0.9390 0.7956 0.9449 0.9694 0.2450 0.2839 0.6752 0.9303 0.5663 0.6 0.9222 0.9340 0.9226 0.9396 0.7953 0.9459 0.9702 0.2476 0.2866 0.6760 0.9313 0.5668 0.8 0.9220 0.9341 0.9221 0.7947 0.9460 0.9706 0.2501 0.2885 0.6749 0.9317 0.5670 1 0.9394 0.7942 0.9710 0.2525 0.2897 0.6763 0.9316 1.2 0.9214 0.9213 0.9393 0.7935 0.9709 0.2539 0.2903 0.6766 As an optimization criterion, we have used Spearman rank order correlation coefficient (that characterizes correlation between HVS-metrics and MOS). Optimization was carried out for all distortion types of TID2008, especially for subsets Actual, Noise3, and Safe.    The modified metric with the recommended β = 0.8 outperformed original for most subsets of distortions. SROCC became up to 0.01 larger than SROCC for the metric PSNR-HVS-M and MOS for all types of distortions. Karen Egiazarian 22/04/2007 22/01/2013

Results of analysis for different visual quality metrics The metric MSSIM almost always assigns the best visual quality to images corrupted by multiplicative noise. Values of the metric PSNR-HVS-M for about half of images are practically the same. For others images, the metric assigns the best visual quality to images with additive noise. Nikolay Ponomarenko 22/04/2007

Results of analysis for proposed visual quality metric According to the metric PSNR-HVS-MW, the images with multiplicative noise possess the best visual quality. For the all considered metrics the values of SROCC are close to zero and even less than zero for most HVS-metrics. Existing metrics are practically unable to take into account signal-dependent distortions. HVS-metric SROCC IFC -0.0128 PSNR-HVS-W -0.1867 MSSIM -0.0253 SSIM -0.0367 NQM -0.0269 UQI -0.0319 FSIM 0.0332 VIF -0.0436 PSNR-HVS -0.0801 VIFP -0.0155 PSNR-HVS-M -0.0387 VSNR -0.029 PSNR-HVS-MW -0.1192 WSNR 0.1448 Nikolay Ponomarenko 22/04/2007

Analysis Visual quality (perception) of an image corrupted by signal-sensitive noise depends on image characteristics especially on possible presence of homogeneous or textural fragments, their brightness. The main attention of the experiments’ participants was paid to objects that are mainly homogeneous. If there is a limited number (area) of such objects, other objects and features were analyzed and compared by observers. Noise in textural regions is masked (fully or partially), and textural fragments of distorted images often seem indistinguishable; If a fragment that attracted the main attention is bright, the image corrupted by multiplicative noise is perceived as the worst and vice versa. Participants often evaluate noisy image quality by comparing distortions mainly for objects that attracted attention (e.g., bodies and faces of people). Nikolay Ponomarenko 22/04/2007

Conclusions The modification of the visual quality metric PSNR-HVS-M that uses weights to incorporate the Weber-Fechner law is presented. 25 image groups with Additive, Poisson and Multiplicative noise in each were created. PSNR of images in one group is the same. Unlike PSNR, visual quality of distorted images differ. Estimated image quality depends on brightness of homogeneous fragments and image semantic contents. Results of research show that the problem is not solved for existing visual quality metrics. Images with signal-dependent noise were included and tested in recently created image database TID2013 that consists of 25 types of distortions and 5 levels for each.