1 Vladimir Lukin 31/08/2009 Processing of Images Based on Blind Evaluation of Noise Type and Characteristics Processing of Images Based on Blind Evaluation of Noise Type and Characteristics Vladimir V. Lukin a, Sergey K. Abramov a, Nikolay N. Ponomarenko a, Mikhail L. Uss a,b, Benoit Vozel b, Kacem Chehdi b, Jaakko Astola c a National Aerospace University, 61070, Kharkov, Ukraine; b University of Rennes I, Lannion cedex, BP 80518, France; c Tampere University of Technology, Institute of Signal Processing, P.O. Box-553, FIN-33101, Tampere, Finland

2 Vladimir Lukin 31/08/2009 Contents Introduction A priori Knowledge on Noise Statistics, Its Use in Image Processing Examples of False Preliminary Assumptions on Noise Type General Aspects of Image Processing Methods for Blind Determination of Noise Type Methods for Blind Determination of Noise Statistical Characteristics Edge Detection Example Requirements to Methods of Blind Determination of Noise Statistical Characteristics Image Filtering Examples Noisy Image Compression Automatic Procedure for Lossy Compression Accuracy of Blind Estimation Methods Conclusions

3 Vladimir Lukin 31/08/2009 Introduction Applications: Systems for single- or multi-channel remote sensing (RS) data processing Goal: Carrying out analysis of mixed noise influence for three particular operations of image processing: edge detection, filtering and compression of noisy images Additive noise Influence on: edge detection, filtering and compression Reasons: transmitting and encoding/decoding errors Multiplicative noise Reason: circuit and atmospheric noise Blind noise parameters evaluation → inaccurate estimates may occur Impulse noise Reasons: insufficient observation interval, speckle effects Image processing Blind noise type determination → false decisions may occur RS image

4 Vladimir Lukin 31/08/2009 A priori Knowledge on Noise Statistics, Its Use in Image Processing Characteristics of noise: 1.Noise type (additive, signal-dependent, multiplicative, impulse, mixed, etc.); 2.PDF, variance, relative variance, probability of impulse noise, etc.; 3.Is noise i.i.d. or spatially correlated; 2D ACF or spectrum in the latter case. Examples: 1.Noise PDF is needed to design optimal edge detectors and to set thresholds for most known edge detectors; 2.Noise statistics is used in image filtering (wavelets, DCT, sigma, Lee, Frost, Kuan) and reconstruction (Katsaggellos et al); 3.Lossy compression of noisy images. Peculiarities: There are no universal methods for coping well with any type and level of noise. Specialized methods that exploit available information on noise type and its characteristics in full extent as a rule provide the most efficient processing.

5 Vladimir Lukin 31/08/2009 A priori Knowledge on Noise Statistics, Its Use in Image Processing Statement: Information on noise type and statistics is to be available or it should be retrieved for data (image) at hand. Otherwise, a used method has to exhibit certain robustness, i.e. ability to perform well enough in conditions of some unknown parameters and characteristics of noise. Fortunately, some methods for blind noise model identification and estimation of its parameters exist and they are rapidly developing*. Goal: Carrying out analysis of these methods for three particular operations of image processing, namely, edge detection, filtering and compression of noisy images. * B. Vozel, S. Abramov, K. Chehdi, V. Lukin, N. Ponomarenko, M. Uss, Blind methods for noise evaluation in multi-component images, book chapter in “Multivariate image processing”, ISTE Ltd, France, 2009 (in press).

6 Vladimir Lukin 31/08/2009 Examples of False Preliminary Assumptions on Noise Type: Optical Images The noisy (dominant Poisson) test image (left) and this image after Gamma-correction (γ = 0.6) (right) Scatter-plots of local estimates of variance for original noisy (left) and corrected images (right)

7 Vladimir Lukin 31/08/2009 Examples of False Preliminary Assumptions on Noise Type: Radar Images A real life K-band SLAR image Scatter-plot of local estimation pairs (variance and squared mean) with two approximation lines fitted by expert (green line) and automatically (red line) Other examples: a model of fully developed speckle, noise in ultrasound images, noise in sub-band images of hyperspectral remote sensing data.

8 Vladimir Lukin 31/08/2009 General Aspects of Image Processing 1.Noise is a phenomenon inherent for any imaging and RS system; 2.Noise presence is often a dominant factor that prevents solving final tasks for which obtained data are intended for; 3.These tasks could be: object detection and localization, sensed terrain classification, segmentation, edge detection, image filtering, segmentation, compression, classification, etc. 4.There are some methods of image processing that do not exploit information on noise type and characteristics (like standard median filter) but methods that use a priori information on noise type and characteristics are able to perform better; 5.There is a lot of different types of noise and their combinations.

9 Vladimir Lukin 31/08/2009 Methods for Blind Determination of Noise Type An automatic system to identify the nature of the degradation affecting an image and select the appropriate algorithm of its processing has been designed*. The system is intended for identifying four sources of degradation made of three different noises: additive, multiplicative and impulse noise and a defocusing blur. Thirteen observation models have been deduced from these four sources: each noise type separately and different possible combinations. The system has been tested for different simulated data and provided appropriately high probability of correct identification of noise/distortion type. * Carton-Vandecandelaere, M.-P., Vozel, B., Klaine, L., Chehdi, K., Application to Multispectral Images of a Blind Identification System for Blur, Additive, Multiplicative and Impulse Noises. Proceedings of EUSIPCO, III, 2002, pp

10 Vladimir Lukin 31/08/2009 Methods for Blind Determination of Noise Statistical Characteristics There are methods for blind estimation of:  additive noise variance*;  mixed noise parameters***. * V. Lukin, S. Abramov, B. Vozel, K. Chehdi, J. Astola, Segmentation-based method for blind evaluation of noise variance in images, SPIE Journal on Applied Remote Sensing, Vol. 2, Aug. 2008, 15 p. ** Lukin V.V., Abramov S.K., Ponomarenko N.N., Vozel B., Chehdi K., Methods for blind evaluation of noise variance in multichannel optical and radar images, Telecommunications and Radio Engineering, Vol. 65 (6), 2006, pp *** A. Foi, Pointwise Shape-Adaptive DCT Image Filtering and Signal-Dependent Noise Estimation: Thesis for the degree of Doctor of Technology, Tampere University of Technology, Tampere, Finland, Dec They are based on different principles:  Curve fitting to scatter-plots;  Analysis of local estimate histograms;  Data analysis in spectral domain;  Bit-plane analysis, etc.  multiplicative noise variance**;

11 Vladimir Lukin 31/08/2009 Edge Detection Example Noise free (a) and noisy (mixed multiplicative and impulse noise, ) (b) test images; edge maps obtained by Sobel (c) and normalized quasirange (d) edge detectors Principle: For most of edge detectors, a local parameter is calculated for each pixel using image values in a given pixel and its neighbors. Then this local parameter is compared to a preset threshold that takes into account noise type and statistics. аbcdаbcd Example:

12 Vladimir Lukin 31/08/2009 Edge Detection Requirement to accuracy: It is desirable to provide within the limits from till. Procedure:  to estimate local mean;  to calculate ;  to determine the corresponding local threshold;  to obtain a detector output;  to compare it to the local threshold;  to undertake a decision. Signal-dependent noise case: If a dependence of local variance on local mean is known in advance or pre-estimated, then a threshold for an edge detector should be set locally as or

13 Vladimir Lukin 31/08/2009 Requirements to Methods of Blind Determination of Noise Statistical Characteristics 1.To produce unbiased estimates; 2.To provide appropriately accurate estimates (characterized by small variance of noise parameter estimation); 3.To be applicable to images of different content and to noise of different level and with different spatial characteristics (a wide range of noise parameters’ variation can be observed in practice); 4.To be fast enough. Note: if an image is multichannel, processing is component-wise and it can be parallel. A question is what are appropriately accurate estimates?

14 Vladimir Lukin 31/08/2009 Image Filtering Examples: Test Images and Quantitative Criteria Goal: Consider how filtering efficiency depends upon accuracy of additive noise variance estimation. Test images:Quantitative criteria:  An error of additive noise variance estimation:  Variation of filtering efficiency Where is MSE provided under condition that is set as the parameter of a used filter. Test01Test02

15 Vladimir Lukin 31/08/2009 Image Filtering Examples: Modified Sigma Filter The plots vs for the test images for 100 and 225 Preliminary conclusion: For the image Test01, smaller output MSE can be reached if when better edge/detail/texture preservation is provided. The opposite situation is observed for the image Test02. If, smaller output MSE is produced. Thus, does not necessarily lead to the best efficiency of noise suppression (according to output MSE criterion). If is within the limits from 0.8 to 1.2, then it is almost guaranteed that one is in the neighborhood of minimal output MSE. This is observed for other filters as well as for multiplicative noise.

The noise variance was estimated in automatic manner robust to the presence of impulses. The resulting estimate is Vladimir Lukin 31/08/2009 Image Filtering Examples: Blind Noise Type and Variance Retrieval The original NOAA image (a), filtered image (b), and the obtained impulse noise map (c) Real-life image: a b c Identification result: Image is identified as corrupted by mixed additive and impulse noise. Impulse noise is specific, spatially correlated, where several neighboring image pixels in rows are corrupted. Estimation result:

17 Vladimir Lukin 31/08/2009 Image Filtering Examples: Blind Noise ACF Retrieval Original SAR image (left) and the obtained filtered image (right) * V. Lukin, N. Ponomarenko, K. Egiazarian, J. Astola, Adaptive DCT-based filtering of images corrupted by spatially correlated noise, Proc. SPIE Conference Image Processing: Algorithms and Systems VI, Vol. 6812, 12 p., If fluctuative noise type is determined correctly and its variance is estimated with high enough accuracy, it becomes possible to blindly evaluate spatial correlation properties of noise as well*. These estimates can be further exploited in DCT based filtering. Taking into account noise spatial spectrum leads to PSNR increasing by dB.

18 Vladimir Lukin 31/08/2009 Image Filtering Examples: Fully Automatic Procedure Stages: 1.Blind determination of noise type, additive or multiplicative (if there is no a priori information); 2.Estimation of its statistical characteristics (variance); 3.Estimation of spatial correlation properties of noise; 4.Image filtering with taking into account the obtained estimates of noise characteristics. Features: This procedure is also applicable if both additive and multiplicative noise components have been identified in a given image. Note that DCT based filters can be easily modified to mixed additive and multiplicative or other signal-dependent noise. Similar modifications are not a problem for the family of sigma filters and hard-switching locally adaptive filters.

19 Vladimir Lukin 31/08/2009 Image Filtering Examples: 3D Processing of Multichannel Images ab (PSNR= dB) c (PSNR= dB)d (PSNR= dB) Automatic procedures based on blind estimation described above can be used for both component-wise and 3D processing of multichannel images. Improvement of PSNR due to 3D processing is 0.6 dB in comparison to component-wise DCT based processing. The noise-free test image Goldhill (a), noisy image ( ) (b), intermediate output image (c), resulting output image (d) Preliminary conclusion:

Noisy Image Compression: Quantitative Criteria 20 Vladimir Lukin 31/08/2009 The standard measures to characterize a compressed image quality -, where is the decompressed image; It is more reasonable to characterize a compressed image quality by quantitative measures calculated with respect to the corresponding noise-free image ( MSE nf, PSNR nf ) rather than to the original noisy one ( MSE or, PSNR or ). - - for 8 bits image representation. Alternative measures to characterize a compressed image quality -, where is the noise free image; -.

Noisy Image Compression: Optimal Operation Point 21 Vladimir Lukin 31/08/2009 Dependences MSE nf (QS n ) for the noisy test gray-scale image Lena for different additive noise levels Optimal operation point (OOP): The argument of the curves MSE nf (CR), MSE nf (bpp) or MSE nf (QS) for which these curves reach theirs minima have been called optimal operation point (OOP): CR OOP, bpp OOP or QS OOP.. OOP is observed and commonly occurs to be more “obvious” for less complex content images and/or for rather intensive noise. Main idea: It is worth compressing a noisy image in the neighborhood of OOP. Main problem: In practice, noise-free image is not at disposal. σ 2 n = 400σ 2 n = 100σ 2 n = 50

Noisy Image Compression: Peculiarities 22 Vladimir Lukin 31/08/2009 Why lossy (not lossless) compression? 1)Lossy compression is able to provide considerably larger CRs (compared to lossless coding) without degrading image resolution and introducing disturbing artefacts; 2)A positive effect of image filtering can be observed due to lossy compression if introduced losses mainly relate to noise removal and useful image content is preserved. The RS (Helsinki region) image corrupted by additive Gaussian noise with σ 2 n = 100 The decoded lossy compressed image (bpp = 0.75)

Noisy Image Compression: Real-life Test Images 23 Vladimir Lukin 31/08/2009 The compressed image (HBC strategy, AGU coder with QS A =35) The automatic procedure for attaining OOP have been also designed for pure multiplicative and Poisson noise. The corresponding homomorphic transform is to be applied (for making noise additive) before compression. Inverse homomorphic transform is to be carried out after decompression. The noisy real-life test image Frisco

24 Vladimir Lukin 31/08/2009 Automatic Procedure for Lossy Compression Stages: 1.Blind determination of noise type: additive, Poisson or multiplicative; 2.Estimation of noise statistical characteristics; 3.Carrying out proper forward homomorphic transform if noise is Poisson or multiplicative; calculation or estimation of variance after homomorphic transform (if needed); 4.Carrying out lossy compression with providing OOP by either setting QS according to above recommendations or by finding proper bpp (for JPEG2000 or SPIHT coders, see details in the paper*). * N. Ponomarenko, V. Lukin, M. Zriakhov, K. Egiazarian, J. Astola, Lossy compression of images with additive noise, Proceedings of International Conference on Advanced Concepts for Intelligent Vision Systems, Antwerpen, Belgium, September, 2005, pp Note that are to be within the limits from till..

Accuracy of Blind Estimation Methods: TID2008 (Tampere Image Database) 25 Vladimir Lukin 31/08/2009 Test noise-free color images of TID2008 (

26 Vladimir Lukin 31/08/2009 Accuracy of Blind Estimation Methods: Spatial Domain Noise variance estimates obtained by the methods* (left) and** (right) with nonoverlapping 5x5 pixels block size for red, green and blue components for the test image set corrupted by spatially uncorrelated additive noise with =65 * Lukin V.V., Abramov S.K., Vozel B., Chehdi K. Improved minimal inter-quantile distance method for blind estimation of noise variance in images, Proceedings of SPIE/EUROPTO Symposium on Satellite Remote Sensing, Florence, Italy, Sept 2007, SPIE Vol. 6748, 12 p. ** V. Lukin, S. Abramov, B. Vozel, K. Chehdi, J. Astola, Segmentation-based method for blind evaluation of noise variance in images, SPIE Journal on Applied Remote Sensing, Vol. 2, Aug. 2008, 15 p. (open access paper)

27 Vladimir Lukin 31/08/2009 Accuracy of Blind Estimation Methods: Wavelet Domain Noise variance estimates obtained by the method* for red, green and blue components for the test image set corrupted by spatially uncorrelated (left) and spatially correlated (right) additive noise with = 65 * L. Sendur, I.W. Selesnick, Bivariate shrinkage with local variance estimation, IEEE Signal Processing Letters, Vol. 9, No 12, 2002, pp (see also Be careful with spatially correlated noise !!!

28 Vladimir Lukin 31/08/2009 Accuracy of Blind Estimation Methods: Spatial Domain & Spatially Correletad Noise Noise variance estimates obtained by the method* with nonoverlapping 5x5 (left) and 7x7 (right) pixels block size for red, green and blue components for the test image set corrupted by spatially correlated additive noise with =65 * Lukin V.V., Abramov S.K., Vozel B., Chehdi K. Improved minimal inter-quantile distance method for blind estimation of noise variance in images, Proceedings of SPIE/EUROPTO Symposium on Satellite Remote Sensing, Florence, Italy, Sept 2007, SPIE Vol. 6748, 12 p. Summary: It is better using 7x7 blocks to have less problems with (possibly) spatiall correlated noise.

29 Vladimir Lukin 31/08/2009 Conclusions 1.It is possible to design multi-stage automatic procedures of image processing based on blind determination of noise type and estimation of its characteristics. 2.The accuracy of such estimates is commonly enough for carrying out efficient edge detection, filtering, and lossy compression in automatic manner. 3.Currently only quite simple situations like identified pure additive or pure multiplicative noise are studied well enough; design and performance analysis for other noise environments are far from completeness. 4.Special attention should be paid to multichannel and hyperspectral images for which, on one hand, necessity to make processing automatic is the most actual and, on the other hand, the situation with such design is the most complex.