Computer Vision - Restoration Hanyang University Jong-Il Park

Department of Computer Science and Engineering, Hanyang University Restoration vs. Enhancement Restoration Objective process A priori knowledge on degradation model Modeling the degradation and applying the inverse process to recover the original To improve an image in some predefined sense Enhancement Subjective process

Department of Computer Science and Engineering, Hanyang University Restoration process

Department of Computer Science and Engineering, Hanyang University Noise models Assume noise is independent of spatial coordinates and it is uncorrelated w.r.t. the image. Gaussian: electronic circuit noise, sensor noise Rayleigh: range images Exponential and gamma: laser images impulse(salt-and-pepper): faulty switching

Department of Computer Science and Engineering, Hanyang University Eg. Sample noisy images

Department of Computer Science and Engineering, Hanyang University Eg. Sample noisy images(cont.)

Department of Computer Science and Engineering, Hanyang University Periodic noise Spatially dependent noise Periodic noise can be reduced significantly via frequency domain filtering

Department of Computer Science and Engineering, Hanyang University Estimation of noise parameters PDF from small patches

Department of Computer Science and Engineering, Hanyang University When the only degradation is noise Periodic noise  subtraction gives a good result Random noise  mean filter, order-statistics filter,…

Department of Computer Science and Engineering, Hanyang University Mean filters Arithmetic mean filters  For Gaussian or uniform noise Geometric mean filters  For Gaussian or uniform noise Harmonic mean filters  Work well for salt noise but fail for pepper noise Contraharmonic mean filters  Suited for impulse noise but require identification(salt or pepper)

Department of Computer Science and Engineering, Hanyang University Arithmetic & Geometric mean filter

Department of Computer Science and Engineering, Hanyang University Contraharmonic filters Q<0 : eliminates salt noise Q=-1  harmonic mean filter Q=0 : arithmetic mean filter Q>0: eliminates pepper noise

Department of Computer Science and Engineering, Hanyang University Eg. Contraharmonic filters

Department of Computer Science and Engineering, Hanyang University Wrong sign in contraharmonic filters Disaster!

Department of Computer Science and Engineering, Hanyang University Order-Statistics filters Median filter Max filter Min filter Midpoint filter Alpha-trimmed mean filter

Department of Computer Science and Engineering, Hanyang University Median filters 3x3 median median median blurred

Department of Computer Science and Engineering, Hanyang University Max and Min filter Max filter Min filter Removes pepper noise Removes dark pixels Removes salt noise Removes light pixels Makes dark objects larger

Department of Computer Science and Engineering, Hanyang University Eg. Comparison (a) Additive uniform noise (b) (a)+additive S&P 5x5 arithmetic mean 5x5 geometric mean 5x5 median5x5 alpha-trimmed Mean(d=5)

Department of Computer Science and Engineering, Hanyang University Adaptive filters Behavior changes locally based on statistical characteristics of local support Simple adaptive filter based on mean and variance 1. If global_var is zero, then f(x,y)=g(x,y) 2. If local_var>global_var, then f(x,y)=g(x,y) (high local var  edge  should be preserved) 3. If local_var==global_var, then arithmetic mean filtering

Department of Computer Science and Engineering, Hanyang University Eg. Adaptive filter

Department of Computer Science and Engineering, Hanyang University Adaptive median filter Cope with impulse noise with large probability Preserve detail while smoothing non-impulse noise Level A:A1=z med -z min A2=z med -z max If A1>0 AND A2<0, go to level B Else increase the window size If window size<=S max repeat level A Else output z xy Level B:B1=z xy -z min B2=z xy -z max If B1>0 AND B2<0, output z xy Else output z med Algorithm

Department of Computer Science and Engineering, Hanyang University Eg. Adaptive median filter medianadaptive median

Department of Computer Science and Engineering, Hanyang University Periodic noise reduction By frequency domain filtering Band reject filter

Department of Computer Science and Engineering, Hanyang University Eg. Periodic noise reduction

Department of Computer Science and Engineering, Hanyang University Noise extraction By bandpass filter Help understanding noise pattern

Department of Computer Science and Engineering, Hanyang University Notch filters

Department of Computer Science and Engineering, Hanyang University Eg. Notch filtering Removing sensor scan- line patterns

Department of Computer Science and Engineering, Hanyang University Optimum notch filtering First isolating the principal contributions of the interference pattern Then subtracting weighted portion of the pattern from the corrupted image

Department of Computer Science and Engineering, Hanyang University Eg. Periodic interference(1/3) Noisy image

Department of Computer Science and Engineering, Hanyang University Eg. Periodic interference(2/3) Extraction of noise interference pattern

Department of Computer Science and Engineering, Hanyang University Eg. Periodic interference(3/3) Restored image by subtracting weighted portion of periodic interference (Refer to the derivation of weights in pp )

Department of Computer Science and Engineering, Hanyang University Linear, Position-Invariant Degradation

Department of Computer Science and Engineering, Hanyang University Degradation knowledge Degradation knowledge about 1. A priori (known) 2. A posteriori (unknown)  blind restoration or blind deconvolution Restoration: determine the original image, given the observed image and knowledge about the degradation (H).

Department of Computer Science and Engineering, Hanyang University Fundamental issue Restoration problem  restoration is to find, such that but, 1. does not exist: singular 2. may exist, but not be unique: ill-conditioned 3. may exist and unique, but there exists, which can be made arbitrarily small, such that which is not negligible Image restoration is ill-conditioned at best and  Image restoration is ill-conditioned at best and singular at worst singular at worst

Department of Computer Science and Engineering, Hanyang University Estimation of degradation function Approaches  Observation  Experimentation  Mathematical modeling

Department of Computer Science and Engineering, Hanyang University Estimation by observation Looking at a small section of the image containing simple structures and then obtaining degradation function Observed sub-image: Estimate of original image:

Department of Computer Science and Engineering, Hanyang University Estimation by experimentation Possible only if equipment similar to the equipment used to acquire the degraded images is available Eg. Use an impulse

Department of Computer Science and Engineering, Hanyang University Estimation by modeling Based on either physical characteristics or basic principles Eg.1. Physical characteristics: atmospheric turbulence Eg.2. Math derivation: motion blur  Starting from  After some manipulation(p.259)  Setting the motion model, we obtain the degradation func.

Department of Computer Science and Engineering, Hanyang University Eg.1. Physical model Atmospheric turbulence

Department of Computer Science and Engineering, Hanyang University Eg.2. Math modeling Motion blur

Department of Computer Science and Engineering, Hanyang University Restoration methods Inverse filtering Wiener filtering Constrained least square filtering Geometric mean filtering Etc..

Department of Computer Science and Engineering, Hanyang University Inverse filtering Poor performance! Very sensitive to noise Noise amplification when H(u,v) is small

Department of Computer Science and Engineering, Hanyang University Eg. Inverse filtering

Department of Computer Science and Engineering, Hanyang University Minimum mean-square error filter Necessary to handle noise explicitly  Statistical characteristics of noise should be incorporated into the restoration process MMSE filter  To find an estimate of the uncorrupted image such that the mean square error between them is minimized:  Assume:  the noise and the image are uncorrelated  The one or the other has zero mean  The gray levels in the estimate are a linear function of the levels in the degraded image  Derivation: Homework

Department of Computer Science and Engineering, Hanyang University MMSE filter (cont.) Frequency domain expression: Approximation of the Wiener filter Wiener filter PS of noisePS of image f

Department of Computer Science and Engineering, Hanyang University Eg. Wiener filtering Using the approximation K is chosen interactively

Department of Computer Science and Engineering, Hanyang University Eg. Restoration by Wiener filter motion blur Severe noise Moderate noise Negligible noise

Department of Computer Science and Engineering, Hanyang University Constrained Least Square Filtering Difficulty in Wiener filter  The power spectra of the undegraded image and noise must be known  Minimization in a statistical sense The constrained LS filtering  requires knowledge of  Mean of the noise  Variance of the noise  Optimal result for each image

Department of Computer Science and Engineering, Hanyang University Vector-matrix form of convolution g: MN-vector (lexicographical order of an image) f: MN-vector H: MNxMN matrix

Department of Computer Science and Engineering, Hanyang University Formulation: Constrained LS filter To find the minimum of a criterion function C defined as subject to the constraint where is the Euclidean vector norm

Department of Computer Science and Engineering, Hanyang University Freq. Domain Sol. : adjustable parameter : Fourier transform of the Laplacian operator

Department of Computer Science and Engineering, Hanyang University Eg. Constrained LS filter Significant improvement over Wiener filter

Department of Computer Science and Engineering, Hanyang University Procedure for computing Define a residual vector Adjust so that  Calculation In general, automatically determined restoration filter yields inferior results to manual adjustment of filter parameters  iteration