Motivation from Real-World Applications EE565 Advanced Image Processing Copyright Xin Li Noisy Photos Noisy ultrasound data

EE565 Advanced Image Processing Copyright Xin Li Applications (Con’t) Thermal imaging Probe into deep space Neuron imaging MRI imaging

EE565 Advanced Image Processing Copyright Xin Li Problem Formulation: Image Denoising What is noise? – How to mathematically model noise? – Why do we use additive white Gaussian noise? Deterministic view (this week) – Equivalence between Gaussian filtering and linear/isotropic diffusion – Nonlinear/anisotropic diffusion Statistical view (next week) – History of Wiener filtering – Wavelet-domain adaptive Wiener filtering Local vs. Nonlocal view (think like a physicist) – Equivalence between wavelets and PDE-based models – State-of-the-art: nonlocal image denoising

What is Noise? EE565 Advanced Image Processing Copyright Xin Li “For many years, users of ultrasound systems assigned a diagnostic value to the appearance of speckle, and they assumed it was tissue microstructure.” – T. L. Szabo “Diagnostic Ultrasound Imaging” pp. 230 ISO 100, f/5.6, 1/350 s ISO 1600, f/5.6, 1/4000 s

Speckle Noise EE565 Advanced Image Processing Copyright Xin Li “speckle is an illusion more dependent on the measurement system than the issue itself.” – T. L. Szabo “Diagnostic Ultrasound Imaging” pp

Mathematical Modeling of Noise Gaussian noise (thermal noise) – Johnson-Nyquist model Random noise (salt-and-pepper noise) – “Dead” pixels in AD conversion or bit errors in transmission Poisson noise (shot noise) – Caused by statistical quantum fluctuations Uniform noise (quantization noise) – Caused by nonlinear quantization operation Signal-dependent noise (film grain) – E.g., multiplicative instead of additive EE565 Advanced Image Processing Copyright Xin Li

Noise Reduction (Hardware) EE565 Advanced Image Processing Copyright Xin Li Image on the left has exposure time of >10 seconds in low light. The image on the right has adequate lighting and 0.1 second exposure

Image Denoising (software) EE565 Advanced Image Processing Copyright Xin Li Filtering: E[X|Y], Y=X+W estimate clean based on noisy Prediction: E[X(n)|X(n-1),…,X(1)] estimate the future based on the past Interpolation: E[X(n)|X(n+1),X(n-1)] estimate unknown based on known Wiener-Kolmogorov theory Norbert Wiener ( )

Mathematical Modeling of Images Deterministic – From calculus to variational calculus 1 – From Fourier transform to wavelet transform 2 – Emphasis: high-level conceptual ideas instead of low-level technical details Statistical/probabilistic – Everything starts from Wiener filtering Local vs. nonlocal – Probe into the fundamental property of images EE565 Advanced Image Processing Copyright Xin Li Medical image Analysis (CS591); 2 Wavelets and Filter Bank (EE591)

EE565 Advanced Image Processing Copyright Xin Li Image as a Surface 3D visualizationsingle-edge image If image can be viewed as a surface, it is then natural to ask: can we apply geometric tools to process this surface (or its equivalent image signals)?

Two Simple Ideas Geometric: from isotropic to anisotropic filtering – Perona and Malik’s idea: nonlinear edge stopping – Easy to implement the idea but remains a mathematical challenge (e.g., convergence proof) Analytical: minimize some objective function that matches signal but not noise – Rudin and Osher’s idea: noise tends to blow up the total variation (TV) of a signal – Easy to explain the idea but the rigorous derivation required background of variational calculus EE565 Advanced Image Processing Copyright Xin Li

EE565 Advanced Image Processing Copyright Xin Li Simplest Case: Laplace Equation Linear Heat Flow Equation: scale A Gaussian filter with zero mean and variance of t Isotropic diffusion:

EE565 Advanced Image Processing Copyright Xin Li Example t=0 t=1 t=2

EE565 Advanced Image Processing Copyright Xin Li Example (Cont.) t=4t=8t=16

EE565 Advanced Image Processing Copyright Xin Li From Isotropic to Anisotropic Gaussian filtering (isotropic diffusion) could remove noise but it would blur images as well Ideally, we want – Filtering (diffusion) within the object boundary – No filtering across the edge orientation How to achieve such “ anisotropic diffusion ” ? – Recall what you have learned about edge detection.

EE565 Advanced Image Processing Copyright Xin Li Perona-Malik ’ s Idea Isotropic diffusion: edge stopping function

EE565 Advanced Image Processing Copyright Xin Li Pursuit of Appropriate g Define 1D case: Encourage diffusion: Discourage diffusion: Edge slope decreases Edge slope increases

EE565 Advanced Image Processing Copyright Xin Li Choices of Edge-Stopping Function K Choice-I Choice-II

EE565 Advanced Image Processing Copyright Xin Li Discrete Implementation

EE565 Advanced Image Processing Copyright Xin Li Numerical Examples

EE565 Advanced Image Processing Copyright Xin Li Scale-space with Anisotropic Diffusion original P-M filter (K=16,100 iterations)

EE565 Advanced Image Processing Copyright Xin Li P-M Filter for Image Denoising Noisy image (PSNR=28.13dB) P-M filtered image (PSNR=29.83dB)

EE565 Advanced Image Processing Copyright Xin Li What is Total Variation? Key idea: it is L 1 instead of L 2 norm (minimizing L 2 will not preserve edges) Clean (TV small) noisy (TV large)

EE565 Advanced Image Processing Copyright Xin Li Variational Formulation Restored image noisy image Total variation (TV) Such that clean image

EE565 Advanced Image Processing Copyright Xin Li How to obtain the corresponding PDE? Euler-Lagrangian Equation Discrete implementation is referred to the posted paper (TV_denoising1992.pdf)

EE565 Advanced Image Processing Copyright Xin Li Variational Interpretation of PM Diffusion*

TV Diffusion for Image Denoising EE565 Advanced Image Processing Copyright Xin Li Noisy image (PSNR=28.13dB) TV filtered image (PSNR=30.42dB)

Good or Bad Models EE565 Advanced Image Processing Copyright Xin Li edgetexture

EE565 Advanced Image Processing Copyright Xin Li Experimental Justification Noisy image (PSNR=28.13dB) TV filtered image (PSNR=22.43dB)

Summary of PDE-based Denoising Think of image as a 3D surface: a mapping from domain (x,y) to range u(x,y) Geometry-driven ideas – Generalization of 1D gradient: total-variation (closely related to surface area) – Generalization of isotropic diffusion (linear filtering): anisotropic diffusion (nonlinear filtering) Discrete implementation: finite—difference method Good vs. bad models – “All models are wrong; some of them are useful” – George Box EE565 Advanced Image Processing Copyright Xin Li