1. Motivation from Real-World Applications EE565 Advanced Image Processing Copyright Xin Li 2009-2012 1 Noisy Photos Noisy ultrasound data

2. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 2 Applications (Con’t) Thermal imaging Probe into deep space Neuron imaging MRI imaging

3. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 3 Problem Formulation: Image Denoising What is noise? – How to mathematically model noise? – Why do we use additive white Gaussian noise? Deterministic view (this week) – Equivalence between Gaussian filtering and linear/isotropic diffusion – Nonlinear/anisotropic diffusion Statistical view (next week) – History of Wiener filtering – Wavelet-domain adaptive Wiener filtering Local vs. Nonlocal view (think like a physicist) – Equivalence between wavelets and PDE-based models – State-of-the-art: nonlocal image denoising

4. What is Noise? EE565 Advanced Image Processing Copyright Xin Li 2009-2012 4 “For many years, users of ultrasound systems assigned a diagnostic value to the appearance of speckle, and they assumed it was tissue microstructure.” – T. L. Szabo “Diagnostic Ultrasound Imaging” pp. 230 ISO 100, f/5.6, 1/350 s ISO 1600, f/5.6, 1/4000 s

5. Speckle Noise EE565 Advanced Image Processing Copyright Xin Li 2009-2012 5 “speckle is an illusion more dependent on the measurement system than the issue itself.” – T. L. Szabo “Diagnostic Ultrasound Imaging” pp. 230-233

6. Mathematical Modeling of Noise Gaussian noise (thermal noise) – Johnson-Nyquist model Random noise (salt-and-pepper noise) – “Dead” pixels in AD conversion or bit errors in transmission Poisson noise (shot noise) – Caused by statistical quantum fluctuations Uniform noise (quantization noise) – Caused by nonlinear quantization operation Signal-dependent noise (film grain) – E.g., multiplicative instead of additive EE565 Advanced Image Processing Copyright Xin Li 2009-2012 6

7. Noise Reduction (Hardware) EE565 Advanced Image Processing Copyright Xin Li 2009-2012 7 Image on the left has exposure time of >10 seconds in low light. The image on the right has adequate lighting and 0.1 second exposure

8. Image Denoising (software) EE565 Advanced Image Processing Copyright Xin Li 2009-2012 8 Filtering: E[X|Y], Y=X+W estimate clean based on noisy Prediction: E[X(n)|X(n-1),…,X(1)] estimate the future based on the past Interpolation: E[X(n)|X(n+1),X(n-1)] estimate unknown based on known Wiener-Kolmogorov theory Norbert Wiener (1894-1964)

9. Mathematical Modeling of Images Deterministic – From calculus to variational calculus 1 – From Fourier transform to wavelet transform 2 – Emphasis: high-level conceptual ideas instead of low-level technical details Statistical/probabilistic – Everything starts from Wiener filtering Local vs. nonlocal – Probe into the fundamental property of images EE565 Advanced Image Processing Copyright Xin Li 2009-2012 9 1 Medical image Analysis (CS591); 2 Wavelets and Filter Bank (EE591)

10. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 10 Image as a Surface 3D visualizationsingle-edge image If image can be viewed as a surface, it is then natural to ask: can we apply geometric tools to process this surface (or its equivalent image signals)?

11. Two Simple Ideas Geometric: from isotropic to anisotropic filtering – Perona and Malik’s idea: nonlinear edge stopping – Easy to implement the idea but remains a mathematical challenge (e.g., convergence proof) Analytical: minimize some objective function that matches signal but not noise – Rudin and Osher’s idea: noise tends to blow up the total variation (TV) of a signal – Easy to explain the idea but the rigorous derivation required background of variational calculus EE565 Advanced Image Processing Copyright Xin Li 2009-2012 11

12. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 12 Simplest Case: Laplace Equation Linear Heat Flow Equation: scale A Gaussian filter with zero mean and variance of t Isotropic diffusion:

13. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 13 Example t=0 t=1 t=2

14. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 14 Example (Cont.) t=4t=8t=16

15. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 15 From Isotropic to Anisotropic Gaussian filtering (isotropic diffusion) could remove noise but it would blur images as well Ideally, we want – Filtering (diffusion) within the object boundary – No filtering across the edge orientation How to achieve such “ anisotropic diffusion ” ? – Recall what you have learned about edge detection.

16. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 16 Perona-Malik ’ s Idea Isotropic diffusion: edge stopping function

17. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 17 Pursuit of Appropriate g Define 1D case: Encourage diffusion: Discourage diffusion: Edge slope decreases Edge slope increases

18. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 18 Choices of Edge-Stopping Function K Choice-I Choice-II

19. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 19 Discrete Implementation

20. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 20 Numerical Examples 100 110 100 200 100

21. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 21 Scale-space with Anisotropic Diffusion original P-M filter (K=16,100 iterations)

22. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 22 P-M Filter for Image Denoising Noisy image (PSNR=28.13dB) P-M filtered image (PSNR=29.83dB)

23. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 23 What is Total Variation? Key idea: it is L 1 instead of L 2 norm (minimizing L 2 will not preserve edges) 050100150200250300 0 20 40 60 80 100 120 140 160 180 200 Clean (TV small) noisy (TV large)

24. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 24 Variational Formulation Restored image noisy image Total variation (TV) Such that clean image

25. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 25 How to obtain the corresponding PDE? Euler-Lagrangian Equation Discrete implementation is referred to the posted paper (TV_denoising1992.pdf)

26. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 26 Variational Interpretation of PM Diffusion*

27. TV Diffusion for Image Denoising EE565 Advanced Image Processing Copyright Xin Li 2009-2012 27 Noisy image (PSNR=28.13dB) TV filtered image (PSNR=30.42dB)

28. Good or Bad Models EE565 Advanced Image Processing Copyright Xin Li 2009-2012 28 edgetexture

29. EE565 Advanced Image Processing Copyright Xin Li 2009-2012 29 Experimental Justification Noisy image (PSNR=28.13dB) TV filtered image (PSNR=22.43dB)

30. Summary of PDE-based Denoising Think of image as a 3D surface: a mapping from domain (x,y) to range u(x,y) Geometry-driven ideas – Generalization of 1D gradient: total-variation (closely related to surface area) – Generalization of isotropic diffusion (linear filtering): anisotropic diffusion (nonlinear filtering) Discrete implementation: finite—difference method Good vs. bad models – “All models are wrong; some of them are useful” – George Box EE565 Advanced Image Processing Copyright Xin Li 2009-2012 30