1. Variational Image Restoration Leah Bar PhD. thesis supervised by: Prof. Nahum Kiryati and Dr. Nir Sochen* School of Electrical Engineering *Department of Applied Mathematics Tel-Aviv University, ISRAEL

2. 2 Inverse problem which has been investigated for more than 40 years. Given the image g and the blur kernel h, restore the original image f. What is image Restoration? Camera out of focus Motion blur Atmospheric turbulence Sensor noise Quantization Image is degraded by deterministic (blur) and random (noise) processes. Blur is assumed as linear shift invariant process with additive noise.

3. 3 Image Restoration - Applications Microscopy

4. 4 Image Restoration - Applications Astronomy

5. 5 Image Restoration - Applications Medical Imaging

6. 6 Frequency domain : Spatial domain: Assuming Gaussian distribution of the noise Bayesian and Variational Viewpoints Maximum Likelihood Variational Noise amplification In high frequencies Ill-Posed Solution Pseudo inverse Filter

7. 7 Bayesian and Variational Regularization Maximum a posteriori prob. MAP Variational (Tikhonov, 1977) Solution – Wiener Filter (over smoothing) smoothness prior

8. 8 Edge Preservation Edges are very important features in image processing, and therefore have to be preserved. Image Deconvolution Image Denoising Preserve Edges observed image - g recovered image - f

9. 9 Total Variation Regularization Rudin, Osher, Fetami 1992 WienerTotal variation

10. 10 Mumford-Shah Segmentation (Mumford and Shah, 1985) gradients within segments total edge length data fidelity Ω: image domain K: edge set f : recovered image g : observed image Canny edgesM-S edgesOriginal Image is modeled as piecewise smooth function separated by edges

11. 11 Deconvolution with Mumford-Shah Regularization gradients within segments total edge length data fidelity M-S functional: difficult to minimize (free-discontinuity problem). Solution is via the  -convergence framework (Ambrosio and Tortorelli 1990) Strategy: approximate the solution by approximation of the problem L. Bar, N. Sochen, N. Kiryati, ECCV 2004

12. 12 F j (u)=sin (ju) u j =1.5  n/j Example:  -convergence A sequence  -converges to if: 1.liminf inequality 2.existence of recovery sequence  -lim(F j )=-1 (De Giorgi, 1979)

13. 13 Fundamental theorem of  -convergence: Suppose that and let a compact set exist such that for all j, then. Moreover if u j is a converging sequence such that then its limit is a minimum point for F.  -convergence

14. Proof: Let satisfy There exists a subsequence converging to some u, such that * This is satisfied for every u and in particular

15. 15 Deconvolution with Mumford-Shah Regularization gradients within segments total edge length data fidelity v(x): smooth function v(x)~0 at edges v(x)~1 otherwise (in segments)

16. 16 Iterate Minimize with respect to v by Euler equation (edge detection) Minimize with respect to f b y Euler equation(image restoration) Deconvolution with Mumford-Shah Regularization

17. 17 Zero padding Convolution Implementation Neumann boundary conditions FFT multiplications

18. 18 Deconvolution with Mumford-Shah Regularization blurred suggested restoration suggested edges ( v )

19. 19 Semi-blind Deconvolution via Mumford-Shah Regularization L. Bar, N. Sochen, N. Kiryati, IEEE Trans. Image Processing, 2006 Blind deconvolution: the blur kernel is unknown Chan and Wong 1998: Suggested: Gaussian kernel parameterized by . - The restored image is very sensitive to the recovered kernel. - The recovered kernel depends on the contents of the image.

20. 20 Semi-blind Deconvolution via Mumford-Shah Regularization blurred suggested methodChan-Wong

21. 21 Image Deblurring in the Presence pf Salt-and-Pepper noise L. Bar, N. Sochen, N. Kiryati, Scale Space, 2005 (best student paper) Special care should be taken in the case of salt-and-pepper noise L 2 fidelity term in not adequate anymore Total Variation

22. 22 Image Deblurring in the Presence pf Salt-and-Pepper noise L. Bar, N. Sochen, N. Kiryati, Scale Space, 2005 (best student paper) Special care should be taken in the case of salt-and-pepper noise L 2 fidelity term in not adequate anymore Sequential approach: Deblurring following median-type filtering-poor 1.Median filter 3x3 window 2.TV restoration 3.Noise remains! 1.Median filter 5x5 window 2.TV restoration 3.Nonlinear distortion!

23. 23 Image Deblurring in the Presence pf Salt-and-Pepper noise Suggested approach: robust L 1 fidelity and Mumford-Shah regularization gradients within segments total edge length data fidelity Iterate Minimize with respect to v by Euler equation (edge detection) Minimize with respect to f b y Euler equation(image restoration)

24. 24 Linearization via fixed point scheme: coefficients in nonlinear terms are lagged by one iteration → linear equation Image Deblurring in the Presence pf Salt-and-Pepper noise Linear operator

25. 25 Results - pill-box kernel (9x9), radius 4, 10% noise suggested 5x5 median + TV3x3 median + TV blurred blurred and noisy

26. 26 Results - pill-box kernel (7x7), radius 3, 1% noise blurred and noisyrecovered

27. 27 blurred and noisyrecovered Results - pill-box kernel (7x7), radius 3, 10% noise

28. 28 blurred and noisyrecovered Results - pill-box kernel (7x7), radius 3, 30% noise

29. 29 What is the theoretical explanation to the simultaneous deblurring and denoising? Is Mumford-Shah regularization better than Total Variation? L. Bar, N. Sochen, N. Kiryati, International Journal of Computer Vision There is discrimination between image and noise edges. Image edges are preserved while impulse noise is removed Theoretical Questions

30. 30 Edge Preservation Relations between: robust statistics anisotropic diffusion line process (half-quadratic) were shown by  Black and Rangarajan, IJCV, 1996  Black, Sapiro, Marimont and Heeger, IEEE T-IP, 1998 robust statistics anisotropic diffusion line process (half quadratic) Perona & Malik, 1987 Geman & Yang, 1993 Charbonnier et al., 1997 Hampel et al., 1986

31. 31 Edge Preservation 1. Robust smoothness Gradient Descent: Influence function-   s   ’(s)=  (s)

32. 32 Edge Preservation 2. Diffusion Isotropic diffusion (heat equation) g is “edge stopping” function Anisotropic diffusion (Perona and Malik, 1987) From robust smoothness point of view Lorentzian

33. 33 Diffusion Illustration Original Isotropic Diffusion Anisotropic Diffusion

34. 34 3. Line-process (Half-Quadratic) (Geman and Yang, 1993) Edge Preservation Dual function b represents edges Penalty function  enforces sparse edges across edges otherwise From robust smoothness point of view

35. 35 Example: Geman-McClure Function Robust Smoothing robust  -function Geman McClure edge stopping function Anisotropic Diffusion edge penalty Line Process (Half-Quadratic)

36. 36 Relation to M-S Terms The Geman-McClure function in half-quadratic form Appears in M-S terms with b = v 2 M-S: extended line process = extended Geman-McClure Edges are forced to be smooth and continuous image edges are preserved

37. 37 Color Deblurring in the Presence of Impulsive Noise L. Bar, A. Brook, N. Sochen, N. Kiryati, VLSM’05 Channels have to be coupled One edge map for all channels

38. 38 Image Restoration in 3D blurred recovered edges

39. 39 Future Work: Space Variant Image Restoration preliminary results

40. 40 Conclusions Novel unified approach to variational segmentation, deblurring and denoising. Mumford-shah regularization reflects the piecewise-smooth model of natural images. Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization is a better edge detector. Restoration outcome is superior to state-of-the-art methods

41. 41 Conclusions Novel unified approach to variational segmentation, deblurring and denoising. Mumford-shah regularization reflects the piecewise-smooth model of natural images. Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization is a better edge detector. Restoration outcome is superior to state-of-the-art methods

42. 42 Conclusions Novel unified approach to variational segmentation, deblurring and denoising. Mumford-shah regularization reflects the piecewise-smooth model of natural images. Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization is a better edge detector. Restoration outcome is superior to state-of-the-art methods

43. 43 Conclusions Novel unified approach to variational segmentation, deblurring and denoising. Mumford-shah regularization reflects the piecewise-smooth model of natural images. Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization is a better edge detector. Restoration outcome is superior to state-of-the-art methods

44. Thank you for your attention