Variational Pairing of Image Segmentation and Blind Restoration Leah Bar Nir Sochen* Nahum Kiryati School of Electrical Engineering *Dept. of Applied Mathematics Tel Aviv University

Segmentation: images meet concepts Borrowed from Georges Koepfler

Segmentation: images meet concepts Borrowed from Georges Koepfler Formalization (Mumford & Shah) min [(fidelity to image) + β (gradients within segments) + α (total edge length)] Segmentation by minimizing a functional

Segmentation: images meet concepts Borrowed from Georges Koepfler Formalization (Mumford & Shah) min [(fidelity to image) + β (gradients within segments) + α (total edge length)] Segmentation by minimizing a functional Calculus of Variations PDE ’ s Numerical Techniques Linear Systems of Equations

Mumford-Shah Segmentation fidelity to imagegradients within segmentstotal edge length Ω: image domain K: edge set f : segmented image g : observed image

Mumford-Shah Segmentation fidelity to imagegradients within segmentstotal edge length Ω: image domain K: edge set f : segmented image g : observed image Problem: Discontinuities in the domains ( Ω/ K, K ) make minimization difficult

Mumford-Shah Segmentation fidelity to imagegradients within segmentstotal edge length Ω: image domain K: edge set f : segmented image g : observed image Problem: Discontinuities in the domains ( Ω/ K, K ) make minimization difficult Solution: Continuous approximation of F(f,K) (Gamma-convergence framework)

Mumford-Shah Segmentation fidelity to imagegradients within segmentstotal edge length Ω: image domain K: edge set f : segmented image g : observed image Problem: Discontinuities in the domains ( Ω/ K, K ) make minimization difficult Solution: Continuous approximation of F(f,K) (Gamma-convergence framework) fidelity to imagegradients in segmentstotal edge length v(x): smooth function v(x)~0 at edges v(x)~1 otherwise (in segments) (Ambrosio & Tortorelli, 1990)

In a blurred image, edges are degraded and segmentation is difficult.

Image Restoration Given the image g and the blur kernel h, restore the original image f. 1. Brute force... ill posed. 2. Tikhonov regularization Minimize... oversmoothing. 3. Total Variation (TV) regularization Minimize... better edge preservation.

Blind Image Restoration Given the image g, restore the original image f (and the blur-kernel h ). - Ill posed (1): sensitivity to small changes in g. - Ill posed (2): maybe the original image was already blurred?

Blind Image Restoration Given the image g, restore the original image f (and the blur-kernel h ). - Ill posed (1): sensitivity to small changes in g. Chan & Wong (1998) Minimize - Ill posed (2): maybe the original image was already blurred? TV-regularization with respect to both the image and the kernel. - The restored image is very sensitive to the recovered kernel - The recovered kernel depends on the contents of the image (bad news)

Chan & Wong - The recovered kernel depends on the contents of the image. source image recovered kernel isotropic blur blind restoration

Chan & Wong (1998) - Performance original blurredrestored isotropic gaussian kernel,  =2.1 recovered kernel - The restored image is very sensitive to the recovered kernel. - The recovered kernel depends on the contents of the image.

In blind image restoration, one can’t get it all. Borrowed from Mickey Mouse (The Sorcerer’s Apprentice)

Vogel & Oman, 1998 You & Kaveh, 1996 Carasso, 2001 Chan & Wong, 1998 Mumford & Shah, 1989 Rudin, Osher & Fatemi, 1992 Chambolle, 1995 Hewer et al, 1998 Kim et al, 2002 Ambrosio & Tortorelli, 1992 Aubert & Kornprobst, 2002 Tikhonov & Arsenin, 1977 Some related work... (Blind) Restoration Segmentation Mathematics, Foundations

The suggested approach Why? What? How? - Segmentation is hard, but easier if the image is sharp - Blind restoration is hard, but easier if the edges are known Blind restoration and segmentation as mutually supporting processes Unified variational framework, iterative algorithm

Combined objective functional Mumford-Shah segmentation + blind restoration Make it work: Use the Γ-convergence approximation Make it work well: Use a parametric blur-kernel Reminder v(x): smooth function v(x)~0 at edges v(x)~1 otherwise (in segments) fidelity, parametric blur gradients in segments “ total edge length ”“ smooth v ” “ wide kernel ”

Minimizing the functional Iterate Minimize with respect to v (segmentation / edge detection) Minimize with respect to f (image restoration) Minimize with respect to σ (blur-kernel recovery)

Iterative Minimization Equations Minimization with respect to v (Euler equation)

Iterative Minimization Equations Minimization with respect to v (Euler equation) Minimization with respect to f (Euler equation)

Iterative Minimization Equations Minimization with respect to v (Euler equation) Minimization with respect to f (Euler equation) Minimization with respect to σ (derivative)

Iterative Minimization Equations Minimization with respect to v (Euler equation) Minimization with respect to f (Euler equation) Minimization with respect to σ (derivative) Calculus of Variations PDE ’ s Numerical Techniques Linear Systems of Equations

Frequently Asked Questions What are the initial values? What is the stopping condition? Does it converge? To a global optimum? We use f=g (output=input), v=1 (no edges) and σ = ε (small blur). We stop when the radius σ of the recovered kernel has converged. Nice theoretical properties Excellent experimental behavior Additional analytic work in progress Typical convergence: σ vs. iteration number

Experimental Results (1): Known Blur Kernel Blurred

Experimental Results (1): Known Blur Kernel Blurred Lucy-Richardson restoration

Experimental Results (1): Known Blur Kernel Blurred Lucy-Richardson restoration Suggested restoration

Experimental Results (1): Known Blur Kernel Blurred Lucy-Richardson restoration Suggested restorationSuggested edges ( v function)

Experimental Results (2): Blind Blurred

Experimental Results (2): Blind BlurredChan-Wong restoration

Experimental Results (2): Blind BlurredChan-Wong restoration Suggested restoration

Experimental Results (2): Blind BlurredChan-Wong restoration Suggested restorationSuggested edges ( v function)

Experimental Results (3): Blind Blurred

Experimental Results (3): Blind Blurred Chan-Wong restoration

Experimental Results (3): Blind Blurred Chan-Wong restoration Suggested restoration

Experimental Results (3): Blind Blurred Chan-Wong restoration Suggested restorationSuggested edges ( v function)

Experimental Results (4): Blind Blurred

Experimental Results (4): Blind Blurred Chan-Wong restoration

Experimental Results (4): Blind Blurred Chan-Wong restoration Suggested restoration

Experimental Results (4): Blind BlurredChan-Wong restoration Suggested restorationSuggested edges ( v function)

Conclusions Image segmentation and (blind) restoration, sont les mots qui vont tres bien ensemble*. Blind restoration is easier if you can use a parametric blur model. *these are words that go together well. The whole is larger than the sum of its parts (in this case).