1. Rob Fergus Courant Institute of Mathematical Sciences New York University A Variational Approach to Blind Image Deconvolution

2. Overview Much recent interest in blind deconvolution: Levin ’06, Fergus et al. ’06, Jia et al. ’07, Joshi et al. ’08, Shan et al. ’08, Whyte et al. ’10 This talk: – Discuss Fergus ‘06 algorithm – Try and give some insight into the variational methods used

3. Removing Camera Shake from a Single Photograph Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis and William T. Freeman Massachusetts Institute of Technology and University of Toronto

4. Image formation process =  Blurry image Sharp image Blur kernel Input to algorithm Desired output Convolution operator Model is approximation Assume static scene & constant blur

5. Why such a hard problem? =  Blurry image Sharp image Blur kernel =  = 

6. Statistics of gradients in natural images Histogram of image gradients Characteristic distribution with heavy tails Log # pixels Image gradient is difference between spatially adjacent pixel intensities

7. Blurry images have different statistics Histogram of image gradients Log # pixels

8. Parametric distribution Histogram of image gradients Use parametric model of sharp image statistics Log # pixels

9. Three sources of information 1. Reconstruction constraint: =  Input blurry image Estimated sharp image Estimated blur kernel 2. Image prior: 3. Blur prior: Positive & Sparse Distribution of gradients

10. Three sources of information y = observed image b = blur kernelx = sharp image

11. Posterior Three sources of information y = observed image b = blur kernelx = sharp image

12. Posterior 1. Likelihood (Reconstruction constraint) 2. Image prior 3. Blur prior Three sources of information y = observed image b = blur kernelx = sharp image

13. y = observed image b = blurx = sharp image 1. Likelihood Reconstruction constraint i - pixel index Overview of model 2. Image prior Mixture of Gaussians fit to empirical distribution of image gradients 3. Blur prior Exponential distribution to keep kernel +ve & sparse

14. The obvious thing to do – Combine 3 terms into an objective function – Run conjugate gradient descent – This is Maximum a-Posteriori (MAP) No success! Posterior 1. Likelihood (Reconstruction constraint) 2. Image prior 3. Blur prior

15. Variational Independent Component Analysis Binary images Priors on intensities Small, synthetic blurs Not applicable to natural images Miskin and Mackay, 2000

16. Variational Bayes Variational Bayesian approach Keeps track of uncertainty in estimates of image and blur by using a distribution instead of a single estimate Toy illustration: Optimization surface for a single variable Maximum a-Posteriori (MAP) Pixel intensity Score

17. Simple 1-D blind deconvolution example y = observed image b = blur x = sharp image n = noise ~ N(0,σ 2 ) Let y = 2

18. σ 2 = 0.1

19. Gaussian distribution:

21. Marginal distribution p(b|y)

22. MAP solution Highest point on surface:

23. Variational Bayes True Bayesian approach not tractable Approximate posterior with simple distribution

24. Fitting posterior with a Gaussian Approximating distribution is Gaussian Minimize

25. KL-Distance vs Gaussian width

26. Variational Approximation of Marginal Variational True marginal MAP

27. Try sampling from the model Let true b = 2 Repeat: Sample x ~ N(0,2) Sample n ~ N(0,σ 2 ) y = xb + n Compute p MAP (b|y), p Bayes (b|y) & p Variational (b|y) Multiply with existing density estimates (assume iid)

28. Actual Setup of Variational Approach Work in gradient domain: Approximate posterior with is Gaussian on each pixel is rectified Gaussian on each blur kernel element Assume Cost function

31. Close-up Original Output

32. Original photograph

33. Our output Blur kernel

34. Our outputOriginal image Close-up

35. Recent Comparison paper Percent success (higher is better) IEEE CVPR 2009 conference

36. Related Problems Bayesian Color Constancy – Brainard & Freeman JOSA 1997 – Given color pixels, deduce: Reflectance Spectrum of surface Illuminant Spectrum – Use Laplace approximation – Similar to Gaussian q(.) From Foundation of Vision by Brian Wandell, Sinauer Associates, 1995

37. Conclusions Variational methods seem to do the right thing for blind deconvolution. Interesting from inference point of view: rare case where Bayesian methods are needed Can potentially be applied to other ill-posed problems in image processing