Deblurring Shaken and Partially Saturated Images Oliver Whyte Josef Sivic Andrew Zisserman

The Problem Static scene Camera moves during exposure We want to recover the sharp image

Model of Camera Shake Blur blurry image sharp image linear forward model weights for each camera pose transformation due to camera pose O. Whyte, J. Sivic, and A. Zisserman. “Non-uniform Deblurring for Shaken Images”. In Proc. CVPR 2010.

General Deblurring Process “Blind” PSF estimation “Non-blind” deblurring Contributions: 1 Saturation 2 Efficient approximation

Deblurring Saturated Images Existing non-blind methods fail Blurry

Deblurring Saturated Images State-of-the-art for non-blind deblurring: “Fast Image Deconvolution using Hyper-Laplacian Priors” Blurry Krishnan & Fergus D. Krishnan and R. Fergus. “Fast Image Deconvolution using Hyper-Laplacian Priors”. In NIPS, 2009.

Deblurring Saturated Images Richardson-Lucy algorithm (1970s) does better: Blurry Krishnan & Fergus Richardson-Lucy W. H. Richardson. "Bayesian-Based Iterative Method of Image Restoration". J. of the Optical Society of America, 1972. L. B. Lucy. "An iterative technique for the rectification of observed distributions". Astronomical Journal, 1974.

Outline A forward model for saturation, and its application in Richardson-Lucy non-blind deblurring Preventing ringing in Richardson-Lucy Efficient implementation of spatially-varying camera shake blur

Related Work Very little work on saturation Harmeling et al. ICIP 10  Cho et al. ICCV 11 Blind deblurring (PSF estimation) of camera shake Fergus et al. SIGGRAPH 06  Shan et al. SIGGRAPH 08  Cho & Lee SIGGRAPH Asia 09  Whyte et al. CVPR 10  Levin et al. CVPR 11 Non-blind deblurring (known PSF) Wiener 1949  Richardson 1972 / Lucy 1974  Dabov et al. SPIE 08  Krishnan & Fergus NIPS 09

Saturation in Image Formation Sensor response Sharp True kernel Blurry saturation level Deblurred Richardson-Lucy deconvolution “Estimated” kernel

Saturation in Image Formation Sensor response Sharp True kernel Blurry saturation level Deblurred Richardson-Lucy deconvolution “Estimated” kernel

1st observation: Linear model is incorrect Replace linear model with non-linear (clipped) model Simple threshold for R Smooth approximation for R Non-differentiable at Smooth and differentiable C. Chen and O. L. Mangasarian. “A Class of Smoothing Functions for Nonlinear and Mixed Complementarity Problems”. Computational Optimization and Applications, 1996

“Saturated” Richardson-Lucy Re-derive Richardson-Lucy with non-linear model Effectively weights data by Automatically downweights saturated pixels Blurry image Current

“Saturated” Richardson-Lucy Sensor response Sharp True kernel Blurry Deblurred “Saturated” Richardson-Lucy deconvolution “Estimated” kernel

“Saturated” Richardson-Lucy True sharp signal Standard RL “Saturated” RL Significant improvement, however… Ringing still appears due to the mis-estimation of pixels near edge of saturated region

2nd observation: Gross errors in deblurred image near saturation True sharp signal “Saturated” RL “Combined” RL U S Split the deblurred image into: Unsaturated – can be estimated accurately Saturated – cannot be estimated accurately Update U in a way that is independent of S

Combined Richardson-Lucy Combined Richardson-Lucy greatly reduces ringing Blurry Krishnan & Fergus Richardson-Lucy Ours

Deblurring Real Saturated Images Blurry Krishnan & Fergus Richardson-Lucy Our result

Deblurring Real Saturated Images Blurry Krishnan & Fergus Richardson-Lucy Our result

Deblurring Real Saturated Images Blurry Krishnan & Fergus Richardson-Lucy Our result

Speeding Up Spatially-Varying Blur Two main approaches to spatially-varying blur Global models (Klein & Drummond 05, Whyte et al. 10, Joshi et al. 10, Gupta et al. 10) Correct – derived from realistic geometric models Computationally expensive – no tricks like FFT convolution Locally-uniform models (Nagy et al. 98, Tai et al. 10, Harmeling et al. 10) Heuristic – independent filters for different regions Much cheaper to use – use FFT for convolutions We would like to combine the two approaches

Locally-Uniform Approximation Approximate blur as being locally-uniform Allows fast computation of each patch assigned a filter M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling. "Efficient Filter Flow for Space-Variant Multiframe Blind Deconvolution". In Proc. CVPR, 2010.

Locally-Uniform Approximation Used for spatially-varying blind deblurring Estimate one filter per patch estimate filter for each patch penalize for adjacent patches heuristics for finding & correcting “bad” filters S. Harmeling, M. Hirsch, and B. Scholkopf. "Space-Variant Single-Image Blind Deconvolution for Removing Camera Shake". In NIPS, 2010.

Locally-Uniform Approximation General global model of spatially-varying blur: blurry image sharp image weights for each camera pose transformation due to camera pose O. Whyte, J. Sivic, and A. Zisserman. “Non-uniform Deblurring for Shaken Images”. In Proc. CVPR 2010.

Locally-Uniform Approximation Possible to combine with global model Computation of 5-10x faster each filter

Accuracy vs. Number of Patches PSFs of a few points in the image: Camera motion: Global descriptor: Exact 12 x 16 patches 6 x 8 patches 3 x 4 patches

Blind PSF Estimation Using Fast Approximation Estimated PSF Exact forward model 67 minutes Estimated PSF Approximate forward model 7 minutes Blurry Image

Conclusion Forward model for saturation Modified Richardson-Lucy algorithm for non-linear image formation model “Combined” RL to prevent ringing Efficient approximation of global model of spatially-varying blur