1. Deblurring Shaken and Partially Saturated Images
Oliver Whyte Josef Sivic Andrew Zisserman

2. The Problem Static scene Camera moves during exposure
We want to recover the sharp image

3. Model of Camera Shake Blur
blurry image
sharp image
linear forward model
weights for each camera pose
transformation due to camera pose
O. Whyte, J. Sivic, and A. Zisserman. “Non-uniform Deblurring for Shaken Images”. In Proc. CVPR 2010.

4. General Deblurring Process
“Blind” PSF
estimation
“Non-blind”
deblurring
Contributions: 1
Saturation 2
Efficient approximation

5. Deblurring Saturated Images
Existing non-blind methods fail
Blurry

6. Deblurring Saturated Images
State-of-the-art for non-blind deblurring: “Fast Image Deconvolution using Hyper-Laplacian Priors”
Blurry
Krishnan & Fergus
D. Krishnan and R. Fergus. “Fast Image Deconvolution using Hyper-Laplacian Priors”. In NIPS, 2009.

7. Deblurring Saturated Images
Richardson-Lucy algorithm (1970s) does better:
Blurry
Krishnan & Fergus
Richardson-Lucy
W. H. Richardson. "Bayesian-Based Iterative Method of Image Restoration". J. of the Optical Society of America, 1972.
L. B. Lucy. "An iterative technique for the rectification of observed distributions". Astronomical Journal, 1974.

8. Outline
A forward model for saturation, and its application in Richardson-Lucy non-blind deblurring
Preventing ringing in Richardson-Lucy
Efficient implementation of spatially-varying camera shake blur

9. Related Work
Very little work on saturation Harmeling et al. ICIP 10  Cho et al. ICCV 11
Blind deblurring (PSF estimation) of camera shake Fergus et al. SIGGRAPH 06  Shan et al. SIGGRAPH 08  Cho & Lee SIGGRAPH Asia 09  Whyte et al. CVPR 10  Levin et al. CVPR 11
Non-blind deblurring (known PSF) Wiener 1949  Richardson 1972 / Lucy 1974  Dabov et al. SPIE 08  Krishnan & Fergus NIPS 09

10. Saturation in Image Formation
Sensor
response
Sharp
True kernel
Blurry
saturation level
Deblurred
Richardson-Lucy
deconvolution
“Estimated” kernel

11. Saturation in Image Formation
Sensor
response
Sharp
True kernel
Blurry
saturation level
Deblurred
Richardson-Lucy
deconvolution
“Estimated” kernel

12. 1st observation: Linear model is incorrect
Replace linear model with non-linear (clipped) model
Simple threshold for R
Smooth approximation for R
Non-differentiable at
Smooth and differentiable
C. Chen and O. L. Mangasarian. “A Class of Smoothing Functions for Nonlinear and Mixed Complementarity Problems”. Computational Optimization and Applications, 1996

13. “Saturated” Richardson-Lucy
Re-derive Richardson-Lucy with non-linear model
Effectively weights data by
Automatically downweights saturated pixels
Blurry image
Current

14. “Saturated” Richardson-Lucy
Sensor
response
Sharp
True kernel
Blurry
Deblurred
“Saturated”
Richardson-Lucy
deconvolution
“Estimated” kernel

15. “Saturated” Richardson-Lucy
True sharp signal
Standard RL
“Saturated” RL
Significant improvement, however…
Ringing still appears due to the mis-estimation of pixels near edge of saturated region

16. 2nd observation: Gross errors in deblurred image near saturation
True sharp signal
“Saturated” RL
“Combined” RL U S
Split the deblurred image into:
Unsaturated – can be estimated accurately
Saturated – cannot be estimated accurately
Update U in a way that is independent of S

17. Combined Richardson-Lucy
Combined Richardson-Lucy greatly reduces ringing
Blurry
Krishnan & Fergus
Richardson-Lucy
Ours

18. Deblurring Real Saturated Images
Blurry
Krishnan & Fergus
Richardson-Lucy
Our result

19. Deblurring Real Saturated Images
Blurry
Krishnan & Fergus
Richardson-Lucy
Our result

20. Deblurring Real Saturated Images
Blurry
Krishnan & Fergus
Richardson-Lucy
Our result

21. Speeding Up Spatially-Varying Blur
Two main approaches to spatially-varying blur
Global models (Klein & Drummond 05, Whyte et al. 10, Joshi et al. 10, Gupta et al. 10)
Correct – derived from realistic geometric models
Computationally expensive – no tricks like FFT convolution
Locally-uniform models (Nagy et al. 98, Tai et al. 10, Harmeling et al. 10)
Heuristic – independent filters for different regions
Much cheaper to use – use FFT for convolutions
We would like to combine the two approaches

22. Locally-Uniform Approximation
Approximate blur as being locally-uniform
Allows fast computation of
each patch assigned a filter
M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling. "Efficient Filter Flow for Space-Variant Multiframe Blind Deconvolution". In Proc. CVPR, 2010.

23. Locally-Uniform Approximation
Used for spatially-varying blind deblurring
Estimate one filter per patch
estimate filter for each patch
penalize for adjacent patches
heuristics for finding & correcting “bad” filters
S. Harmeling, M. Hirsch, and B. Scholkopf. "Space-Variant Single-Image Blind Deconvolution for Removing Camera Shake". In NIPS, 2010.

24. Locally-Uniform Approximation
General global model of spatially-varying blur:
blurry image
sharp image
weights for each camera pose
transformation due to camera pose
O. Whyte, J. Sivic, and A. Zisserman. “Non-uniform Deblurring for Shaken Images”. In Proc. CVPR 2010.

25. Locally-Uniform Approximation
Possible to combine with global model
Computation of x faster
each filter

26. Accuracy vs. Number of Patches
PSFs of a few points in the image:
Camera motion:
Global descriptor:
Exact
12 x 16 patches
6 x 8 patches
3 x 4 patches

27. Blind PSF Estimation Using Fast Approximation
Estimated PSF Exact forward model 67 minutes
Estimated PSF Approximate forward model 7 minutes
Blurry Image

28. Conclusion Forward model for saturation
Modified Richardson-Lucy algorithm for non-linear image formation model
“Combined” RL to prevent ringing
Efficient approximation of global model of spatially-varying blur