Shaojie Zhuo, Dong Guo, Terence Sim School of Computing, National University of Singapore CVPR2010 Reporter: 周 澄 (A.J.) 01/16/2011 Key words: image deblur, flash/no- flash technique, blur kernel estimate, sharp image reconstruction

Goal Major contributions Related work Flash deblurring framework ‐MAP optimization Practical implementation Results Limitations Future works Outline

Deblur a shaken image by using a corresponding flash image generated from a conventional hand-held camera. Goal

Propose a novel approach by adding a flash image as a key constraint. Well handle both the flash artifacts and the deconvolution artifacts. ‐Additional constraints introduced. High quality results. ‐Insensitive to noise, fine image details. Major contributions

Non-blind image deconvolution (Single image method). Def: Given the estimated blur kernel, the second step is to reconstruct a sharp image from the blurred image. Related work

[Fergus et al. TOG2006] Removing camera shake from a single photograph. Used a variational Bayes inference method with natural image statistics to estimate the motion blur kernel. Related work

[Jia. CVPR 2007] Single image motion deblurring using transparency. Investigated the relationship between object boundary transparency and the image motion blur and estimated the blur kernel from the alpha matte of motion objects. Related work

[Shan et al. TOG 2008] High-quality motion deblurring from a single image. Formulated the deblurring problem as an MAP problem and proposed a high-order derivatives image noise model and a local image prior to avoid trivial solution. Related work

Fergus, Jia, and Shan’s method are all able to obtain accurate kernels when the blur is small. =>Fail at large shaking & noise. Related work

Wiener filter and the Richardson-Lucy (RL) deconvolution. [Levin et al. CVPR2009] Understanding and evaluating blind deconvolution algorithms. Related work

Wiener filter and the Richardson-Lucy (RL) deconvolution. =>Suffer from deconvolution artifacts such as amplified noise and ringing artifacts. Related work

Regularization methods. ‐In order to reduce artifacts. [Wang et al. SIIMS2008] A new alternating minimization algorithm for total variation image reconstruction. [Yuan et al. TOG2008] Progressive interscale and intra-scale non-blind image deconvolution. Related work

Regularization methods. =>Lost fine image details since you cannot separate image details from artifacts like noise or ringing artifacts properly. Related work

Additional hardware(Hybrid camera). High resolution camera + low resolution video camera. =>Some image details still lost due to non-invertible motion blur. Related work

Multiple images solution. [Yuan et al. TOG2007] Image deblurring with blurred/noisy image pairs. [Chen et al. CVPR2008] Robust dual motion deblurring. [YW Tai et al. CVPR2005] Local color transfer via probabilistic segmentation by expectation-maximization. Related work

Flash/no-flash technique. [Agrawal et al. TOG2005] Removing photography artifacts using gradient projection and flashexposure sampling. [Petschnigg et al. TOG2004] Digital photography with flash and no-flash image pairs. [Eisemann et al. TOG2004] Flash photography enhancement via intrinsic relighting. Related work

Traditional flash/no-flash technique. =>Need good alignment between two images. Related work

Input: A blur image B and corresponding flash image F. Output: A visually pleasant sharp image I with least flash artifacts or deconvolution artifacts. Flash deblurring framework

Problem formulation: Given the blurred image B and flash image F, our goal is to estimate a blur kernel K and a sharp image I, so that I,K and B can be represented by the convolution model and the gradients of I are close to those in F.(We talk about gradients later) Flash deblurring framework

Kernel estimation Sharp image reconstruction Maximum-a-posteriori (MAP) framework

B = I * K + n, where n is the image noise which modeled as a set of independent and identically distributed(i.i.d.) Gaussian noise. The convolution model

Problem formulation: where L( ． ) = - log (p( ． )) MAP optimization

Likelihood term(SSD): Analyze the SSD of the estimated kernel convolution result and the blur image. MAP optimization

Kernel prior term: where the parameter α ≦ 1; α=0.8 used in the paper. MAP optimization

Key idea: The robust flash gradient constraint encourages the gradients of reconstructed image to be close to those in F, while at the locations of flash artifacts, ambient shadows or noise it allow their gradient to differ to avoid flash artifacts and keep ambient illumination. Flash gradient constraint

Flash gradient constraint: observation. MAP optimization 1D scanlines of intensities and gradients in R channel of the three images. In the gradient plot, ΔI(cyan) is much close to ΔF(magenta), which acts as a guide to reconstruct the sharp image I.

Flash gradient constraint: where is the Lorentzian robust estimator and ε is a predefined constant. MAP optimization

Objective function: Here we have the likelihood term, flash gradient constraint and kernel prior term, where λ f and λ k are the used to balance the three terms. MAP optimization

Flash deblurring framework Kernel estimation Sharp image reconstruction Maximum-a-posteriori (MAP) framework

Kernel estimation Sharp image reconstruction Fix K, we can estimate I by solving: Where the weight of re-weight least square for each pixel i at each iteration:

Kernel estimation Sharp image reconstruction Fix I, we can estimate K by solving: Both equation can be solved by iterative re-weight least squares(IRLS).

Kernel estimation Sharp image reconstruction The two steps are alternated until diff(K) is smaller than the threshold. To avoid local minimum when blur kernel is large, the kernel estimation is performed in a coarse-to-fine manner in the scale space.

Kernel estimation Sharp image reconstruction Three common artifacts. 1.Flash artifact regions. 2. λ f is set to be large to suppress noise or ringing artifacts. 3.Over-saturated regions.

Kernel estimation Sharp image reconstruction Build a mask image M to change the weight of flash gradient constraint locally.

Kernel estimation Sharp image reconstruction Flash artifacts detect. ‐|| K * I - F || 2. ‐Only need to manually mark the the flash shadow edge, since the shadow regions still contains useful gradients.

Kernel estimation Sharp image reconstruction Add in the sparse gradient constraint. The objective func becomes: where “ο” denotes the pixel-wise multiplication operator. Solved by IRLS.

Take flash image first, and then use high speed capturing mode capture blur image. As the time between two shots is small, the the motion during the shots is basically a translation, which just causes a shift in the estimated blur kernel. Therefore, no image alignment is required. Practical implementation

Resaults – kernel RMS error

Resaults

Results

It cannot handle the spatially invariant motion blur model. ‐Additional alignment needed. ‐Ex: Busy traffic. The exposure time between flash/no- flash images should be close. ‐Temporal incoherence. Two images should share same aperture value. ‐May generate blur artifacts caused by different focus. Limitations

Extend to video. ‐Coherent preserve. Support hybrid system such as combining with IR depth sensor or stereo sensor to get more information from image. Future works