Sparsity-based Image Deblurring with Locally Adaptive and Nonlocally Robust Regularization Weisheng Dong a, Xin Li b, Lei Zhang c, Guangming Shi a a Xidian University, b West Virginia University, c HongKong Polytechnic University This work is partially supported by NSF CCF , HK RGC General Research Fund (PolyU 5375/09E), NSFC (No , , ,and ), and the Fundamental Research Funds of the Central Universities of China (No. K )

History of Image Restoration (Non-blind Image Deconvolution) MethodISNR(dB) Forward: Fourier-wavelet regularized deconvolution for ill-conditioned systems (Neelamani, Choi, Baraniuk, TSP’2004)7.30 An Expectation-Maxization algorithm for wavelet-based image restoration (Figueiredo,Nowak TIP’2003)7.59 Total-Variation based image deconvolution: a majorization-minimization approach (Bioucas-Dias,Figueiredo,Oliveira ICASSP’2006) 8.52 A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration (Bioucas-Dias,Figueiredo TIP’2007) 8.63 Pointwise Shape-Adaptive DCT (Foi,Katkovnik,Egiazarian TIP’2007)8.57 Block-matching and collaborative 3D filtering (Dabov,Foi,Katkovnik,Egiazarian TIP’2007)8.34 Image restoration through L0 analysis-based sparse optimization in tight frames (Portilla ICIP’2009)9.10 Variational Bayesian Image Restoration With a Product of Spatially Weighted Total Variation Image Priors (Chantas, Galatsanos, Molina, Katsaggelos, TIP 2010) 9.61 This work (sparsity-based image deblurring with local adaptive and nonlocal robust regularization)9.00 Patch-based Image Deconvolution Via Joint Modeling of Sparse Priors (Jia, Evans ICIP’2011)8.98 Fine-granularity and spatially-adaptive regularization for projection-based image deblurring (Li TIP’2011)10.10 Centralized sparse representation for image restoration (Dong, Zhang, Shi ICCV’2011)10.40 Cameraman 256×256, 9×9 uniform blur, BSNR=40dB

Lessons We Have Learned “All models are wrong; but some are useful” – G. Box “All models are wrong; but some are useful” – G. Box Local models: wavelet/DCT, total-variation (TV), spatially-weighted TV (SWTV), … Local models: wavelet/DCT, total-variation (TV), spatially-weighted TV (SWTV), … Nonlocal models: nonlocal-mean, BM3D, nonlocal TV, ASDS-AR-NL (precursor of this work), … Nonlocal models: nonlocal-mean, BM3D, nonlocal TV, ASDS-AR-NL (precursor of this work), … DCT>BM3D BM3D>DCT

One Simple Message Local variation and nonlocal invariance are two sides of the same coin Local variation and nonlocal invariance are two sides of the same coin Local variation Nonlocal invariance Kanizsa Triangle

Local View: Dictionary Learning Daubechies’ wavelet,1988 Do&Vetterli’s contourlet,2005 Bell&Sejnowski’ICA,1996 Elad&Aharon’K-SVD,2006 Lagrange’s idea the magic of l 1 NP-hard HOTTY

Nonlocal View: Structural Clustering Kmeans-based clustering NLM denoising (Buades et al. CVPR’2005)

Variational Formulation NL-similarity penalty term Structural clustering penalty term

Key Derivations Iterative thresholding (via surrogate functions) Typo in the paper

Summary of Algorithm

Connection with Other Competing Works Patch-based Image Deconvolution Via Joint Modeling of Sparse Priors (ICIP’2011) Patch-based Image Deconvolution Via Joint Modeling of Sparse Priors (ICIP’2011) Nonlocal total-variation for image restoration (UCLA Math TR) Nonlocal total-variation for image restoration (UCLA Math TR) Deconvolution network (CVPR’2010) Deconvolution network (CVPR’2010) Handling Outliers in Non-Blind Image Deconvolution (ICCV”2011) Handling Outliers in Non-Blind Image Deconvolution (ICCV”2011) Close the Loop: Joint Blind Image Restoration and Recognition with Sparse Representation Prior (ICCV’2011) Close the Loop: Joint Blind Image Restoration and Recognition with Sparse Representation Prior (ICCV’2011)

Experimental Results MATLAB codes accompanying this work are available From my homepage:

Image Comparison Results (I) original Noisy and blurred SWTV (28.96dB) L 0 -sparsity (29.04dB) BM3D (30.22dB) LANL (31.33dB)

Image Comparison Results (II) original SWTV (27.96dB) BM3D (27.22dB) Noisy and blurred L 0 -sparsity (27.12dB) LANL (29.15dB)

Image Comparison Results (III) LANL (31.33dB) CSR (32.09dB) LANL (29.15dB) CSR (29.75dB)

Conclusions and Perspectives What should we care about? What should we care about? Pursue even higher ISNR value for cameraman? Pursue even higher ISNR value for cameraman? A collection of benchmark images? A collection of benchmark images? Landweber vs. Lucy-Richardson Landweber vs. Lucy-Richardson Application side: Application side: Motion deblurring: from non-blind to blind image deconvolution? Motion deblurring: from non-blind to blind image deconvolution? What will be the next episode like the malfunctioned mirror of Hubble Space Telescope? What will be the next episode like the malfunctioned mirror of Hubble Space Telescope? Dehazing: from linear blur to nonlinear hazing Dehazing: from linear blur to nonlinear hazing