1. Variational Bayesian Image Processing on Stochastic Factor Graphs Xin Li Lane Dept. of CSEE West Virginia University

2. Outline Statistical modeling of natural images Statistical modeling of natural images From old-fashioned local models to newly-proposed nonlocal models From old-fashioned local models to newly-proposed nonlocal models Factor graph based image modeling Factor graph based image modeling A powerful framework unifying local and nonlocal approaches A powerful framework unifying local and nonlocal approaches EM-based inference on stochastic factor graphs EM-based inference on stochastic factor graphs Applications and experimental results Applications and experimental results Denoising, inpainting, interpolation, post-processing, inverse halftoning, deblurring...... Denoising, inpainting, interpolation, post-processing, inverse halftoning, deblurring......

3. Cast Signal/Image Processing Under a Bayesian Framework Image restoration (Besag et al. ’ 1991) Image restoration (Besag et al. ’ 1991) Image denoising (Simoncelli&Adelson ’ 1996) Image denoising (Simoncelli&Adelson ’ 1996) Interpolation (Mackay ’ 1992) and super-resolution (Schultz& Stevenson ’ 1996 ) Interpolation (Mackay ’ 1992) and super-resolution (Schultz& Stevenson ’ 1996 ) Inverse halftoning (Wong ’ 1995) Inverse halftoning (Wong ’ 1995) Image segmentation (Bouman&Shapiro ’ 1994) Image segmentation (Bouman&Shapiro ’ 1994) x: Unobservable data y: Observation data Image prior (the focus of this talk) Likelihood (varies from application to application)

4. Statistical Modeling of Natural Images: the Pursuit of a Good Prior Local models Local models Markov Random Field (MRF) and its extensions (e.g., 2D Kalman-filtering, Field-of-Expert) Markov Random Field (MRF) and its extensions (e.g., 2D Kalman-filtering, Field-of-Expert) Sparsity-based: DCT, wavelets, steerable pyramids, geometric wavelets (edgelets, curvelets, ridgelets, bandelets) Sparsity-based: DCT, wavelets, steerable pyramids, geometric wavelets (edgelets, curvelets, ridgelets, bandelets) Nonlocal models Nonlocal models Bilateral filtering (Tomasi et al. ICCV ’ 1998) Bilateral filtering (Tomasi et al. ICCV ’ 1998) Texture synthesis (Efros&Leung ICCV ’ 1999) Texture synthesis (Efros&Leung ICCV ’ 1999) Exemplar-based inpainting (Criminisi et al. TIP ’ 2004) Exemplar-based inpainting (Criminisi et al. TIP ’ 2004) Nonlocal mean denoising (Buades et al. ’ CVPR ’ 2005) Nonlocal mean denoising (Buades et al. ’ CVPR ’ 2005) Total Least-Square denoising (Hirakawa&Parks TIP ’ 2006) Total Least-Square denoising (Hirakawa&Parks TIP ’ 2006) Block-matching 3D denoising (Dabov et al. TIP ’ 2007) Block-matching 3D denoising (Dabov et al. TIP ’ 2007)

5. Introducing a New Language of Factor Graphs Why Factor Graphs? Why Factor Graphs? The most general form of graphical probability models (both MRF and Bayesian networks can be converted to FGs) The most general form of graphical probability models (both MRF and Bayesian networks can be converted to FGs) Widely used in computer science and engineering ( ) Widely used in computer science and engineering (forward-backward algorithm, Viterbi algorithm, turbo decoding algorithm, Pearl ’ s belief propagation algorithm, Kalman filter 1 ) What is Factor Graph? What is Factor Graph? a bipartite graph that expresses which variables are arguments of which local functions Factor/function node (solid squares) vs. variable nodes (empty circles) Factor/function node (solid squares) vs. variable nodes (empty circles) B1B1 B2B2 B7B7 B8B8 B3B3 B4B4 B5B5 B6B6 f1f1 f2f2 f3f3 f4f4 f1f1 f2f2 f3f3 f4f4 1,2,4 3,6 5,7 7,8 L:F  V 1 Kschischang, F.R.; Frey, B.J.; Loeliger, H.-A., "Factor graphs and the sum-product algorithm," IEEE Transactions on Information Theory,, vol.47, no.2, pp.498-519, Feb 2001

6. Variable Nodes=Image Patches Neuroscience: receptive fields of neighboring cells in human vision system have severe overlapping Neuroscience: receptive fields of neighboring cells in human vision system have severe overlapping Engineering: patch has been under the disguise of many different names such as windows in digital filters, blocks in JPEG and the support of wavelet bases Engineering: patch has been under the disguise of many different names such as windows in digital filters, blocks in JPEG and the support of wavelet bases Cited from D. Hubel, “Eye, Brain and Vision”, 1988

7. Factorization: the Art of Statistical Image Modeling Wavelet-based statistical models (geometric proximity defines the neighborhood) Locally linear embedding 1 (perceptual similarity defines the neighborhood) SP ML Domain-Markovian Range-Markovian 1 S.T. Roweis and L.K. Saul, “ Nonlinear Dimensionality Reduction by Locally Linear Embedding ” (22 December 2000),Science 290 (5500), 2323.

8. Unification Using Factor Graphs f1f1 f2f2 f3f3 f4f4 B1B1 B2B2 B3B3 B4B4 naive Bayesian (DCT/wavelet-based models) MRF-based B0B0 B1B1 B2B2 B3B3 x B0B0 B1B1 B3B3 B2B2 B0B0 B1B1 B2B2 B3B3 kNN/kmeans clustering (nonlocal image models)

9. A Manifold Interpretation of Nonlocal Image Prior A Manifold Interpretation of Nonlocal Image Prior MRNMRN B1B1 BkBk B0B0 How to maximize the sparsity of a representation? Conventional wisdom: adapt basis to signal (e.g., basis pursuit, matching pursuit) New proposal: adapt signal to basis (by probing its underlying organization principle)

10. Organizing Principle: Latent Variable L P(y|x) xy image denoising image inpainting image coding image halftoning L B 11 B 22 B 14 B 13 B 12 B 41 B 31 B 21 B 33 B 32 B 23 B 24 B 34 B 44 B 43 B 42 fBfB fAfA fCfC image deblurring sparsifying transform “ Nature is not economical of structures but organizing principles. ” - Stanislaw M. Ulam L

11. Maximum-Likelihood Estimation of Graph Structure L Pack into 3D Array D For. Trans. Coring B0B0 BkBk B1B1 … Inv. Trans. unpack into 2D patches B0B0 BkBk B1B1 … ^ ^^ Update the estimate of L Update the estimate of x loop over every factor node f j A variational interpretation of such EM-based inference on FGs is referred to the paper P(y|x)

12. Problem 1: Image Denoising PSNR(DB) PERFORMANCE COMPARISON AMONG DIFFERENT SCHEMES FOR 12 TEST IMAGES ATσw = 100 SSIM PERFORMANCE COMPARISON AMONG DIFFERENT SCHEMES FOR 12 TEST IMAGES ATσw = 100 BM3D (kNN,iter=2) SFG (kmeans,iter=20) σwσw org. 200 400 600 800 1000

13. Problem 2: Image Recovery top-down: test1, test3, test5 top-down: test2, test4, test6 DCT FoE EXP BM3D LSP SFG PSNR(dB) performance comparison SSIM performance comparison Local models: DCT, FoE and LSP Nonlocal models: EXP, BM3D 1 and SFG 1 Our own extension into image recovery xy

14. xybicubicNEDI 1 FG 28.70dB 27.34dB 28.19dB 31.76dB 32.36dB 32.63dB 34.71dB 34.45dB 37.35dB 18.81dB 15.37dB 16.45dB Problem 3: Resolution Enhancement 1 X. Li and M. Orchard, “ New edge directed interpolation ”, IEEE TIP, 2001

15. 29.06dB 31.56dB 34.96dB xy DT KRFG 1 28.46dB 31.16dB 36.51dB 17.90dB 18.49dB 29.25dB 26.04dB 24.63dB 29.91dB Problem 4: Irregular Interpolation DT- Delauney Triangle-based (griddata under MATLAB) KR- Kernal Regression-based (Takeda et al. IEEE TIP 2007 w/o parameter optimization) 1 X. Li, “ Patch-based image interpolation: algorithms and applications, ” Inter. Workshop on Local and Non-Local Approximation (LNLA) ’ 2008 25% kept

16. Problem 5: Post-processing JPEG-decoded at rate of 0.32bpp (PSNR=32.07dB) SFG-enhanced at rate of 0.32bpp (PSNR=33.22dB) SPIHT-decoded at rate of 0.20bpp (PSNR=26.18dB) SFG-enhanced at rate of 0.20bpp (PSNR=27.33dB) Maximum-Likelihood (ML) Decoding Maximum a Posterior (MAP) Decoding

17. Problem 6: Inverse Halftoning without nonlocal prior 1 (PSNR=31.84dB, SSIM=0.8390) with nonlocal prior (PSNR=32.82dB, SSIM=0.8515) 1 Available from Image Halftoning Toolbox released by UT-Austin Researchers

18. Conclusions and Perspectives Despite the rich structures in natural images, the underlying organization principle is simple (self- similarity Despite the rich structures in natural images, the underlying organization principle is simple (self- similarity We have shown how similarity can lead to sparsity in a nonlinear representation of images We have shown how similarity can lead to sparsity in a nonlinear representation of images FG only represents one mathematical language for interpreting such principle (multifractal formalism is another) FG only represents one mathematical language for interpreting such principle (multifractal formalism is another) Image processing (low-level vision) could benefit from data clustering (higher-level vision): how does human visual cortex learn to decode the latent variable L through unsupervised learning? Image processing (low-level vision) could benefit from data clustering (higher-level vision): how does human visual cortex learn to decode the latent variable L through unsupervised learning? Reproducible Research: MATLAB codes accompanying this work are available at http://www.csee.wvu.edu/~xinl/sfg.html (more will be added)http://www.csee.wvu.edu/~xinl/sfg.html