1. 1 Bayesian Image Modeling by Generalized Sparse Markov Random Fields and Loopy Belief Propagation Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan http://www.smapip.is.tohoku.ac.jp/~kazu/ Collaborators Muneki Yasuda (GSIS, Tohoku University, Japan) Sun Kataoka (GSIS, Tohoku University, Japan) D. M. Titterington (Department of Statistics, University of Glasgow, UK) 20 March, 2013SPDSA2013, Sendai, Japan

2. 2 Outline 1.Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation 2.Bayesian Image Modeling by Generalized Sparse Prior 3.Noise Reductions by Generalized Sparse Prior 4.Concluding Remarks 20 March, 2013SPDSA2013, Sendai, Japan

3. 3 Probabilistic Model and Belief Propagation Probabilistic Information Processing Probabilistic Models Bayes Formulas Belief Propagation =Bethe Approximation Bayesian Networks Markov Random Fields 20 March, 2013 SPDSA2013, Sendai, Japan V: Set of all the nodes (vertices) in graph G E: Set of all the links (edges) in graph G ji Message =Effective Field

4. 4 Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation Prior Probability of natural images is assumed to be the following pairwise Markov random fields: 4 20 March, 2013 SPDSA2013, Sendai, Japan

5. Supervised Learning Scheme by Loopy Belief Propagation in Pairwise Markov random fields: 5 Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation 5 Histogram from Supervised Data 20 March, 2013 SPDSA2013, Sendai, Japan M. Yasuda, S. Kataoka and K.Tanaka, J. Phys. Soc. Jpn, Vol.81, No.4, Article No.044801, 2012.

6. 6 Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation Supervised Learning Scheme by Loopy Belief Propagation in Pairwise Markov random fields: 6 20 March, 2013 SPDSA2013, Sendai, Japan

7. 7 Bayesian Image Modeling by Generalized Sparse Prior Assumption: Prior Probability is given as the following Gibbs distribution with the interaction  between every nearest neighbour pair of pixels: 7 p=0: q-state Potts model p=2: Discrete Gaussian Graphical Model 20 March, 2013 SPDSA2013, Sendai, Japan

8. 8 Prior in Bayesian Image Modeling 8 In the region of 0<p<0.3504…, the first order phase transition appears and the solution  * does not exist. 20 March, 2013 SPDSA2013, Sendai, Japan Loopy Belief Propagation q=16

9. 9 Bayesian Image Modeling by Generalized Sparse Prior: Conditional Maximization of Entropy 9 Lagrange Multiplier Loopy Belief Propagation 20 March, 2013 SPDSA2013, Sendai, Japan K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012.

10. 10 Prior Analysis by LBP and Conditional Maximization of Entropy in Bayesian Image Modeling 10 Prior Probability q=16 p=0.2 q=16 p=0.5 Repeat until A converges 20 March, 2013 SPDSA2013, Sendai, Japan LBP

11. 11 Prior Analysis by LBP and Conditional Maximization of Entropy in Generalized Sparse Prior 11 Log-Likelihood for p when the original image f * is given q=16 LBP Free Energy of Prior 20 March, 2013 SPDSA2013, Sendai, Japan K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012.

12. 12 Degradation Process in Bayesian Image Modeling Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise. 12 20 March, 2013SPDSA2013, Sendai, Japan

13. 13 Noise Reductions by Generalized Sparse Prior Posterior Probability 13 Lagrange Multipliers 20 March, 2013 SPDSA2013, Sendai, Japan K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012.

14. 14 Noise Reduction Procedures Based on LBP and Conditional Maximization of Entropy 14 Repeat until C converge Repeat until u, B and L converge 20 March, 2013SPDSA2013, Sendai, Japan Marginals and Free Energy in LBP Repeat until C and M converge Input p and data g

15. 15 Noise Reductions by Generalized Sparse Priors and Loopy Belief Propagation p=0.5 p=0.2 Original Image Degraded Image RestoredImage p=1 20 March, 2013SPDSA2013, Sendai, Japan K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012.

16. 16 Noise Reductions by Generalized Sparse Priors and Loopy Belief Propagation p=0.5 p=0.2 Original Image Degraded Image RestoredImage p=1 20 March, 2013SPDSA2013, Sendai, Japan K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012.

17. 17 Summary Formulation of Bayesian image modeling for image processing by means of generalized sparse priors and loopy belief propagation are proposed. Our formulation is based on the conditional maximization of entropy with some constraints. In our sparse priors, although the first order phase transitions often appear, our algorithm works well also in such cases. 20 March, 2013SPDSA2013, Sendai, Japan

18. 18 References 1.S. Kataoka, M. Yasuda, K. Tanaka and D. M. Titterington: Statistical Analysis of the Expectation-Maximization Algorithm with Loopy Belief Propagation in Bayesian Image Modeling, Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.50-63,2012. 2.M. Yasuda and K. Tanaka: TAP Equation for Nonnegative Boltzmann Machine: Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.192-209, 2012. 3.S. Kataoka, M. Yasuda and K. Tanaka: Statistical Analysis of Gaussian Image Inpainting Problems, Journal of the Physical Society of Japan, Vol.81, No.2, Article No.025001, 2012. 4.M. Yasuda, S. Kataoka and K.Tanaka: Inverse Problem in Pairwise Markov Random Fields using Loopy Belief Propagation, Journal of the Physical Society of Japan, Vol.81, No.4, Article No.044801, pp.1-8, 2012. 5.M. Yasuda, Y. Kabashima and K. Tanaka: Replica Plefka Expansion of Ising systems, Journal of Statistical Mechanics: Theory and Experiment, Vol.2012, No.4, Article No.P04002, pp.1-16, 2012. 6.K. Tanaka, M. Yasuda and D. M. Titterington: Bayesian image modeling by means of generalized sparse prior and loopy belief propagation, Journal of the Physical Society of Japan, Vol.81, No.11, Article No.114802, 2012. 7.M. Yasuda and K. Tanaka: Susceptibility Propagation by Using Diagonal Consistency, Physical Review E, Vol.87, No.1, Article No.012134, 2013. 20 March, 2013SPDSA2013, Sendai, Japan