Probabilistic image processing and Bayesian network Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ References K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, vol.35, pp.R81-R150 (2002). K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004). RC2005 (19 July, 2005, Sendai)

Bayesian Network and Belief Propagation Bayes Formula Probabilistic Model Probabilistic Information Processing Belief Propagation J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988). C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996). RC2005 (19 July, 2005, Sendai)

Formulation of Belief Propagation Link between belief propagation and statistical mechanics. Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and　Practice (MIT Press, 2001). Generalized belief propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). Information geometrical interpretation of belief propagation S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004). RC2005 (19 July, 2005, Sendai)

Application of Belief Propagation Image Processing K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002). A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002). Low Density Parity Check Codes Y. Kabashima and D. Saad: Statistical mechanics of　low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004). CDMA Multiuser Detection Algorithm Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003). T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005). Satisfability Problem O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265　(2001). M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002). RC2005 (19 July, 2005, Sendai)

Contents Introduction Belief Propagation Bayesian Image Analysis and Gaussian Graphical Model Image Segmentation Concluding Remarks RC2005 (19 July, 2005, Sendai)

It is very hard to calculate exactly except some special cases. Belief Propagation How should we treat the calculation of the summation over 2N configurations. It is very hard to calculate exactly except some special cases. Formulation for approximate algorithm Accuracy of the approximate algorithm RC2005 (19 July, 2005, Sendai)

Tractable Model Probabilistic models with no loop are tractable. Factorizable Probabilistic models with loop are not tractable. Not Factorizable RC2005 (19 July, 2005, Sendai)

Probabilistic Model on a Graph with Loops Marginal Probability RC2005 (19 July, 2005, Sendai)

Message Passing Rule of Belief Propagation The reducibility conditions can be rewritten as the following fixed point equations. This fixed point equations is corresponding to the extremum condition of the Bethe free energy. And the fixed point equations can be numerically solved by using the natural iteration. The algorithm is corresponding to the loopy belief propagation. 1 3 4 2 5 Fixed Point Equations for Massage RC2005 (19 July, 2005, Sendai)

Approximate Representation of Marginal Probability 1 4 2 5 3 In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel. These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels. The messages are determined so as to satisfy the reducibility conditions. Fixed Point Equations for Messages RC2005 (19 July, 2005, Sendai)

Fixed Point Equation and Iterative Method RC2005 (19 July, 2005, Sendai)

Contents Introduction Belief Propagation Bayesian Image Analysis and Gaussian Graphical Model Image Segmentation Concluding Remarks RC2005 (19 July, 2005, Sendai)

Bayesian Image Analysis Noise Transmission Original Image Degraded Image RC2005 (19 July, 2005, Sendai)

Bayesian Image Analysis Degradation Process Additive White Gaussian Noise Transmission Original Image Degraded Image RC2005 (19 July, 2005, Sendai)

Bayesian Image Analysis A Priori Probability Generate Standard Images Similar? RC2005 (19 July, 2005, Sendai)

Bayesian Image Analysis A Posteriori Probability Gaussian Graphical Model RC2005 (19 July, 2005, Sendai)

Bayesian Image Analysis A Priori Probability Degraded Image Degraded Image Original Image Pixels A Posteriori Probability RC2005 (19 July, 2005, Sendai)

Hyperparameter Determination by Maximization of Marginal Likelihood In the image restoration, we usually have to estimate the hyperparameters alpha and p. In statistics, the maximum likelihood estimation is often employed. In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image. The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model. We can calculate these partition functions approximately by using the Bethe approximation. Marginalization Degraded Image Original Image Marginal Likelihood RC2005 (19 July, 2005, Sendai)

Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Q-Function In the image restoration, we usually have to estimate the hyperparameters alpha and p. In statistics, the maximum likelihood estimation is often employed. In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image. The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model. We can calculate these partition functions approximately by using the Bethe approximation. Incomplete Data Equivalent RC2005 (19 July, 2005, Sendai)

Iterate the following EM-steps until convergence: Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Marginal Likelihood Q-Function In the image restoration, we usually have to estimate the hyperparameters alpha and p. In statistics, the maximum likelihood estimation is often employed. In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image. The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model. We can calculate these partition functions approximately by using the Bethe approximation. Iterate the following EM-steps until convergence: EM Algorithm A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977). RC2005 (19 July, 2005, Sendai)

One-Dimensional Signal 127 255 100 200 Original Signal Degraded Signal Estimated Signal EM Algorithm RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical Model EM Algorithm with Belief Propagation Original Image Degraded Image MSE: 1512 MSE: 1529 RC2005 (19 July, 2005, Sendai)

Exact Results of Gaussian Graphical Model Multi-dimensional Gauss integral formula RC2005 (19 July, 2005, Sendai)

Comparison of Belief Propagation with Exact Results in Gaussian Graphical Model MSE Belief Propagation 327 0.000611 36.302 -5.19201 Exact 315 0.000759 37.919 -5.21444 Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. MSE Belief Propagation 260 0.000574 33.998 -5.15241 Exact 236 0.000652 34.975 -5.17528 RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical Model Original Image Degraded Image Belief Propagation Exact Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. MSE: 1512 MSE: 325 MSE:315 Lowpass Filter Wiener Filter Median Filter MSE: 411 MSE: 545 MSE: 447 RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical Model Original Image Degraded Image Belief Propagation Exact Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. MSE: 1529 MSE: 260 MSE236 Lowpass Filter Wiener Filter Median Filter MSE: 224 MSE: 372 MSE: 244 RC2005 (19 July, 2005, Sendai)

Extension of Belief Propagation Generalized Belief Propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). Generalized belief propagation is equivalent to the cluster variation method in statistical mechanics R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951). T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957). RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical Model MSE Belief Propagation 327 0.000611 36.302 -5.19201 Generalized Belief Propagation 315 0.000758 37.909 -5.21172 Exact 0.000759 37.919 -5.21444 Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. MSE Belief Propagation 260 0.000574 33.998 -5.15241 Generalized Belief Propagation 236 0.000652 34.971 -5.17256 Exact 34.975 -5.17528 RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical Model and Conventional Filters MSE Belief Propagation 327 Lowpass Filter (3x3) 388 (5x5) 413 Generalized Belief Propagation 315 Median Filter 486 445 Exact Wiener Filter 864 548 Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. GBP (3x3) Lowpass (5x5) Median (5x5) Wiener RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical Model and Conventional Filters MSE Belief Propagation 260 Lowpass Filter (3x3) 241 (5x5) 224 Generalized Belief Propagation 236 Median Filter 331 244 Exact Wiener Filter 703 372 Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. GBP (5x5) Lowpass (5x5) Median (5x5) Wiener RC2005 (19 July, 2005, Sendai)

Contents Introduction Belief Propagation Bayesian Image Analysis and Gaussian Graphical Model Image Segmentation Concluding Remarks RC2005 (19 July, 2005, Sendai)

Image Segmentation by Gauss Mixture Model RC2005 (19 July, 2005, Sendai)

Image Segmentation by Combining Gauss Mixture Model with Potts Model Belief Propagation Potts Model RC2005 (19 July, 2005, Sendai)

Image Segmentation Belief Propagation Original Image Histogram Gauss Mixture Model Gauss Mixture Model and Potts Model Histogram Belief Propagation RC2005 (19 July, 2005, Sendai)

Motion Detection a Segmentation b Detection AND c Segmentation Gauss Mixture Model and Potts Model with Belief Propagation RC2005 (19 July, 2005, Sendai)

Contents Introduction Belief Propagation Bayesian Image Analysis and Gaussian Graphical Model Image Segmentation Concluding Remarks RC2005 (19 July, 2005, Sendai)

Summary Formulation of belief propagation Accuracy of belief propagation in Bayesian image analysis by means of Gaussian graphical model (Comparison between the belief propagation and exact calculation) Some applications of Bayesian image analysis and belief propagation RC2005 (19 July, 2005, Sendai)

Related Problem Statistical Performance Spin Glass Theory H. Nishimori: Statistical Physics of Spin Glasses and Information Processing: An Introduction, Oxford University Press, Oxford, 2001. RC2005 (19 July, 2005, Sendai)

確率的情報処理の動向 田中和之・樺島祥介編著, “ミニ特集/ベイズ統計・統計力学と情報処理”, 計測と制御 2003年8月号． 田中和之，田中利幸，渡辺治 他著，“連載/確率的情報処理と統計力学 ～様々なアプローチとそのチュートリアル～”，数理科学2004年11月号から開始． 田中和之，岡田真人，堀口剛 他著，“小特集/確率を手なづける秘伝の計算技法 ～古くて新しい確率・統計モデルのパラダイム～”，電子情報通信学会誌 2005年9月号 RC2005 (19 July, 2005, Sendai)