29 June, 2006 Kyoto University 1 画像処理における確率伝搬法と EM アルゴリズムの統計的性能評価 東北大学大学院情報科学研究科田中和之 Reference 田中和之 : ガウシアングラフィカルモデルにもとづく確率的情報処理におけ る一般化された信念伝搬法, 電子情報通信学会論文誌 (D-II), Vol.J88-D-II, No.12, pp , 2005 共同研究者 : D. M. Titterington (University of Glasgow) 皆川まりか ( 東北大 )

29 June, 2006 Kyoto University 2 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

29 June, 2006Kyoto University3 MRF, Belief Propagation and Statistical Performance Geman and Geman (1986): IEEE Transactions on PAMI Image Processing for Markov Random Fields (MRF) (Simulated Annealing, Line Fields) Nishimori and Wong (1999): Physical Review E Statistical Performance Estimation for MRF (Infinite Range Model and Replica Theory) (Infinite Range Model and Replica Theory) Tanaka and Morita (1995): Physics Letters A Cluster Variation Method for MRF in Image Processing Cluster Variation Method (CVM) = Generalized Belief Propagation (GBP) Is it possible to estimate the performance of loopy belief propagation statistically?

29 June, 2006 Kyoto University 4 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

29 June, 2006 Kyoto University 5 Bayesian Image Restoration Original Image Degraded Image transmission Noise

29 June, 2006Kyoto University6 Bayes Formula and Probabilistic Image Processing Original ImageDegraded Image Prior Probability Posterior Probability Degradation Process Pixel

29 June, 2006 Kyoto University 7 Prior Probability in Probabilistic Image Processing Samples are generated by MCMC. Markov Chain Monte Carlo Method

29 June, 2006Kyoto University8 Degradation Process Additive White Gaussian Noise Histogram of Gaussian Random Numbers

29 June, 2006 Kyoto University 9 Degradation Process and Prior Degradation Process Prior Probability Density Function Posterior Probability Density Function Multi-Dimensional Gaussian Integral Formula

29 June, 2006Kyoto University10 Statistical Performance by Sample Average Prior Probability Degradation Process Posterior Probability

29 June, 2006Kyoto University11 Statistical Performance Analysis Prior Probability Degradation Process Posterior Probability

29 June, 2006Kyoto University12 Statistical Performance Analysis Multi-Dimentional Gaussian Integral Formula Nishimori (2000)

29 June, 2006 Kyoto University 13 Probabilistic Image Processing Marginalized Marginal Likelihood Posterior Probability Density Function

29 June, 2006 Kyoto University 14 Marginal Likelihood in Probabilistic Image Processing Marginalized Marginal Likelihood

29 June, 2006 Kyoto University 15 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Marginal Likelihood Iterate the following EM-steps until convergence: EM Algorithm Q-Function A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).

29 June, 2006 Kyoto University 16 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Marginal Likelihood Incomplete Data Equivalent Q-Function Pixel

29 June, 2006 Kyoto University 17 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

29 June, 2006 Kyoto University 18 Statistical Behaviour of EM (Expectation Maximization) Algorithm Numerical Experiments for Standard Image Statistical Behaviour of EM Algorithm

29 June, 2006 Kyoto University 19 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

29 June, 2006Kyoto University20 Belief Propagation and Markov Random Field ２ １ ３ ４ ５ Graphical Model with Cycles ２ １ ３ ４ ５ Fixed Point Equation Marginal Probability

29 June, 2006Kyoto University21 Gaussian Graphical Model and Loopy Belief Propagation Y. Weiss and W. T. Freeman, Correctness of belief propagation in Gaussian graphical models of arbitrary topology, Neural Computation, 13, 2173 (2001). K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing, J. Phys. A, Math. & Gen., 37, 8675 (2004). Dynamics of Algorithm in LBP? Loopy Belief Propagation for Gaussian Graphical Model Statistical Analysis

29 June, 2006Kyoto University22 Kullback-Leibler Divergence of Gaussian Graphical Model Entropy Term

29 June, 2006Kyoto University23 Loopy Belief Propagation Trial Function Tractable Form

29 June, 2006Kyoto University24 Loopy Belief Propagation Trial Function Marginal Distribution of GGM is also GGM

29 June, 2006Kyoto University25 Loopy Belief Propagation Bethe Free Energy in GGM

29 June, 2006Kyoto University26 Loopy Belief Propagation V ii and V ij do not depend on pixel i and link ij ２ １ ３ ４ ５

29 June, 2006 Kyoto University 27 Iteration Procedure Fixed Point Equation Iteration

29 June, 2006 Kyoto University 28 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Loopy Belief Propagation Exact

29 June, 2006 Kyoto University 29 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

29 June, 2006 Kyoto University 30 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Statistical Behaviour of EM Algorithm Numerical Experiments for Standard Image

29 June, 2006 Kyoto University 31 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

29 June, 2006Kyoto University32 Generalized Belief Propagation Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms J. S. Yedidia, W. T. Freeman and Y. Weiss: Transactions on Information Theory Generalized Belief Propagation for Gaussian Graphical Model K. Tanaka: IEICE Transactions on Information and Systems 2005.

29 June, 2006Kyoto University33 Generalized Belief Propagation Cluster: Set of nodes Example: System consisting of 4 nodes Every subcluster of the element of B does not belong to B.

29 June, 2006Kyoto University34 Selection of B in LBP and GBP LBP (Bethe Approx. ） GBP (Square Approx. in CVM)

29 June, 2006Kyoto University35 Selection of B and C in Loopy Belief Propagation LBP (Bethe Approx. ） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters

29 June, 2006Kyoto University36 Selection of B and C in Generalized Belief Propagation GBP (Square Approximation in CVM ） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters

29 June, 2006Kyoto University37 Generalized Belief Propagation Trial Function Marginal Distribution of GGM is also GGM

29 June, 2006Kyoto University38 Generalized Belief Propagation ２ １ ３ ４ ５

29 June, 2006 Kyoto University 39 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Loopy Belief Propagation Exact Generalized Belief Propagation

29 June, 2006 Kyoto University 40 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

29 June, 2006 Kyoto University 41 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

29 June, 2006 Kyoto University 42 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Statistical Behaviour of EM Algorithm Numerical Experiments for Standard Image

29 June, 2006Kyoto University43 Image Restoration by Gaussian Graphical Model Original Image MSE:314MSE:328 MSE:604 MSE: 1511 Degraded Image LBP Mean Field Method Exact SolutionGBP MSE: 314 TAP MSE:318

29 June, 2006Kyoto University44 Image Restoration by Gaussian Graphical Model Original Image MSE:236MSE:260 MSE: 565 MSE: 1529 Degraded Image BP Mean Field Method Exact SolutionGBP MSE:236 TAP MSE:248

29 June, 2006Kyoto University45 Image Restoration by Gaussian Graphical Model MSE MF LBP TAP GBP Exact MSE MF LBP TAP GBP Exact

29 June, 2006Kyoto University46 Image Restoration by Gaussian Graphical Model and Conventional Filters MSE MF604 Lowpass Filter (3x3)386 LBP328(5x5)405 TAP318 Median Filter (3x3)491 GBP314(5x5)448 Exact314 Wiener Filter (3x3)863 (5x5)551 GBP (3x3) Lowpass (5x5) Median (5x5) Wiener

29 June, 2006Kyoto University47 Image Restoration by Gaussian Graphical Model and Conventional Filters MSE MF565 Lowpass Filter (3x3)241 LBP260(5x5)224 TAP248 Median Filter (3x3)331 GBP236(5x5)244 Exact236 Wiener Filter (3x3)703 (5x5)372 GBP (5x5) Lowpass (5x5) Median (5x5) Wiener

29 June, 2006 Kyoto University 48 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

29 June, 2006Kyoto University49 Summary Statistical Analysis of EM Algorithm in Generalized Belief Propagation for Gaussian Graphical Model Future Problems General Scheme of Statistical Analysis for EM Algorithm with Generalized Belief Propagation. CVM for spin glass models may be useful.

29 June, 2006Kyoto University50 Markov Chain Monte Carlo Method x(t)x(t)x(t+1) w(x(t+1)|x(t))

29 June, 2006Kyoto University51 Frequency XiXi Marginal Probabilities can be estimated from histograms. Markov Chain Monte Carlo Method

29 June, 2006Kyoto University52 Markov Chain Monte Carlo Method MCMC (τ=50) MCMC (τ=1) Exact Non-Synchronized Update Numerical Experiments for Standard Image