24 November, 2011National Tsin Hua University, Taiwan1 Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University, Sendai, Japan

24 November, 2011National Tsin Hua University, Taiwan2 Contents 1.Introduction 2.Bayesian Statistics 3.Probabilistic Image Processing 4.Gaussian Graphical Model 5.Belief Propagation 6.Various Applications of Probabilistic Information Processing 7.Summary

National Tsin Hua University, Taiwan 3 Computational model for information processing in data with uncertainty Probabilistic Inference Probabilistic model with graphical structure （ Bayesian network ） Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness. Inference system for data with uncertainty modeling Node is random variable. Arrow is conditional probability. Mathematical expression of uncertainty =>Probability and Statistics Graph with cycles Important aspect 24 November, 2011

National Tsin Hua University, Taiwan 4 Computational Model for Probabilistic Information Processing Probabilistic Information Processing Probabilistic Model Bayes Formula Algorithm Monte Carlo Method Markov Chain Monte Carlo Method Randomized Algorithm Approximate Method Belief Propagation Variational Bayes Method Expectation Propagation Randomness and Approximation 24 November, 2011

National Tsin Hua University, Taiwan5 Contents 1.Introduction 2.Bayesian Statistics 3.Probabilistic Image Processing 4.Gaussian Graphical Model 5.Belief Propagation 6.Various Applications of Probabilistic Information Processing 7.Summary

24 November, 2011National Tsin Hua University, Taiwan 6 Joint Probability and Conditional Probability a b Conditional Probability of Event A=a when Event B=b has happened. Probability of Event A=a Joint Probability of Events A=a and B=b Random Variable State Variable Probability Distribution Joint Probability Distribution

24 November, 2011National Tsin Hua University, Taiwan 7 Joint Probability and Independency of Events a b In this case, the conditional probability can be expressed as Events A and B are independent of each other a b

24 November, 2011National Tsin Hua University, Taiwan 8 Marginal Probability Let us suppose that the sample space  is expressed by Ω= (A=0) ∪ (A=1) ∪ … ∪ (A=M  1) where every pair of events is exclusive of each other. Marginal Probability of Event B=b in Joint Probability Pr{A=a,B=b} Marginalize a b Simplified Notation Summation over all the possible events in which every pair of events are exclusive of each other. ab = Message Graph with Two Nodes and One Edge

24 November, Marginal Probabiilty of High- Dimentional Joint Probabilty Marginalization with respect to c and d a b cd Hyperedge ab c d Hypergraph Message ab cd Hyperedge ab c d Hypergraph Message National Tsin Hua University, Taiwan Marginalization with respect to a, c and d

24 November, 2011National Tsin Hua University, Taiwan 10 Bayes Formulas a b Prior Probability Posterior Probability Marginal Likelihood Bayesian Network

24 November, 2011National Tsin Hua University, Taiwan11 Contents 1.Introduction 2.Bayesian Statistics 3.Probabilistic Image Processing 4.Gaussian Graphical Model 5.Belief Propagation 6.Various Applications of Probabilistic Information Processing 7.Summary

National Tsin Hua University, Taiwan 12 Image Restoration by Probabilistic Model Original Image Degraded Image Transmission Noise Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability. Bayes Formula 24 November, 2011

National Tsin Hua University, Taiwan 13 Prior Probability in Probabilistic Image Processing xixi xjxj x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 x7x7 x8x8 x9x9 x 10 x 11 x 12 xixi xjxj State Variable of Light Intensity at i-th Pixel in Original Image xixi

24 November, 2011National Tsin Hua University, Taiwan 14 Additive White Gaussian Noise Conditional Probability of Degradation Process X1X1 X2X2 X3X3 X4X4 X5X5 X6X6 X7X7 X8X8 X9X9 X 10 X 11 X 12 y1y1 y2y2 y3y3 y4y4 y5y5 y6y6 y7y7 y8y8 y9y9 y 10 y 11 y 12 xixi State Variable of Light Intensity at i-th Pixel in Original Image xixi yiyi State Variable of Light Intensity at i-th Pixel in Original Image yiyi xixi yiyi

24 November, 2011National Tsin Hua University, Taiwan 15 Bayesian Image Analysis Original Image Degraded Image Prior Probability Posterior Probability Degradation Process Image Processing is reduced to computations of avereages, variance at each pixel and covariances of each pair of neghbouring pixels

24 November, 2011 National Tsin Hua University, Taiwan 16 Statistical Estimation of Hyperparameters Marginalized with respect to X Original Image Marginal Likelihood Degraded Image Hyperparameters  are determined  so as to maximize the marginal likelihood Pr{Y=y| ,  } with respect to ,  EM (Expectation Maximization) Algorithm

24 November, 2011National Tsin Hua University, Taiwan17 Contents 1.Introduction 2.Bayesian Statistics 3.Probabilistic Image Processing 4.Gaussian Graphical Model 5.Belief Propagation 6.Various Applications of Probabilistic Information Processing 7.Summary

24 November, 2011National Tsin Hua University, Taiwan 18 Gaussian Graphical Model (Gauss Markov Random Fields) Multidimensional Gauss Integral Formulas Maximum Likelihood Estimation  EM Algorithm

24 November, 2011National Tsin Hua University, Taiwan 19 One-Dimensional Signal Processing EM Algorithm Original Signal Degraded Signal Estimated Signal

24 November, 2011 National Tsin Hua University, Taiwan 20 Bayesian Image Analysis by Gaussian Graphical Model Iteration Procedure of EM algorithm in Gaussian Graphical Model EM

24 November, 2011National Tsin Hua University, Taiwan 21 Image Restoration by Gaussian Graphical Model and Conventional Filters MSE Gaussian Graphical Model315 Lowpass Filter (3x3)388 (5x5)413 Median Filter (3x3)486 (5x5)445 (3x3) Lowpass (5x5) Median Gaussian Graphical Model Original Image Degraded Image (  =40) V:Set of all the pixels

24 November, 2011National Tsin Hua University, Taiwan22 Contents 1.Introduction 2.Bayesian Statistics 3.Probabilistic Image Processing 4.Gaussian Graphical Model 5.Belief Propagation 6.Various Applications of Probabilistic Information Processing 7.Summary

National Tsin Hua University, Taiwan 23 What is an important point in computational complexity? How should we treat the calculation of the summation over 2 N configuration? N fold loops If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40. Markov Chain Monte Carlo Method Belief Propagation Method This Talk 24 November, 2011

National Tsin Hua University, Taiwan 24 Strategy of Approximate Algorithm in Probabilistic Information Processing It is very hard to compute marginal probabilities exactly except some tractable cases. What is the tractable cases in which marginal probabilities can be computed exactly? Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases? 24 November, 2011

National Tsin Hua University, Taiwan 25 Graphical Representations of Tractable Models

24 November, 2011National Tsin Hua University, Taiwan 26 Graphical Representations of Tractable Models abcde ab bcde X ab bcde a b a bcde

24 November, 2011National Tsin Hua University, Taiwan 27 Graphical Representations of Tractable Models cde X cde b c a bcde a bc a bc b cde X

24 November, 2011National Tsin Hua University, Taiwan 28 Graphical Representations of Tractable Models a bcde b cde abcde c de d e

24 November, 2011National Tsin Hua University, Taiwan 29 Graphical Representations of Tractable Models a b e c d f a b e c d f a b e c d f e c d f e c d f e f

24 November, 2011National Tsin Hua University, Taiwan 30 Belief Propagation for Probabilistic Model on Square Grid Graph E : Set of all the links

24 November, 2011National Tsin Hua University, Taiwan 31 Marginal Probability 2 2

24 November, 2011National Tsin Hua University, Taiwan 32 Marginal Probability

24 November, 2011National Tsin Hua University, Taiwan 33 Belief Propagation Message Update Rule

24 November, 2011National Tsin Hua University, Taiwan 34 Belief Propagation on Graph with Cycles Simultaneous Fixed Point Equations of Messages Average, variances and covariances can be expressed in terms of messages

24 November, 2011 National Tsin Hua University, Taiwan 35 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

24 November, 2011National Tsin Hua University, Taiwan 36 Fundamental Structures of Belief Propagation in Probabilistic Image Processing Three Inputs and One Output Message Passing Rules

24 November, 2011National Tsin Hua University, Taiwan 37 Belief Propagation and EM Algorithm Input Output BPEM Update Rule of BP ２ １ ３ ４ ５

24 November, 2011 National Tsin Hua University, Taiwan 38 Maximization of Marginal Likelihood by EM Algorithm Loopy Belief Propagation Exact

24 November, National Tsin Hua University, Taiwan Image Restoration by Gaussian Graphical Model Original Image MSE:315 MSE: 545 MSE: 447 MSE: 411 MSE: 1512 Degraded Image Lowpass Filter Median Filter Exact Wiener Filter Belief Propagation MSE:325

24 November, 2011National Tsin Hua University, Taiwan 40 Digital Images Inpainting based on MRF Input Output Markov Random Fields M. Yasuda, J. Ohkubo and K. Tanaka: Proceedings of CIMCA&IAWTIC2005.

24 November, 2011National Tsin Hua University, Taiwan41 Contents 1.Introduction 2.Bayesian Statistics 3.Probabilistic Image Processing 4.Gaussian Graphical Model 5.Belief Propagation 6.Various Applications of Probabilistic Information Processing 7.Summary

24 November, 2011National Tsin Hua University, Taiwan 42 Belief Propagation for Bayesian Networks

24 November, 2011National Tsin Hua University, Taiwan 43 Factor Graph Representations of Bayesian Networks and Belief Propagations

National Tsin Hua University, Taiwan 44 Error Correcting Code Y. Kabashima and D. Saad: J. Phys. A, vol.37, High Performance Decoding Algorithm code 010 error decode Parity Check Code Turbo Code, Low Density Parity Check (LDPC) Code majority rule Error Correcting Codes 24 November, 2011

National Tsin Hua University, Taiwan 45 Error Correcting Codes and Belief Propagation Received Word Code Word Binary Symmetric Channel

24 November, 2011National Tsin Hua University, Taiwan 46 Error Correcting Codes and Belief Propagation Fundamental Concept for Turbo Codes and LDPC Codes

24 November, 2011National Tsin Hua University, Taiwan 47 Satisfactory Problem (3-SAT)

24 November, 2011National Tsin Hua University, Taiwan48 Contents 1.Introduction 2.Bayesian Statistics 3.Probabilistic Image Processing 4.Gaussian Graphical Model 5.Belief Propagation 6.Statistical Performance Analysis 7.Various Applications of Probabilistic Information Processing 8.Summary

24 November, 2011National Tsin Hua University, Taiwan49 Summary Fundamental Structures of Bayesian modeling have been introduced. Formulation of probabilistic image processing algorithms by means of loopy belief propagation has been summarized. Various applications of Bayesian Network Systems have been reviewed.

24 November, 2011National Tsin Hua University, Taiwan50 References 1.K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp , M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp , K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no , pp.1-9, January K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no , pp.1-18, M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp , S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no , 2010.

24 November, 2011National Tsin Hua University, Taiwan51 Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese). Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., 2009 (in Japanese).