Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Kazuyuki Tanaka Graduate School of Information Sciences,

Slides:

Advertisements

Similar presentations

Bayesian Belief Propagation

Advertisements

Loopy Belief Propagation a summary. What is inference? Given: –Observabled variables Y –Hidden variables X –Some model of P(X,Y) We want to make some.

Convergent Message-Passing Algorithms for Inference over General Graphs with Convex Free Energies Tamir Hazan, Amnon Shashua School of Computer Science.

Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.

3 March, 2003University of Glasgow1 Statistical-Mechanical Approach to Probabilistic Inference --- Cluster Variation Method and Generalized Loopy Belief.

Belief Propagation by Jakob Metzler. Outline Motivation Pearl’s BP Algorithm Turbo Codes Generalized Belief Propagation Free Energies.

Graduate School of Information Sciences, Tohoku University

1 Bayesian Image Modeling by Generalized Sparse Markov Random Fields and Loopy Belief Propagation Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan.

24 November, 2011National Tsin Hua University, Taiwan1 Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing.

Some Surprises in the Theory of Generalized Belief Propagation Jonathan Yedidia Mitsubishi Electric Research Labs (MERL) Collaborators: Bill Freeman (MIT)

1 物理フラクチュオマティクス論 Physical Fluctuomatics 応用確率過程論 Applied Stochastic Process 第 5 回グラフィカルモデルによる確率的情報処理 5th Probabilistic information processing by means of.

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 7th “More is different” and.

Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Appendix Kazuyuki Tanaka Graduate School of Information.

1 October, 2007 ALT&DS2007 (Sendai, Japan ） 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate School of Information.

1 Physical Fluctuomatics 5th and 6th Probabilistic information processing by Gaussian graphical model Kazuyuki Tanaka Graduate School of Information Sciences,

3 September, 2009 SSP2009, Cardiff, UK 1 Probabilistic Image Processing by Extended Gauss-Markov Random Fields Kazuyuki Tanaka Kazuyuki Tanaka, Muneki.

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki.

28 February, 2003University of Glasgow1 Cluster Variation Method and Probabilistic Image Processing -- Loopy Belief Propagation -- Kazuyuki Tanaka Graduate.

Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 1st Review of probabilistic information processing Kazuyuki.

10 December, 2008 CIMCA2008 (Vienna) 1 Statistical Inferences by Gaussian Markov Random Fields on Complex Networks Kazuyuki Tanaka, Takafumi Usui, Muneki.

Survey Propagation. Outline Survey Propagation: an algorithm for satisfiability 1 – Warning Propagation – Belief Propagation – Survey Propagation Survey.

September 2007 IW-SMI2007, Kyoto 1 A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model Kazuyuki Tanaka Graduate School of Information.

Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe Theoretical Analysis of Accuracy of Gaussian Belief Propagation.

Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 12th Bayesian network and belief propagation in statistical inference Kazuyuki Tanaka.

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)1 Statistical performance analysis by loopy belief propagation in probabilistic image processing Kazuyuki.

Readings: K&F: 11.3, 11.5 Yedidia et al. paper from the class website

29 December, 2008 National Tsing Hua University, Taiwan 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate.

Phisical Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm Kazuyuki Tanaka Graduate School.

Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics 1st Review of probabilistic information processing Kazuyuki Tanaka Graduate School of.

Belief Propagation and its Generalizations Shane Oldenburger.

Graduate School of Information Sciences, Tohoku University

@ 15/7/2003 Tokyo Institute of Technology 1 Propagating beliefs in spin- glass models Yoshiyuki Kabashima Dept. of Compt. Intel. & Syst.

Markov Networks: Theory and Applications Ying Wu Electrical Engineering and Computer Science Northwestern University Evanston, IL 60208

Physical Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 13th Quantum-mechanical extensions of probabilistic information processing Kazuyuki.

30 November, 2005 CIMCA2005, Vienna 1 Statistical Learning Procedure in Loopy Belief Propagation for Probabilistic Image Processing Kazuyuki Tanaka Graduate.

29 June, 2006 Kyoto University 1 画像処理における確率伝搬法と EM アルゴリズムの統計的性能評価東北大学大学院情報科学研究科田中和之 Reference 田中和之 : ガウシアングラフィカルモデルにもとづく確率的情報処理におけ.

ICPR2004 (24 July, 2004, Cambridge) 1 Probabilistic image processing based on the Q-Ising model by means of the mean- field method and loopy belief propagation.

10 October, 2007 University of Glasgow 1 EM Algorithm with Markov Chain Monte Carlo Method for Bayesian Image Analysis Kazuyuki Tanaka Graduate School.

Graduate School of Information Sciences, Tohoku University

Physical Fluctuomatics 13th Quantum-mechanical extensions of probabilistic information processing Kazuyuki Tanaka Graduate School of Information Sciences,

Statistical-Mechanical Approach to Probabilistic Image Processing -- Loopy Belief Propagation and Advanced Mean-Field Method -- Kazuyuki Tanaka and Noriko.

Graduate School of Information Sciences, Tohoku University

Sublinear Computational Time Modeling in Statistical Machine Learning Theory for Markov Random Fields Kazuyuki Tanaka GSIS, Tohoku University, Sendai,

Graduate School of Information Sciences, Tohoku University, Japan

Graduate School of Information Sciences, Tohoku University

Graduate School of Information Sciences, Tohoku University

Graduate School of Information Sciences Tohoku University, Japan

Generalized Belief Propagation

量子情報処理にむけてのクラスター変分法と確率伝搬法の定式化

Graduate School of Information Sciences, Tohoku University

一般化された確率伝搬法の数学的構造東北大学大学院情報科学研究科田中和之

Cluster Variation Method for Correlation Function of Probabilistic Model with Loopy Graphical Structure Kazuyuki Tanaka Graduate School of Information.

Graduate School of Information Sciences, Tohoku University

Graduate School of Information Sciences

Physical Fluctuomatics 7th~10th Belief propagation

Expectation-Maximization & Belief Propagation

Graduate School of Information Sciences, Tohoku University

Advanced Mean Field Methods in Quantum Probabilistic Inference

Probabilistic image processing and Bayesian network

Probabilistic image processing and Bayesian network

Graduate School of Information Sciences, Tohoku University

Graduate School of Information Sciences

Probabilistic image processing and Bayesian network

Readings: K&F: 11.3, 11.5 Yedidia et al. paper from the class website

Cluster Variation Method for Correlation Function of Probabilistic Model with Loopy Graphical Structure Kazuyuki Tanaka Graduate School of Information.

Graduate School of Information Sciences, Tohoku University

Graduate School of Information Sciences, Tohoku University

Graduate School of Information Sciences, Tohoku University

Kazuyuki Tanaka Graduate School of Information Sciences

Presentation transcript:

Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University

Physics Fluctuomatics (Tohoku University) 2 Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese), Chapter 8. Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese), Chapters 6-9.

Physics Fluctuomatics (Tohoku University) 3 What is an important point in computational complexity? How should we treat the calculation of the summation over 2 N configuration? N fold loops If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40. Markov Chain Monte Carlo Method Belief Propagation Method This Talk

Physics Fluctuomatics (Tohoku University) 4 Probabilistic Model and Belief Propagation Probabilistic Information Processing Probabilistic Models Bayes Formulas Belief Propagation J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988). C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996). Bayesian Networks

Physics Fluctuomatics (Tohoku University) 5 Mathematical Formulation of Belief Propagation Similarity of Mathematical Structures between Mean Field Theory and Bepief Propagation Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and Practice (MIT Press, 2001). Generalization of Belief Propagation S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). Interpretations of Belief Propagation based on Information Geometry S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004).

Physics Fluctuomatics (Tohoku University) 6 Generalized Extensions of Belief Propagation based on Cluster Variation Method Generalized Belief Propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free- energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). Key Technology is the cluster variation method in Statistical Physics R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951). T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957).

Physics Fluctuomatics (Tohoku University) 7 Belief Propagation in Statistical Physics In graphical models with tree graphical structures, Bethe approximation is equivalent to Transfer Matrix Method in Statistical Physics and give us exact results for computations of statistical quantities. In Graphical Models with Cycles, Belief Propagation is equivalent to Bethe approximation or Cluster Variation Method. Bethe Approximation Trandfer Matrix Method (Tree Structures) Belief Propagation Cluster Variation Method (Kikuchi Approximation) Generalized Belief Propagation

Physics Fluctuomatics (Tohoku University) 8 Applications of Belief Propagations Image Processing K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002). A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002). Low Density Parity Check Codes Y. Kabashima and D. Saad: Statistical mechanics of low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004). CDMA Multiuser Detection Algorithm Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003). T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005). Satisfability Problem O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265 (2001). M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).

Physics Fluctuomatics (Tohoku University) 9 Strategy of Approximate Algorithm in Probabilistic Information Processing It is very hard to compute marginal probabilities exactly except some tractable cases. What is the tractable cases in which marginal probabilities can be computed exactly? Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases?

Physics Fluctuomatics (Tohoku University) 10 Graphical Representations of Tractable Probabilistic Models ABCDE ABCDE BCD X XX = =

Physics Fluctuomatics (Tohoku University) 11 Graphical Representations of Tractable Probabilistic Models ABCDE AB BCDE X

Physics Fluctuomatics (Tohoku University) 12 Graphical Representations of Tractable Probabilistic Models ABCDE AB BCDE X AB BCDE

Physics Fluctuomatics (Tohoku University) 13 Graphical Representations of Tractable Probabilistic Models ABCDE AB BCDE X AB BCDE A B

Physics Fluctuomatics (Tohoku University) 14 Graphical Representations of Tractable Probabilistic Models ABCDE AB BCDE X AB BCDE A B A BCDE

Physics Fluctuomatics (Tohoku University) 15 Graphical Representations of Tractable Probabilistic Models A BCDE

Physics Fluctuomatics (Tohoku University) 16 Graphical Representations of Tractable Probabilistic Models CDE X A BCDE A BC

Physics Fluctuomatics (Tohoku University) 17 Graphical Representations of Tractable Probabilistic Models CDE X CDE A BCDE A BC A BC X

Physics Fluctuomatics (Tohoku University) 18 Graphical Representations of Tractable Probabilistic Models CDE X CDE B C A BCDE A BC A BC X

Physics Fluctuomatics (Tohoku University) 19 Graphical Representations of Tractable Probabilistic Models CDE X CDE B C A BCDE A BC A BC B CDE X

Physics Fluctuomatics (Tohoku University) 20 Graphical Representations of Tractable Probabilistic Models ABCDE

Physics Fluctuomatics (Tohoku University) 21 Graphical Representations of Tractable Probabilistic Models A BCDE ABCDE

Physics Fluctuomatics (Tohoku University) 22 Graphical Representations of Tractable Probabilistic Models A BCDE B CDE ABCDE

Physics Fluctuomatics (Tohoku University) 23 Graphical Representations of Tractable Probabilistic Models A BCDE B CDE ABCDE C DE

Physics Fluctuomatics (Tohoku University) 24 Graphical Representations of Tractable Probabilistic Models A BCDE B CDE ABCDE C DE D E

Physics Fluctuomatics (Tohoku University) 25 Graphical Representations of Tractable Probabilistic Models ABCEE CCD X XX = = F E X A B E C D F

Physics Fluctuomatics (Tohoku University) 26 Graphical Representations of Tractable Probabilistic Models A B E C D F

Physics Fluctuomatics (Tohoku University) 27 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A C A C

Physics Fluctuomatics (Tohoku University) 28 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A B E C D F B C B C

Physics Fluctuomatics (Tohoku University) 29 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A B E C D F E C D F

Physics Fluctuomatics (Tohoku University) 30 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A B E C D F E C D F E C D F

Physics Fluctuomatics (Tohoku University) 31 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A B E C D F E C D F E C D F E F

Physics Fluctuomatics (Tohoku University) 32 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F

Physics Fluctuomatics (Tohoku University) 33 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F A B E C E D E F ＝

Physics Fluctuomatics (Tohoku University) 34 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F A B E C E D E F ＝＝ E C E D E F

Physics Fluctuomatics (Tohoku University) 35 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F A B E C E D E F ＝＝ E C E D E F E C D F ＝

Physics Fluctuomatics (Tohoku University) 36 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F A C E D E F ＝＝ E C E D E F ＝ E C B C A C B C A B E C D F

Physics Fluctuomatics (Tohoku University) 37 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F ＝ E C D F ＝ E C A B E C E C D F A B E C D F Recursion Formulas for Messages

Physics Fluctuomatics (Tohoku University) 38 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C E C E C D F E F E C D F E D E C D F E C A B E C A C A B E C B C A C A C B C B C E D E DE F E F A B E C D F Step 1 Step 2 Step 3

Physics Fluctuomatics (Tohoku University) 39 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages Step 1 Step 2 Step 3 A B E C D F A B E C D F A B E C D F A B E C D F ＝ E C D F ＝ B C ＝ A B E C ＝

Physics Fluctuomatics (Tohoku University) 40 Belief Propagation Probabilistic Models with no Cycles

Physics Fluctuomatics (Tohoku University) 41 Belief Propagation Probabilistic Model on Tree Graph

Physics Fluctuomatics (Tohoku University) 42 Probabilistic Model on Tree Graph

Physics Fluctuomatics (Tohoku University) 43 Belief Propagation Probabilistic Model on Tree Graph

Physics Fluctuomatics (Tohoku University) 44 Belief Propagation for Probabilistic Model on Tree Graph No Cycles!!

Physics Fluctuomatics (Tohoku University) 45 Belief Propagation for Probabilistic Model on Square Grid Graph E : Set of all the links

Physics Fluctuomatics (Tohoku University) 46 Belief Propagation for Probabilistic Model on Square Grid Graph

Physics Fluctuomatics (Tohoku University) 47 Belief Propagation for Probabilistic Model on Square Grid Graph

Physics Fluctuomatics (Tohoku University) 48 Marginal Probability

Physics Fluctuomatics (Tohoku University) 49 Marginal Probability 2

Physics Fluctuomatics (Tohoku University) 50 Marginal Probability 2 2

Physics Fluctuomatics (Tohoku University) 51 Marginal Probability

Physics Fluctuomatics (Tohoku University) 52 Marginal Probability 1 2

Physics Fluctuomatics (Tohoku University) 53 Marginal Probability

Physics Fluctuomatics (Tohoku University) 54 Belief Propagation for Probabilistic Model on Square Grid Graph

Physics Fluctuomatics (Tohoku University) 55 Belief Propagation for Probabilistic Model on Square Grid Graph

Physics Fluctuomatics (Tohoku University) 56 Belief Propagation for Probabilistic Model on Square Grid Graph

Physics Fluctuomatics (Tohoku University) 57 Belief Propagation for Probabilistic Model on Square Grid Graph Message Update Rule

Physics Fluctuomatics (Tohoku University) 58 Belief Propagation for Probabilistic Model on Square Grid Graph Fixed Point Equations for Messages

Physics Fluctuomatics (Tohoku University) 59 Fixed Point Equation and Iterative Method Fixed Point Equation

Physics Fluctuomatics (Tohoku University) 60 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 61 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 62 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 63 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 64 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 65 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 66 Belief Propagation for Probabilistic Model on Square Grid Graph Four Kinds of Update Rule with Three Inputs and One Output

Physics Fluctuomatics (Tohoku University) 67 Interpretation of Belief Propagation based on Information Theory Free Energy Kullback-Leibler Divergence

Physics Fluctuomatics (Tohoku University) 68 Interpretation of Belief Propagation based on Information Theory Free Energy KL Divergence

Physics Fluctuomatics (Tohoku University) 69 Interpretation of Belief Propagation based on Information Theory Bethe Free Energy Free Energy KL Divergence

Physics Fluctuomatics (Tohoku University) 70 Interpretation of Belief Propagation based on Information Theory

Physics Fluctuomatics (Tohoku University) 71 Interpretation of Belief Propagation based on Information Theory Lagrange Multipliers to ensure the constraints

Physics Fluctuomatics (Tohoku University) 72 Interpretation of Belief Propagation based on Information Theory Extremum Condition

Interpretation of Bethe Approximation (7) Extremum Condition 73 Physics Fluctuomatics (Tohoku University)

74 Interpretation of Belief Propagation based on Information Theory Extremum Condition

Physics Fluctuomatics (Tohoku University) 75 Interpretation of Belief Propagation based on Information Theory Message Update Rule

Physics Fluctuomatics (Tohoku University) 76 Interpretation of Belief Propagation based on Information Theory = Message Passing Rule of Belief Propagation

Physics Fluctuomatics (Tohoku University) Graphical Representations for Probabilistic Models Probability distribution with two random variables is assigned to a edge Node Edge Probability distribution with one random variable is assigned to a graph with one node Hyper -edge 77

Physics Fluctuomatics (Tohoku University) Hyper -graph Bayesian Network and Graphical Model Tree 31 2 Cycle More practical probabilistic models are expressed in terms of a product of functions and is assigned to chain, tree, cycle or hyper-graph representation. 123 Chain

Physics Fluctuomatics (Tohoku University) 79 Graphical Representations of Tractable Probabilistic Models X = C D E A C B D F G X E H I = A B C D E F G H I X

Physics Fluctuomatics (Tohoku University) 80 Graphical Representations of Tractable Probabilistic Models A B C D E F G H I C D E A B C F G D E H I C D E A B C F G D E H I A B C D E F G H I = =xxx=

Physics Fluctuomatics (Tohoku University) 81 Graphical Representations of Tractable Probabilistic Models A B C D E F G H I C D E A B C C D E A B C F G D E H I = =x = A B C D E

Physics Fluctuomatics (Tohoku University) 82 Graphical Representations of Tractable Probabilistic Models C D E A B C F G D E H I xxx C D E A B C x C D E F G D E H I A B C x xx C D E A B C x C D E F G D E H I x x C D E C D E F G H I

Physics Fluctuomatics (Tohoku University) 83 Belief Propagation on Hypergraph Representations in terms of Cactus Tree C D E C D E F G H I F G D E H I A B C F G D E H I A B C A B C D E A B C D E F G H I Update Flow of Messages in computing the marginal probability Pr{C}

Physics Fluctuomatics (Tohoku University) 84 Interpretation of Belief Propagation for Hypergraphs based on Information Theory We consider hypergraphs which satisfy Cactus Tree Hypergraph V: Set of all the nodes E: Set of all the hyperedges

Physics Fluctuomatics (Tohoku University) 85 Interpretation of Belief Propagation based on Information Theory Free Energy Kullback-Leibler Divergence

Physics Fluctuomatics (Tohoku University) 86 Interpretation of Belief Propagation based on Information Theory Free Energy KL Divergence

Physics Fluctuomatics (Tohoku University) 87 Interpretation of Belief Propagation based on Information Theory Free Energy KL Divergence Bethe Free Energy

Physics Fluctuomatics (Tohoku University) 88 Interpretation of Belief Propagation based on Information Theory

Physics Fluctuomatics (Tohoku University) 89 Interpretation of Belief Propagation based on Information Theory Lagrange Multipliers to ensure the constraints

Physics Fluctuomatics (Tohoku University) 90 Interpretation of Belief Propagation based on Information Theory Extremum Condition

Physics Fluctuomatics (Tohoku University) 91 Interpretation of Belief Propagation based on Information Theory Extremum Condition

Physics Fluctuomatics (Tohoku University) 92 Summary Belief Propagation and Message Passing Rule Interpretation of Belief Propagation in the stand point of Information Theory Future Talks 11th Probabilistic image processing by means of physical models 12th Bayesian network and belief propagation in statistical inference

Physics Fluctuomatics (Tohoku University) 93 Practice 9-1 We consider a probability distribution P(a,b,c,d,x,y) defined by Show that marginal Probability is expressed by

Physics Fluctuomatics (Tohoku University) 94 Practice 9-2 By substituting to, derive the following equation.

Physics Fluctuomatics (Tohoku University) 95 Practice 9-3 Make a program to solve the nonlinear equation x=tanh(Cx) for various values of C. Obtain the solutions for C=0.5, 1.0, 2.0 numerically. Discuss how the iterative procedures converge to the fixed points of the equations in the cases of C=0.5, 1.0, 2.0 by drawing the graphs of y=tanh(Cx) and y=x.