Graduate School of Information Sciences, Tohoku University

Slides:



Advertisements
Similar presentations
CS188: Computational Models of Human Behavior
Advertisements

Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.
3 March, 2003University of Glasgow1 Statistical-Mechanical Approach to Probabilistic Inference --- Cluster Variation Method and Generalized Loopy Belief.
Belief Propagation by Jakob Metzler. Outline Motivation Pearl’s BP Algorithm Turbo Codes Generalized Belief Propagation Free Energies.
Graduate School of Information Sciences, Tohoku University
1 Bayesian Image Modeling by Generalized Sparse Markov Random Fields and Loopy Belief Propagation Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan.
Graphical Models Lei Tang. Review of Graphical Models Directed Graph (DAG, Bayesian Network, Belief Network) Typically used to represent causal relationship.
Today Logistic Regression Decision Trees Redux Graphical Models
Machine Learning CUNY Graduate Center Lecture 21: Graphical Models.
24 November, 2011National Tsin Hua University, Taiwan1 Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 7th “More is different” and.
Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Appendix Kazuyuki Tanaka Graduate School of Information.
1 October, 2007 ALT&DS2007 (Sendai, Japan ) 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate School of Information.
1 Physical Fluctuomatics 5th and 6th Probabilistic information processing by Gaussian graphical model Kazuyuki Tanaka Graduate School of Information Sciences,
3 September, 2009 SSP2009, Cardiff, UK 1 Probabilistic Image Processing by Extended Gauss-Markov Random Fields Kazuyuki Tanaka Kazuyuki Tanaka, Muneki.
Undirected Models: Markov Networks David Page, Fall 2009 CS 731: Advanced Methods in Artificial Intelligence, with Biomedical Applications.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 2nd Probability and its fundamental.
Bayesian Statistics and Belief Networks. Overview Book: Ch 13,14 Refresher on Probability Bayesian classifiers Belief Networks / Bayesian Networks.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki.
Markov Random Fields Probabilistic Models for Images
28 February, 2003University of Glasgow1 Cluster Variation Method and Probabilistic Image Processing -- Loopy Belief Propagation -- Kazuyuki Tanaka Graduate.
Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 1st Review of probabilistic information processing Kazuyuki.
10 December, 2008 CIMCA2008 (Vienna) 1 Statistical Inferences by Gaussian Markov Random Fields on Complex Networks Kazuyuki Tanaka, Takafumi Usui, Muneki.
Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe Theoretical Analysis of Accuracy of Gaussian Belief Propagation.
Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 12th Bayesian network and belief propagation in statistical inference Kazuyuki Tanaka.
14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)1 Statistical performance analysis by loopy belief propagation in probabilistic image processing Kazuyuki.
29 December, 2008 National Tsing Hua University, Taiwan 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate.
Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Kazuyuki Tanaka Graduate School of Information Sciences,
1 CMSC 671 Fall 2001 Class #21 – Tuesday, November 13.
Phisical Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm Kazuyuki Tanaka Graduate School.
Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics 1st Review of probabilistic information processing Kazuyuki Tanaka Graduate School of.
Belief Propagation and its Generalizations Shane Oldenburger.
Wei Sun and KC Chang George Mason University March 2008 Convergence Study of Message Passing In Arbitrary Continuous Bayesian.
Exact Inference in Bayes Nets. Notation U: set of nodes in a graph X i : random variable associated with node i π i : parents of node i Joint probability:
Graduate School of Information Sciences, Tohoku University
Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 2nd Probability and its fundamental properties Kazuyuki Tanaka Graduate School of Information.
Physical Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 13th Quantum-mechanical extensions of probabilistic information processing Kazuyuki.
30 November, 2005 CIMCA2005, Vienna 1 Statistical Learning Procedure in Loopy Belief Propagation for Probabilistic Image Processing Kazuyuki Tanaka Graduate.
ICPR2004 (24 July, 2004, Cambridge) 1 Probabilistic image processing based on the Q-Ising model by means of the mean- field method and loopy belief propagation.
10 October, 2007 University of Glasgow 1 EM Algorithm with Markov Chain Monte Carlo Method for Bayesian Image Analysis Kazuyuki Tanaka Graduate School.
Graduate School of Information Sciences, Tohoku University
Physical Fluctuomatics 13th Quantum-mechanical extensions of probabilistic information processing Kazuyuki Tanaka Graduate School of Information Sciences,
Statistical-Mechanical Approach to Probabilistic Image Processing -- Loopy Belief Propagation and Advanced Mean-Field Method -- Kazuyuki Tanaka and Noriko.
Graduate School of Information Sciences, Tohoku University
Graduate School of Information Sciences, Tohoku University
Sublinear Computational Time Modeling in Statistical Machine Learning Theory for Markov Random Fields Kazuyuki Tanaka GSIS, Tohoku University, Sendai,
Graduate School of Information Sciences, Tohoku University, Japan
Graduate School of Information Sciences, Tohoku University
Graduate School of Information Sciences, Tohoku University
Graduate School of Information Sciences Tohoku University, Japan
Generalized Belief Propagation
Bayesian Statistics and Belief Networks
Graduate School of Information Sciences, Tohoku University
Markov Random Fields Presented by: Vladan Radosavljevic.
Cluster Variation Method for Correlation Function of Probabilistic Model with Loopy Graphical Structure Kazuyuki Tanaka Graduate School of Information.
Graduate School of Information Sciences
Physical Fluctuomatics 7th~10th Belief propagation
Expectation-Maximization & Belief Propagation
Graduate School of Information Sciences, Tohoku University
Advanced Mean Field Methods in Quantum Probabilistic Inference
Probabilistic image processing and Bayesian network
Probabilistic image processing and Bayesian network
Graduate School of Information Sciences, Tohoku University
Graduate School of Information Sciences
Cluster Variation Method for Correlation Function of Probabilistic Model with Loopy Graphical Structure Kazuyuki Tanaka Graduate School of Information.
Graduate School of Information Sciences, Tohoku University
Graduate School of Information Sciences, Tohoku University
Graduate School of Information Sciences, Tohoku University
Kazuyuki Tanaka Graduate School of Information Sciences
Presentation transcript:

Graduate School of Information Sciences, Tohoku University Physical Fluctuomatics Applied Stochastic Process 11th Bayesian network and belief propagation in statistical inference Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Textbooks Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese). Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Probabilistic Model and Belief Propagation Bayesian Networks Bayes Formulas Probabilistic Models Probabilistic Information Processing Belief Propagation J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988). C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996). Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

What is an important point in computational complexity? How should we treat the calculation of the summation over 2N configuration? If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40. N fold loops Markov Chain Monte Carlo Method Belief Propagation Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Joint Probability and Conditional Probability Fundamental Probabilistic Theory for Image Processing Joint Probability and Conditional Probability Probability of Event A=a Joint Probability of Events A=a and B=b Conditional Probability of Event B=b when Event A=a has happened. A B Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Fundamental Probabilistic Theory for Image Processing Marginal Probability of Event B A B C D Marginalization Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Fundamental Probabilistic Theory for Image Processing Causal Independence A B C A B C Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Fundamental Probabilistic Theory for Image Processing Causal Independence A B C D A B C D Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Fundamental Probabilistic Theory for Image Processing Causal Independence Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Fundamental Probabilistic Theory for Image Processing Directed Graph B B Undirected Graph C C Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Simple Example of Bayesian Networks Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Simple Example of Bayesian Networks Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Simple Example of Bayesian Networks Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Simple Example of Bayesian Networks Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Simple Example of Bayesian Networks Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Simple Example of Bayesian Networks Undirected Graph C S R W C S R W Directed Graph Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Bayesian Network and Graphical Model Probability distribution with one random variable is assigned to a graph with one node. 1 Node Probability distribution with two random variables is assigned to a edge. 1 2 Edge Hyper -edge 3 1 2 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Bayesian Network and Graphical Model Bayesian network is expressed in terms of a product of functions and is assigned to chain, tree, cycle or hyper-graph representation. 1 2 3 Chain 4 Tree 1 2 3 Cycle 3 1 2 3 1 2 4 5 Hyper -graph Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Graphical Representations of Tractable Probabilistic Models = X X X A A B C D E F G H I = Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Graphical Representations of Tractable Probabilistic Models = A B C D E F G H I A B C C D E F G D E H I = x x x = Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Graphical Representations of Tractable Probabilistic Models = A B C D E A B C C D E = x = Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Graphical Representations of Tractable Probabilistic Models x x x A B C C D E x A B C A B C C D E C D E F G D E H I x x x x F G C D E C D E F G D E H I x x D C E H Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) I

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Belief Propagation on Hypergraph Representations in terms of Cactus Tree C D E F G H I F G D F G D E H I E H I A B C D E A B C A B C A B C D E F G H I Update Flow of Messages in computing the marginal probability Pr{C} Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Belief Propagation for Bayesian Networks Belief propagation cannot give us exact computations in Bayesian networks on cycle graphs. Applications of belief propagation to Bayesian networks on cycle graphs provide us many powerful approximate computational models and practical algorithms for probabilistic information processing. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Simple Example of Bayesian Networks Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Joint Probability of Probabilistic Model with Graphical Representation including Cycles 1 3 2 4 6 5 8 7 Directed Graph Undirected Hypergraph Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Marginal Probability Distributions 1 3 2 4 6 5 8 7 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Approximate Representations of Marginal Probability Distributions in terms of Messages 1 3 2 4 6 5 8 7 1 3 2 4 6 5 8 7 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Approximate Representations of Marginal Probability Distributions in terms of Messages 3 4 6 5 8 7 1 3 2 4 6 5 8 7 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 3 2 4 6 5 8 7 Basic Strategies of Belief Propagations in Probabilistic Model with Graphical Representation including Cycles 1 3 2 4 6 5 8 7 3 4 6 5 8 7 Approximate Expressions of Marginal Probabilities Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Belief Propagation Algorithm Simultaneous Fixed Pint Equations for Belief Propagations in Hypergraph Representations 6 1 3 2 4 1 3 2 4 6 5 8 7 Belief Propagation Algorithm Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Fixed Point Equation and Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Belief Propagation for Bayesian Networks Belief propagation can be applied to Bayesian networks also on hypergraphs as powerful approximate algorithms. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Numerical Experiments Belief Propagation Exact 1 3 2 4 6 5 8 7 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Numerical Experiments Belief Propagation 1 3 2 4 6 5 8 7 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Linear Response Theory 3 1 2 4 6 5 8 7 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Numerical Experiments 1 3 2 4 6 5 8 7 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Interpretation of Belief Propagation based on Information Theory We consider hypergraphs which satisfy Cactus Tree V: Set of all the nodes E: Set of all the hyperedges Hypergraph Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Interpretation of Belief Propagation based on Information Theory Kullback-Leibler Divergence Free Energy Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Interpretation of Belief Propagation based on Information Theory Free Energy KL Divergence Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Interpretation of Belief Propagation based on Information Theory KL Divergence Free Energy Bethe Free Energy Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Interpretation of Belief Propagation based on Information Theory Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Interpretation of Belief Propagation based on Information Theory Lagrange Multipliers to ensure the constraints Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Interpretation of Belief Propagation based on Information Theory Extremum Condition Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Interpretation of Belief Propagation based on Information Theory Extremum Condition In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel. These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels. The messages are determined so as to satisfy the reducibility conditions. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Summary Bayesian Network for Probabilistic Inference Belief Propagation for Bayesian Networks Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Practice 11-1 Compute the exact values of the marginal probability Pr{Xi} for every nodes i(=1,2,…,8), numerically, in the Bayesian network defined by the joint probability distribution Pr{X1,X2,…,X8} as follows: Each conditional probability table and probability table is given in Figure 3.12 and Table 3.11 in Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Practice 11-2 Make a program to compute the approximate values of the marginal probability Pr{Xi} for every nodes i(=1,2,…,8) by using the belief propagation method in the Bayesian network defined by the joint probability distribution Pr{X1,X2,…,X8} as follows: Each conditional probability table and probability table is given in Figure 3.12 and Table 3.11 in Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009. The algorithm has appeared explicitly in the above textbook. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)