1. Graduate School of Information Sciences, Tohoku University
Physical Fluctuomatics Applied Stochastic Process 11th Bayesian network and belief propagation in statistical inference
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

2. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Textbooks
Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese).
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

3. Probabilistic Model and Belief Propagation
Bayesian Networks
Bayes Formulas
Probabilistic Models
Probabilistic Information Processing
Belief Propagation
J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).
C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996).
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

4. What is an important point in computational complexity?
How should we treat the calculation of the summation over 2N configuration?
If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.
N fold loops
Markov Chain Monte Carlo Method
Belief Propagation Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

5. Joint Probability and Conditional Probability
Fundamental Probabilistic Theory for Image Processing
Joint Probability and Conditional Probability
Probability of Event A=a
Joint Probability of Events A=a and B=b
Conditional Probability of Event B=b
when Event A=a has happened. A B
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

6. Fundamental Probabilistic Theory for Image Processing
Marginal Probability of Event B A B C D
Marginalization
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

7. Fundamental Probabilistic Theory for Image Processing
Causal Independence A B C A B C
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

8. Fundamental Probabilistic Theory for Image Processing
Causal Independence A B C D A B C D
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

9. Fundamental Probabilistic Theory for Image Processing
Causal Independence
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

10. Fundamental Probabilistic Theory for Image Processing
Directed Graph B B
Undirected Graph C C
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

11. Simple Example of Bayesian Networks
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

12. Simple Example of Bayesian Networks
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

13. Simple Example of Bayesian Networks
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

14. Simple Example of Bayesian Networks
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

15. Simple Example of Bayesian Networks
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

16. Simple Example of Bayesian Networks
Undirected Graph C S R W C S R W
Directed Graph
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

17. Bayesian Network and Graphical Model
Probability distribution with one random variable is assigned to a graph with one node. 1
Node
Probability distribution with two random variables is assigned to a edge. 1 2
Edge
Hyper
-edge 3 1 2
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

18. Bayesian Network and Graphical Model
Bayesian network is expressed in terms of a product of functions and is assigned to chain, tree, cycle or hyper-graph representation. 1 2 3
Chain 4
Tree 1 2 3
Cycle 3 1 2 3 1 2 4 5
Hyper
-graph
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

19. Graphical Representations of Tractable Probabilistic Models= X X X A A B C D E F G H I =
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

20. Graphical Representations of Tractable Probabilistic Models= A B C D E F G H I A B C C D E F G D E H I = x x x =
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

21. Graphical Representations of Tractable Probabilistic Models= A B C D E A B C C D E = x =
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

22. Graphical Representations of Tractable Probabilistic Modelsx x x A B C C D E x A B C A B C C D E C D E F G D E H I x x x x F G C D E C D E F G D E H I x x D C E H
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) I

23. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Belief Propagation on Hypergraph Representations in terms of Cactus Tree C D E F G H I F G D F G D E H I E H I A B C D E A B C A B C A B C D E F G H I
Update Flow of Messages in computing the marginal probability Pr{C}
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

24. Belief Propagation for Bayesian Networks
Belief propagation cannot give us exact computations in Bayesian networks on cycle graphs.
Applications of belief propagation to Bayesian networks on cycle graphs provide us many powerful approximate computational models and practical algorithms for probabilistic information processing.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

25. Simple Example of Bayesian Networks
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

26. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Joint Probability of Probabilistic Model with Graphical Representation including Cycles 1 3 2 4 6 5 8 7
Directed Graph
Undirected Hypergraph
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

27. Marginal Probability Distributions1 3 2 4 6 5 8 7
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

28. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Approximate Representations of Marginal Probability Distributions in terms of Messages 1 3 2 4 6 5 8 7 1 3 2 4 6 5 8 7
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

29. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Approximate Representations of Marginal Probability Distributions in terms of Messages 3 4 6 5 8 7 1 3 2 4 6 5 8 7
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

30. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)1 3 2 4 6 5 8 7
Basic Strategies of Belief Propagations in Probabilistic Model with Graphical Representation including Cycles 1 3 2 4 6 5 8 7 3 4 6 5 8 7
Approximate Expressions of Marginal Probabilities
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

31. Belief Propagation Algorithm
Simultaneous Fixed Pint Equations for Belief Propagations in Hypergraph Representations 6 1 3 2 4 1 3 2 4 6 5 8 7
Belief Propagation Algorithm
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

32. Fixed Point Equation and Iterative Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

33. Belief Propagation for Bayesian Networks
Belief propagation can be applied to Bayesian networks also on hypergraphs as powerful approximate algorithms.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

34. Numerical Experiments
Belief Propagation
Exact 1 3 2 4 6 5 8 7
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

35. Numerical Experiments
Belief Propagation 1 3 2 4 6 5 8 7
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

36. Linear Response Theory3 1 2 4 6 5 8 7
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

37. Numerical Experiments1 3 2 4 6 5 8 7
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

38. Interpretation of Belief Propagation based on Information Theory
We consider hypergraphs which satisfy
Cactus Tree
V: Set of all the nodes
E: Set of all the hyperedges
Hypergraph
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

39. Interpretation of Belief Propagation based on Information Theory
Kullback-Leibler Divergence
Free Energy
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

40. Interpretation of Belief Propagation based on Information Theory
Free Energy
KL Divergence
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

41. Interpretation of Belief Propagation based on Information Theory
KL Divergence
Free Energy
Bethe Free
Energy
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

42. Interpretation of Belief Propagation based on Information Theory
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

43. Interpretation of Belief Propagation based on Information Theory
Lagrange Multipliers to ensure the constraints
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

44. Interpretation of Belief Propagation based on Information Theory
Extremum Condition
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

45. Interpretation of Belief Propagation based on Information Theory
Extremum
Condition
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

46. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Summary
Bayesian Network for Probabilistic Inference
Belief Propagation for Bayesian Networks
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

47. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Practice 11-1
Compute the exact values of the marginal probability Pr{Xi} for every nodes i(=1,2,…,8), numerically, in the Bayesian network defined by the joint probability distribution Pr{X1,X2,…,X8} as follows:
Each conditional probability table and probability table is given in Figure 3.12 and Table 3.11 in
Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

48. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Practice 11-2
Make a program to compute the approximate values of the marginal probability Pr{Xi} for every nodes i(=1,2,…,8) by using the belief propagation method in the Bayesian network defined by the joint probability distribution Pr{X1,X2,…,X8} as follows:
Each conditional probability table and probability table is given in Figure 3.12 and Table 3.11 in
Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009.
The algorithm has appeared explicitly in the above textbook.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)