1. Physical Fluctuomatics 7th~10th Belief propagation
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
Physics Fluctuomatics (Tohoku University)

2. Physics Fluctuomatics (Tohoku University)
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 8.
Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese), Chapters 6-9.
Physics Fluctuomatics (Tohoku University)

3. What is an important point in computational complexity?
How should we treat the calculation of the summation over 2N configuration?
If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.
N fold loops
Markov Chain Monte Carlo Method
Belief Propagation Method
This Talk
Physics Fluctuomatics (Tohoku University)

4. Probabilistic Model and Belief Propagation
Bayesian Networks
Bayes Formulas
Probabilistic Models
Probabilistic Information Processing
Belief Propagation
J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).
C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996).
Physics Fluctuomatics (Tohoku University)

5. Mathematical Formulation of Belief Propagation
Similarity of Mathematical Structures between Mean Field Theory and Bepief Propagation
Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding
corrupted messages, Europhys. Lett. 44 (1998).
M. Opper and D. Saad (eds), Advanced Mean Field Methods
---Theory and　Practice (MIT Press, 2001).
Generalization of Belief Propagation
S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy
approximations and generalized belief propagation algorithms,
IEEE Transactions on Information Theory, 51 (2005).
Interpretations of Belief Propagation based on Information Geometry
S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy,
and information geometry, Neural Computation, 16 (2004).
Physics Fluctuomatics (Tohoku University)

6. Physics Fluctuomatics (Tohoku University)
Generalized Extensions of Belief Propagation based on Cluster Variation Method
Generalized Belief Propagation
J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).
Key Technology is the cluster variation method in Statistical Physics
R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951).
T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957).
Physics Fluctuomatics (Tohoku University)

7. Belief Propagation in Statistical Physics
In graphical models with tree graphical structures, Bethe approximation is equivalent to Transfer Matrix Method in Statistical Physics and give us exact results for computations of statistical quantities.
In Graphical Models with Cycles, Belief Propagation is equivalent to Bethe approximation or Cluster Variation Method.
Bethe
Approximation
Trandfer Matrix
Method
(Tree Structures)
Belief Propagation
Cluster Variation
Method
(Kikuchi
Approximation)
Generalized Belief Propagation
Physics Fluctuomatics (Tohoku University)

8. Applications of Belief Propagations
Image Processing
K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002).
A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).
Low Density Parity Check Codes
Y. Kabashima and D. Saad: Statistical mechanics of　low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004).
S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004).
CDMA Multiuser Detection Algorithm
Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003).
T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005).
Satisfability Problem
O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265　(2001).
M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).
Physics Fluctuomatics (Tohoku University)

9. Physics Fluctuomatics (Tohoku University)
Strategy of Approximate Algorithm in Probabilistic Information Processing
It is very hard to compute marginal probabilities exactly except some tractable cases.
What is the tractable cases in which marginal probabilities can be computed exactly?
Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases?
Physics Fluctuomatics (Tohoku University)

10. Graphical Representations of Tractable Probabilistic Models= X X A B X B C C D D E = A B C D E
Physics Fluctuomatics (Tohoku University)

11. Graphical Representations of Tractable Probabilistic ModelsX B C D E
Physics Fluctuomatics (Tohoku University)

12. Graphical Representations of Tractable Probabilistic ModelsX A B B C D E A B B C D E
Physics Fluctuomatics (Tohoku University)

13. Graphical Representations of Tractable Probabilistic ModelsX A B B C D E A B B C D E A B
Physics Fluctuomatics (Tohoku University)

14. Graphical Representations of Tractable Probabilistic ModelsX A B B C D E A B B C D E A B A B C D E
Physics Fluctuomatics (Tohoku University)

15. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

16. Graphical Representations of Tractable Probabilistic ModelsX A B C C D E
Physics Fluctuomatics (Tohoku University)

17. Graphical Representations of Tractable Probabilistic ModelsX A B C C D E X A B C C D E
Physics Fluctuomatics (Tohoku University)

18. Graphical Representations of Tractable Probabilistic ModelsX A B C C D E X A B C C D E B C
Physics Fluctuomatics (Tohoku University)

19. Graphical Representations of Tractable Probabilistic ModelsX A B C C D E X A B C C D E B C B C D E
Physics Fluctuomatics (Tohoku University)

20. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

21. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

22. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

23. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

24. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

25. Graphical Representations of Tractable Probabilistic Models= X X X A C B C C E D E X E F A D = C E B F
Physics Fluctuomatics (Tohoku University)

26. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

27. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

28. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

29. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

30. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

31. Graphical Representations of Tractable Probabilistic Models
Physics Fluctuomatics (Tohoku University)

32. Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A D C E B F
Physics Fluctuomatics (Tohoku University)

33. Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A D C E B F A D E ＝ C E E F B
Physics Fluctuomatics (Tohoku University)

34. Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A D C E B F A D E ＝ C E E F B D E ＝ C E E F
Physics Fluctuomatics (Tohoku University)

35. Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A D C E B F A D E ＝ C E E F B D D E ＝ ＝ C E C E E F F
Physics Fluctuomatics (Tohoku University)

36. Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A D C E B F A C D E ＝ C E C B E F A D A C D E ＝ ＝ C E C E C B E F B F
Physics Fluctuomatics (Tohoku University)

37. Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A D D ＝ ＝ C E C E B F F E C D F A B A
Recursion Formulas for Messages C E C E B
Physics Fluctuomatics (Tohoku University)

38. Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A D C E
Step 1 B F A C B C E D E F
Step 2 E C D F A B E C
Step 3 A B E C E C D F E C D F A B E C
Physics Fluctuomatics (Tohoku University)

39. Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages B C ＝ A B E C D F
Step 1 E C D F ＝
Step 2 A B E C D F A B E C D F ＝ A B E C D F A B E C ＝
Step 3
Physics Fluctuomatics (Tohoku University)

40. Physics Fluctuomatics (Tohoku University)
Belief Propagation
Probabilistic Models with no Cycles 1 2 3 4 5 6
Physics Fluctuomatics (Tohoku University)

41. Physics Fluctuomatics (Tohoku University)
Belief Propagation
Probabilistic Model on Tree Graph 1 2 3 4 5 6 1 2 4 3 6 5
Physics Fluctuomatics (Tohoku University)

42. Probabilistic Model on Tree Graph1 2 3 4 5 6
Physics Fluctuomatics (Tohoku University)

43. Physics Fluctuomatics (Tohoku University)
Belief Propagation
Probabilistic Model on Tree Graph 1 2 3 4 5 6
Physics Fluctuomatics (Tohoku University)

44. Belief Propagation for Probabilistic Model on Tree Graph
No Cycles!!
Physics Fluctuomatics (Tohoku University)

45. Belief Propagation for Probabilistic Model on Square Grid Graph
E: Set of all the links
Physics Fluctuomatics (Tohoku University)

46. Belief Propagation for Probabilistic Model on Square Grid Graph
Physics Fluctuomatics (Tohoku University)

47. Belief Propagation for Probabilistic Model on Square Grid Graph
Physics Fluctuomatics (Tohoku University)

48. Physics Fluctuomatics (Tohoku University)
Marginal Probability
Physics Fluctuomatics (Tohoku University)

49. Physics Fluctuomatics (Tohoku University)
Marginal Probability 2
Physics Fluctuomatics (Tohoku University)

50. Physics Fluctuomatics (Tohoku University)
Marginal Probability 2 2
Physics Fluctuomatics (Tohoku University)

51. Physics Fluctuomatics (Tohoku University)
Marginal Probability
Physics Fluctuomatics (Tohoku University)

52. Physics Fluctuomatics (Tohoku University)
Marginal Probability 1 2
Physics Fluctuomatics (Tohoku University)

53. Physics Fluctuomatics (Tohoku University)
Marginal Probability 1 2 1 2
Physics Fluctuomatics (Tohoku University)

54. Belief Propagation for Probabilistic Model on Square Grid Graph
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Physics Fluctuomatics (Tohoku University)

55. Belief Propagation for Probabilistic Model on Square Grid Graph
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions. 1 4 5 3 2 6 8 7
Physics Fluctuomatics (Tohoku University)

56. Belief Propagation for Probabilistic Model on Square Grid Graph
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions. 2 1 7 6 8 1 4 5 3 2 6 8 7
Physics Fluctuomatics (Tohoku University)

57. Belief Propagation for Probabilistic Model on Square Grid Graph3 2 1 5 4
Message Update Rule
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions. 2 1 7 6 8 1 4 5 3 2 6 8 7
Physics Fluctuomatics (Tohoku University)

58. Belief Propagation for Probabilistic Model on Square Grid Graph3 2 1 5 4 2 1 3 4 5
Fixed Point Equations for Messages
Physics Fluctuomatics (Tohoku University)

59. Fixed Point Equation and Iterative Method
Physics Fluctuomatics (Tohoku University)

60. Fixed Point Equation and Iterative Method
Physics Fluctuomatics (Tohoku University)

61. Fixed Point Equation and Iterative Method
Physics Fluctuomatics (Tohoku University)

62. Fixed Point Equation and Iterative Method
Physics Fluctuomatics (Tohoku University)

63. Fixed Point Equation and Iterative Method
Physics Fluctuomatics (Tohoku University)

64. Fixed Point Equation and Iterative Method
Physics Fluctuomatics (Tohoku University)

65. Fixed Point Equation and Iterative Method
Physics Fluctuomatics (Tohoku University)

66. Belief Propagation for Probabilistic Model on Square Grid Graph
Four Kinds of Update Rule with Three Inputs and One Output
Physics Fluctuomatics (Tohoku University)

67. Interpretation of Belief Propagation based on Information Theory
Kullback-Leibler Divergence
Free Energy
Physics Fluctuomatics (Tohoku University)

68. Interpretation of Belief Propagation based on Information Theory
Free Energy
KL Divergence
Physics Fluctuomatics (Tohoku University)

69. Interpretation of Belief Propagation based on Information Theory
KL Divergence
Free Energy
Bethe Free
Energy
Physics Fluctuomatics (Tohoku University)

70. Interpretation of Belief Propagation based on Information Theory
Physics Fluctuomatics (Tohoku University)

71. Interpretation of Belief Propagation based on Information Theory
Lagrange Multipliers to ensure the constraints
Physics Fluctuomatics (Tohoku University)

72. Interpretation of Belief Propagation based on Information Theory
Extremum Condition
Physics Fluctuomatics (Tohoku University)

73. Interpretation of Bethe Approximation (7)
Extremum
Condition
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Physics Fluctuomatics (Tohoku University)

74. Interpretation of Belief Propagation based on Information Theory
Extremum
Condition 1 4 2 5 3 1 4 5 3 2 6 8 7
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Physics Fluctuomatics (Tohoku University)

75. Interpretation of Belief Propagation based on Information Theory
Message Update Rule 1 4 2 5 3 1 4 5 3 2 6 8 7
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Physics Fluctuomatics (Tohoku University)

76. Interpretation of Belief Propagation based on Information Theory
Message Passing Rule of Belief Propagation 1 4 5 3 2 6 8 7
The reducibility conditions can be rewritten as the following fixed point equations.
This fixed point equations is corresponding to the extremum condition of the Bethe free energy.
And the fixed point equations can be numerically solved by using the natural iteration.
The algorithm is corresponding to the loopy belief propagation. 1 3 4 2 5 1 4 2 5 3 =
Physics Fluctuomatics (Tohoku University)

77. Graphical Representations for Probabilistic Models
Probability distribution with one random variable is assigned to a graph with one node. 1
Node
Probability distribution with two random variables is assigned to a edge. 1 2
Edge
Hyper
-edge 3 1 2
Physics Fluctuomatics (Tohoku University)

78. Bayesian Network and Graphical Model
More practical probabilistic models are expressed in terms of a product of functions and is assigned to chain, tree, cycle or hyper-graph representation. 1 2 3
Chain 4
Tree 1 2 3
Cycle 3 1 2 3 1 2 4 5
Hyper
-graph
Physics Fluctuomatics (Tohoku University)

79. Graphical Representations of Tractable Probabilistic Models= X X X A A B C D E F G H I =
Physics Fluctuomatics (Tohoku University)

80. Graphical Representations of Tractable Probabilistic Models= A B C D E F G H I A B C C D E F G D E H I = x x x =
Physics Fluctuomatics (Tohoku University)

81. Graphical Representations of Tractable Probabilistic Models= A B C D E A B C C D E = x =
Physics Fluctuomatics (Tohoku University)

82. Graphical Representations of Tractable Probabilistic Modelsx x x A B C C D E x A B C A B C C D E C D E F G D E H I x x x x F G C D E C D E F G D E H I x x D C E H
Physics Fluctuomatics (Tohoku University) I

83. Physics Fluctuomatics (Tohoku University)
Belief Propagation on Hypergraph Representations in terms of Cactus Tree C D E F G H I F G D F G D E H I E H I A B C D E A B C A B C A B C D E F G H I
Update Flow of Messages in computing the marginal probability Pr{C}
Physics Fluctuomatics (Tohoku University)

84. Physics Fluctuomatics (Tohoku University)
Interpretation of Belief Propagation for Hypergraphs based on Information Theory
We consider hypergraphs which satisfy
Cactus Tree
V: Set of all the nodes
E: Set of all the hyperedges
Hypergraph
Physics Fluctuomatics (Tohoku University)

85. Interpretation of Belief Propagation based on Information Theory
Kullback-Leibler Divergence
Free Energy
Physics Fluctuomatics (Tohoku University)

86. Interpretation of Belief Propagation based on Information Theory
Free Energy
KL Divergence
Physics Fluctuomatics (Tohoku University)

87. Interpretation of Belief Propagation based on Information Theory
KL Divergence
Free Energy
Bethe Free
Energy
Physics Fluctuomatics (Tohoku University)

88. Interpretation of Belief Propagation based on Information Theory
Physics Fluctuomatics (Tohoku University)

89. Interpretation of Belief Propagation based on Information Theory
Lagrange Multipliers to ensure the constraints
Physics Fluctuomatics (Tohoku University)

90. Interpretation of Belief Propagation based on Information Theory
Extremum Condition
Physics Fluctuomatics (Tohoku University)

91. Interpretation of Belief Propagation based on Information Theory
Extremum
Condition
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Physics Fluctuomatics (Tohoku University)

92. Physics Fluctuomatics (Tohoku University)
Summary
Belief Propagation and Message Passing Rule
Interpretation of Belief Propagation in the stand point of Information Theory
Future Talks
11th Probabilistic image processing by means of physical models 12th Bayesian network and belief propagation in statistical inference
Physics Fluctuomatics (Tohoku University)

93. Physics Fluctuomatics (Tohoku University)
Practice 9-1
We consider a probability distribution P(a,b,c,d,x,y) defined by
Show that marginal Probability
is expressed by
Physics Fluctuomatics (Tohoku University)

94. Physics Fluctuomatics (Tohoku University)
Practice 9-2
By substituting
to 　　　　　　　　　　　　, derive the following equation.
Physics Fluctuomatics (Tohoku University)

95. Physics Fluctuomatics (Tohoku University)
Practice 9-3
Make a program to solve the nonlinear equation x=tanh(Cx) for various values of C. Obtain the solutions for C=0.5, 1.0, 2.0 numerically. Discuss how the iterative procedures converge to the fixed points of the equations in the cases of C=0.5, 1.0, 2.0 by drawing the graphs of y=tanh(Cx) and y=x.
Physics Fluctuomatics (Tohoku University)