Presentation is loading. Please wait.

Presentation is loading. Please wait.

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki.

Similar presentations


Presentation on theme: "Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki."— Presentation transcript:

1 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/

2 Physical Fuctuomatics (Tohoku University) 2 Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese), Chapter 8. Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese), Chapters 6-9.

3 Physical Fuctuomatics (Tohoku University) 3 What is an important point in computational complexity? How should we treat the calculation of the summation over 2 N configuration? N fold loops If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40. Markov Chain Monte Carlo Method Belief Propagation Method This Talk Previous Talk

4 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 4 Probabilistic Model and Belief Propagation Probabilistic Information Processing Probabilistic Models Bayes Formulas Belief Propagation J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988). C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996). Bayesian Networks

5 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 5 Mathematical Formulation of Belief Propagation Similarity of Mathematical Structures between Mean Field Theory and Bepief Propagation Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and Practice (MIT Press, 2001). Generalization of Belief Propagation S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). Interpretations of Belief Propagation based on Information Geometry S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004).

6 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 6 Generalized Extensions of Belief Propagation based on Cluster Variation Method Generalized Belief Propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free- energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). Key Technology is the cluster variation method in Statistical Physics R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951). T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957).

7 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 7 Belief Propagation in Statistical Physics In graphical models with tree graphical structures, Bethe approximation is equivalent to Transfer Matrix Method in Statistical Physics and give us exact results for computations of statistical quantities. In Graphical Models with Cycles, Belief Propagation is equivalent to Bethe approximation or Cluster Variation Method. Bethe Approximation Trandfer Matrix Method (Tree Structures) Belief Propagation Cluster Variation Method (Kikuchi Approximation) Generalized Belief Propagation

8 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 8 Applications of Belief Propagations Image Processing K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002). A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002). Low Density Parity Check Codes Y. Kabashima and D. Saad: Statistical mechanics of low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004). CDMA Multiuser Detection Algorithm Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003). T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005). Satisfability Problem O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265 (2001). M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).

9 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 9 Strategy of Approximate Algorithm in Probabilistic Information Processing It is very hard to compute marginal probabilities exactly except some tractable cases. What is the tractable cases in which marginal probabilities can be computed exactly? Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases?

10 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 10 Graphical Representations of Tractable Probabilistic Models ABCDE ABCDE BCD X XX = =

11 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 11 Graphical Representations of Tractable Probabilistic Models ABCDE AB BCDE X

12 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 12 Graphical Representations of Tractable Probabilistic Models ABCDE AB BCDE X AB BCDE

13 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 13 Graphical Representations of Tractable Probabilistic Models ABCDE AB BCDE X AB BCDE A B

14 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 14 Graphical Representations of Tractable Probabilistic Models ABCDE AB BCDE X AB BCDE A B A BCDE

15 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 15 Graphical Representations of Tractable Probabilistic Models A BCDE

16 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 16 Graphical Representations of Tractable Probabilistic Models CDE X A BCDE A BC

17 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 17 Graphical Representations of Tractable Probabilistic Models CDE X CDE A BCDE A BC A BC X

18 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 18 Graphical Representations of Tractable Probabilistic Models CDE X CDE B C A BCDE A BC A BC X

19 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 19 Graphical Representations of Tractable Probabilistic Models CDE X CDE B C A BCDE A BC A BC B CDE X

20 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 20 Graphical Representations of Tractable Probabilistic Models ABCDE

21 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 21 Graphical Representations of Tractable Probabilistic Models A BCDE ABCDE

22 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 22 Graphical Representations of Tractable Probabilistic Models A BCDE B CDE ABCDE

23 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 23 Graphical Representations of Tractable Probabilistic Models A BCDE B CDE ABCDE C DE

24 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 24 Graphical Representations of Tractable Probabilistic Models A BCDE B CDE ABCDE C DE D E

25 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 25 Graphical Representations of Tractable Probabilistic Models ABCEE CCD X XX = = F E X A B E C D F

26 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 26 Graphical Representations of Tractable Probabilistic Models A B E C D F

27 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 27 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A C A C

28 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 28 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A B E C D F B C B C

29 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 29 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A B E C D F E C D F

30 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 30 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A B E C D F E C D F E C D F

31 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 31 Graphical Representations of Tractable Probabilistic Models A B E C D F A B E C D F A B E C D F E C D F E C D F E F

32 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 32 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F

33 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 33 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F A B E C E D E F =

34 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 34 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F A B E C E D E F = = E C E D E F

35 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 35 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F A B E C E D E F = = E C E D E F E C D F =

36 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 36 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F A C E D E F = = E C E D E F = E C B C A C B C A B E C D F

37 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 37 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C D F = E C D F = E C A B E C E C D F A B E C D F Recursion Formulas for Messages

38 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 38 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A B E C E C E C D F E F E C D F E D E C D F E C A B E C A C A B E C B C A C A C B C B C E D E DE F E F A B E C D F Step 1 Step 2 Step 3

39 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 39 Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages Step 1 Step 2 Step 3 A B E C D F A B E C D F A B E C D F A B E C D F = E C D F = B C = A B E C =

40 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 40 Belief Propagation 1 2 3 4 5 6 Probabilistic Models with no Cycles

41 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 41 Belief Propagation 1 2 4 1 1 3 2 6 5 2 1 2 3 4 5 6 Probabilistic Model on Tree Graph

42 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 42 Probabilistic Model on Tree Graph 1 2 3 4 5 6

43 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 43 Belief Propagation 1 2 3 4 5 6 Probabilistic Model on Tree Graph

44 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 44 Belief Propagation for Probabilistic Model on Tree Graph No Cycles!!

45 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 45 Belief Propagation for Probabilistic Model on Square Grid Graph E : Set of all the links

46 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 46 Belief Propagation for Probabilistic Model on Square Grid Graph

47 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 47 Belief Propagation for Probabilistic Model on Square Grid Graph

48 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 48 Marginal Probability

49 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 49 Marginal Probability 2

50 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 50 Marginal Probability 2 2

51 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 51 Marginal Probability

52 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 52 Marginal Probability 1 2

53 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 53 Marginal Probability 1 2 1 2

54 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 54 Belief Propagation for Probabilistic Model on Square Grid Graph

55 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 55 Belief Propagation for Probabilistic Model on Square Grid Graph 1 4 53 2 6 8 7

56 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 56 Belief Propagation for Probabilistic Model on Square Grid Graph 2 17 6 8 1 4 53 2 6 8 7

57 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 57 Belief Propagation for Probabilistic Model on Square Grid Graph 2 17 6 8 Message Update Rule 1 4 53 2 6 8 7 3 2 1 5 4 1 2

58 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 58 Belief Propagation for Probabilistic Model on Square Grid Graph 2 1 3 4 5 3 2 1 5 4 1 2 Fixed Point Equations for Messages

59 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 59 Fixed Point Equation and Iterative Method Fixed Point Equation

60 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 60 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

61 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 61 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

62 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 62 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

63 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 63 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

64 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 64 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

65 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 65 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

66 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 66 Belief Propagation for Probabilistic Model on Square Grid Graph Four Kinds of Update Rule with Three Inputs and One Output

67 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 67 Interpretation of Belief Propagation based on Information Theory Free Energy Kullback-Leibler Divergence

68 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 68 Interpretation of Belief Propagation based on Information Theory Free Energy KL Divergence

69 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 69 Interpretation of Belief Propagation based on Information Theory Bethe Free Energy Free Energy KL Divergence

70 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 70 Interpretation of Belief Propagation based on Information Theory

71 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 71 Interpretation of Belief Propagation based on Information Theory Lagrange Multipliers to ensure the constraints

72 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 72 Interpretation of Belief Propagation based on Information Theory Extremum Condition

73 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 73 Interpretation of Belief Propagation based on Information Theory 1 42 5 3 1 4 53 2 6 8 7 Extremum Condition

74 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 74 Interpretation of Belief Propagation based on Information Theory 1 42 5 3 1 4 53 2 6 8 7 Message Update Rule

75 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 75 Interpretation of Belief Propagation based on Information Theory 1 3 42 5 1 4 5 3 2 6 8 7 1 42 53 = Message Passing Rule of Belief Propagation

76 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 76 Summary Belief Propagation Transfer Matrix Method Bethe Approximation and Cluster Variation Method Iterative Algorithm Future Talks 10th Probabilistic image processing by means of physical models 11th Bayesian network and belief propagation in statistical inference

77 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 77 Practice 9-1 We consider a probability distribution P(a,b,c,d,x,y) defined by Show that marginal Probability is expressed by

78 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 78 Practice 9-2 By substituting to, derive the following equation.

79 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 79 Practice 9-3 Make a program to solve the nonlinear equation x=tanh(Cx) for various values of C. Obtain the solutions for C=0.5, 1.0, 2.0 numerically. Discuss how the iterative procedures converge to the fixed points of the equations in the cases of C=0.5, 1.0, 2.0 by drawing the graphs of y=tanh(Cx) and y=x.


Download ppt "Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki."

Similar presentations


Ads by Google