Graduate School of Information Sciences, Tohoku University Physical Fluctuomatics Applied Stochastic Process 6th Graphical model and physical model Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5. References H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011. M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) System of a lot of elements with mutual relation Common Concept between Information Sciences and Physics ０,1 101101110001 010011101110101000111110000110000101000000111010101110101010 Bit Data Data is constructed from many bits A sequence is formed by deciding the arrangement of bits. Main Interests Information Processing: Data Physics: Material, Natural Phenomena A lot of elements have mutual relation of each other Some physical concepts in Physical models are useful for the design of computational models in probabilistic information processing. Material Molecule Materials are constructed from a lot of molecules. Molecules have interactions of each other. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Why is a physical viewpoint effective in probabilistic information processing? Matrials are constructed from a lot of molecules. (1023 molecules exist in 1 mol.) Molecules have intermolecular forces of each other Theoretical physicists always have to treat such multiple summation. Development of Approximate Methods Probabilistic information processing is also usually reduced to multiple summations or integrations. Application of physical approximate methods to probabilistic information processing Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Probabilistic Model for Ferromagnetic Materials Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Probabilistic Model for Ferromagnetic Materials > = > Prior probability prefers to the configuration with the least number of red lines. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
More is different in Probabilistic Model for Ferromagnetic Materials Small p Large p Sampling by Markov Chain Monte Carlo method Disordered State Ordered State Critical Point (Large fluctuation) More is different. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fundamental Probabilistic Models for Magnetic Materials +1 -1 Since h is positive, the probablity of up spin is larger than the one of down spin． Average h ：External Field Variance Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fundamental Probabilistic Models for Magnetic Materials +1 +1 -1 -1 J ：Interaction -1 +1 +1 -1 Since J is positive, (a1,a2)=(+1,+1) and (-1,-1) have the largest probability． Average Variance Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fundamental Probabilistic Models for Magnetic Materials E：Set of All the neighbouring Pairs of Nodes J h Translational Symmetry Problem: Compute Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fundamental Probabilistic Models for Magnetic Materials J h Translational Symmetry Problem: Compute Spontaneous Magnetization Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Mean Field Approximation for Ising Model We assume that the probability for configurations satisfying are large. i Jm h Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Mean Field Approximation for Ising Model We assume that all random variables ai are independent of each other, approximately. Fixed Point Equation of m Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fixed Point Equation and Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fixed Point Equation and Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fixed Point Equation and Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fixed Point Equation and Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fixed Point Equation and Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fixed Point Equation and Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Fixed Point Equation and Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Marginal Probability Distribution in Mean Field Approximation Jm h Jm：Mean Field Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Advanced Mean Field Method l：Effective Field Bethe Approximation l h J Fixed Point Equation for l Kikuchi Method (Cluster Variation Meth) Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Average of Ising Model on Square Grid Graph J h Mean Field Approximation Bethe Approximation Kikuchi Method (Cluster Variation Method) Exact Solution （L. Onsager，C.N.Yang） Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Model Representation in Statistical Physics Gibbs Distribution Partition Function Energy Function Free Energy Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Gibbs Distribution and Free Energy Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a) Free Energy Functional of Trial Probability Distribution Q(a) Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional Normalization Condition Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Kullback-Leibler Divergence and Free Energy Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Mean Field Approximation as Information Theory Minimization of Kullback-Leibler Divergence between and Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P] Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Mean Field Approximation as Information Theory J h Translational Symmetry Problem: Compute Magnetization Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Kullback-Leibler Divergence in Mean Field Approximation for Ising Model Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Minimization of Kullback-Leibler Divergence and Mean Field Equation Set of all the neighbouring nodes of the node i Variation i Fixed Point Equations for {Qi|iV} Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Conventional Mean Field Equation in Ising Model h h J J Translational Symmetry Fixed Point Equation Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (1) Translational Symmetry J h Compute and Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (2) Free Energy KL Divergence Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (3) KL Divergence Free Energy Bethe Free Energy Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (4) Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (5) Lagrange Multipliers to ensure the constraints Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (6) Extremum Condition Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (7) Extremum Condition In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel. These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels. The messages are determined so as to satisfy the reducibility conditions. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (8) Extremum Condition 1 4 2 5 3 1 4 5 3 2 6 8 7 In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel. These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels. The messages are determined so as to satisfy the reducibility conditions. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (9) Message Update Rule 1 4 2 5 3 1 5 3 2 6 8 7 In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel. These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels. The messages are determined so as to satisfy the reducibility conditions. 4 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (10) Message Passing Rule of Belief Propagation 1 4 5 3 2 6 8 7 The reducibility conditions can be rewritten as the following fixed point equations. This fixed point equations is corresponding to the extremum condition of the Bethe free energy. And the fixed point equations can be numerically solved by using the natural iteration. The algorithm is corresponding to the loopy belief propagation. 1 3 4 2 5 1 4 2 5 3 = It corresponds to Bethe approximation in the statistical mechanics. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (11) The reducibility conditions can be rewritten as the following fixed point equations. This fixed point equations is corresponding to the extremum condition of the Bethe free energy. And the fixed point equations can be numerically solved by using the natural iteration. The algorithm is corresponding to the loopy belief propagation. Translational Symmetry Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) Summary Statistical Physics and Information Theory Probabilistic Model of Ferromagnetism Mean Field Theory Gibbs Distribution and Free Energy Free Energy and Kullback-Leibler Divergence Interpretation of Mean Field Approximation as Information Theory Interpretation of Bethe Approximation as Information Theory Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)