1. Graduate School of Information Sciences, Tohoku University
Physical Fluctuomatics Applied Stochastic Process 6th Graphical model and physical model
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

2. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5.
References
H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001.
H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011.
M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

3. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
System of a lot of elements with mutual relation Common Concept between Information Sciences and Physics
０,1
Bit
Data
Data is constructed from many bits
A sequence is formed by deciding the arrangement of bits.
Main Interests
Information Processing:
Data
Physics:
Material,
Natural Phenomena
A lot of elements have mutual relation of each other
Some physical concepts in Physical models are useful for the design of computational models in probabilistic information processing.
Material
Molecule
Materials are constructed from a lot of molecules.
Molecules have interactions of each other.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

4. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Why is a physical viewpoint effective in probabilistic information processing?
Matrials are constructed from a lot of molecules.
(1023 molecules exist in 1 mol.)
Molecules have intermolecular forces of each other
Theoretical physicists always have to treat such multiple summation.
Development of Approximate Methods
Probabilistic information processing is also usually reduced to multiple summations or integrations.
Application of physical approximate methods
to probabilistic information processing
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

5. Probabilistic Model for Ferromagnetic Materials
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

6. Probabilistic Model for Ferromagnetic Materials
> =
>
Prior probability prefers to the configuration with the least number of red lines.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

7. More is different in Probabilistic Model for Ferromagnetic Materials
Small p
Large p
Sampling by Markov Chain Monte Carlo method
Disordered State
Ordered State
Critical Point
(Large fluctuation)
More is different.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

8. Fundamental Probabilistic Models for Magnetic Materials+1 -1
Since h is positive, the probablity of up spin is larger than the one of down spin．
Average
h ：External Field
Variance
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

9. Fundamental Probabilistic Models for Magnetic Materials+1 +1 -1 -1
J ：Interaction -1 +1 +1 -1
Since J is positive, (a1,a2)=(+1,+1) and (-1,-1) have the largest probability．
Average
Variance
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

10. Fundamental Probabilistic Models for Magnetic Materials
E：Set of All the neighbouring Pairs of Nodes J h
Translational Symmetry
Problem: Compute
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

11. Fundamental Probabilistic Models for Magnetic MaterialsJ h
Translational Symmetry
Problem: Compute
Spontaneous Magnetization
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

12. Mean Field Approximation for Ising Model
We assume that the probability for configurations satisfying
are large. i Jm h
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

13. Mean Field Approximation for Ising Model
We assume that all random variables ai are independent of each other, approximately.
Fixed Point Equation of m
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

14. Fixed Point Equation and Iterative Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

15. Fixed Point Equation and Iterative Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

16. Fixed Point Equation and Iterative Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

17. Fixed Point Equation and Iterative Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

18. Fixed Point Equation and Iterative Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

19. Fixed Point Equation and Iterative Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

20. Fixed Point Equation and Iterative Method
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

21. Marginal Probability Distribution in Mean Field ApproximationJm h
Jm：Mean Field
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

22. Advanced Mean Field Method
l：Effective Field
Bethe Approximation l h J
Fixed Point Equation for l
Kikuchi Method (Cluster Variation Meth)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

23. Average of Ising Model on Square Grid GraphJ h
Mean Field Approximation
Bethe Approximation
Kikuchi Method (Cluster Variation Method)
Exact Solution （L. Onsager，C.N.Yang）
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

24. Model Representation in Statistical Physics
Gibbs Distribution
Partition Function
Energy Function
Free Energy
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

25. Gibbs Distribution and Free Energy
Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a)
Free Energy Functional of Trial Probability Distribution Q(a)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

26. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional
Normalization Condition
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

27. Kullback-Leibler Divergence and Free Energy
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

28. Interpretation of Mean Field Approximation as Information Theory
Minimization of Kullback-Leibler Divergence between
and
Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P]
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

29. Interpretation of Mean Field Approximation as Information TheoryJ h
Translational Symmetry
Problem: Compute
Magnetization
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

30. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Kullback-Leibler Divergence in Mean Field Approximation for Ising Model
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

31. Minimization of Kullback-Leibler Divergence and Mean Field Equation
Set of all the neighbouring nodes of the node i
Variation i
Fixed Point Equations for {Qi|iV}
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

32. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

33. Conventional Mean Field Equation in Ising Modelh h J J
Translational Symmetry
Fixed Point Equation
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

34. Interpretation of Bethe Approximation (1)
Translational Symmetry J h
Compute
and
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

35. Interpretation of Bethe Approximation (2)
Free Energy
KL Divergence
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

36. Interpretation of Bethe Approximation (3)
KL Divergence
Free Energy
Bethe Free
Energy
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

37. Interpretation of Bethe Approximation (4)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

38. Interpretation of Bethe Approximation (5)
Lagrange Multipliers to ensure the constraints
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

39. Interpretation of Bethe Approximation (6)
Extremum Condition
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

40. Interpretation of Bethe Approximation (7)
Extremum
Condition
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

41. Interpretation of Bethe Approximation (8)
Extremum
Condition 1 4 2 5 3 1 4 5 3 2 6 8 7
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

42. Interpretation of Bethe Approximation (9)
Message Update Rule 1 4 2 5 3 1 5 3 2 6 8 7
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions. 4
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

43. Interpretation of Bethe Approximation (10)
Message Passing Rule of Belief Propagation 1 4 5 3 2 6 8 7
The reducibility conditions can be rewritten as the following fixed point equations.
This fixed point equations is corresponding to the extremum condition of the Bethe free energy.
And the fixed point equations can be numerically solved by using the natural iteration.
The algorithm is corresponding to the loopy belief propagation. 1 3 4 2 5 1 4 2 5 3 =
It corresponds to Bethe approximation in the statistical mechanics.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

44. Interpretation of Bethe Approximation (11)
The reducibility conditions can be rewritten as the following fixed point equations.
This fixed point equations is corresponding to the extremum condition of the Bethe free energy.
And the fixed point equations can be numerically solved by using the natural iteration.
The algorithm is corresponding to the loopy belief propagation.
Translational Symmetry
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

45. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)
Summary
Statistical Physics and Information Theory
Probabilistic Model of Ferromagnetism
Mean Field Theory
Gibbs Distribution and Free Energy
Free Energy and Kullback-Leibler Divergence
Interpretation of Mean Field Approximation as Information Theory
Interpretation of Bethe Approximation as Information Theory
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)