1. Probabilistic image processing and Bayesian network
Kazuyuki Tanaka
Graduate School of Information Sciences,
Tohoku University
References
K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, vol.35, pp.R81-R150 (2002).
K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp (2004).
RC2005 (19 July, 2005, Sendai)

2. Bayesian Network and Belief Propagation
Bayes Formula
Probabilistic Model
Probabilistic Information
Processing
Belief Propagation
J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).
C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996).
RC2005 (19 July, 2005, Sendai)

3. Formulation of Belief Propagation
Link between belief propagation and statistical mechanics.
Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998).
M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and　Practice (MIT Press, 2001).
Generalized belief propagation
J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).
Information geometrical interpretation
of belief propagation
S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004).
RC2005 (19 July, 2005, Sendai)

4. Application of Belief Propagation
Image Processing
K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002).
A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).
Low Density Parity Check Codes
Y. Kabashima and D. Saad: Statistical mechanics of　low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004).
S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004).
CDMA Multiuser Detection Algorithm
Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003).
T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005).
Satisfability Problem
O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265　(2001).
M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).
RC2005 (19 July, 2005, Sendai)

5. Contents Introduction Belief Propagation
Bayesian Image Analysis and Gaussian Graphical Model
Image Segmentation
Concluding Remarks
RC2005 (19 July, 2005, Sendai)

6. It is very hard to calculate exactly except some special cases.
Belief Propagation
How should we treat the calculation of the summation over 2N configurations.
It is very hard to calculate exactly except some special cases.
Formulation for approximate algorithm
Accuracy of the approximate algorithm
RC2005 (19 July, 2005, Sendai)

7. Tractable Model Probabilistic models with no loop are tractable.
Factorizable
Probabilistic models with loop are not tractable.
Not Factorizable
RC2005 (19 July, 2005, Sendai)

8. Probabilistic Model on a Graph with Loops
Marginal Probability
RC2005 (19 July, 2005, Sendai)

9. Message Passing Rule of Belief Propagation
The reducibility conditions can be rewritten as the following fixed point equations.
This fixed point equations is corresponding to the extremum condition of the Bethe free energy.
And the fixed point equations can be numerically solved by using the natural iteration.
The algorithm is corresponding to the loopy belief propagation. 1 3 4 2 5
Fixed Point Equations for Massage
RC2005 (19 July, 2005, Sendai)

10. Approximate Representation of Marginal Probability1 4 2 5 3
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.
Fixed Point Equations for Messages
RC2005 (19 July, 2005, Sendai)

11. Fixed Point Equation and Iterative Method
RC2005 (19 July, 2005, Sendai)

12. Contents Introduction Belief Propagation
Bayesian Image Analysis and Gaussian Graphical Model
Image Segmentation
Concluding Remarks
RC2005 (19 July, 2005, Sendai)

13. Bayesian Image Analysis
Noise
Transmission
Original Image
Degraded Image
RC2005 (19 July, 2005, Sendai)

14. Bayesian Image Analysis
Degradation Process
Additive White Gaussian Noise
Transmission
Original Image
Degraded Image
RC2005 (19 July, 2005, Sendai)

15. Bayesian Image Analysis
A Priori Probability
Generate
Standard Images
Similar?
RC2005 (19 July, 2005, Sendai)

16. Bayesian Image Analysis
A Posteriori Probability
Gaussian Graphical Model
RC2005 (19 July, 2005, Sendai)

17. Bayesian Image Analysis
A Priori Probability
Degraded Image
Degraded Image
Original Image
Pixels
A Posteriori
Probability
RC2005 (19 July, 2005, Sendai)

18. Hyperparameter Determination by Maximization of Marginal Likelihood
In the image restoration, we usually have to estimate the hyperparameters alpha and p.
In statistics, the maximum likelihood estimation is often employed.
In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image.
The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model.
We can calculate these partition functions approximately by using the Bethe approximation.
Marginalization
Degraded Image
Original Image
Marginal Likelihood
RC2005 (19 July, 2005, Sendai)

19. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm
Q-Function
In the image restoration, we usually have to estimate the hyperparameters alpha and p.
In statistics, the maximum likelihood estimation is often employed.
In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image.
The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model.
We can calculate these partition functions approximately by using the Bethe approximation.
Incomplete Data
Equivalent
RC2005 (19 July, 2005, Sendai)

20. Iterate the following EM-steps until convergence:
Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm
Marginal Likelihood
Q-Function
In the image restoration, we usually have to estimate the hyperparameters alpha and p.
In statistics, the maximum likelihood estimation is often employed.
In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image.
The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model.
We can calculate these partition functions approximately by using the Bethe approximation.
Iterate the following EM-steps until convergence:
EM Algorithm
A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data
via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).
RC2005 (19 July, 2005, Sendai)

21. One-Dimensional Signal
127
255
100
200
Original Signal
Degraded Signal
Estimated Signal
EM Algorithm
RC2005 (19 July, 2005, Sendai)

22. Image Restoration by Gaussian Graphical Model
EM Algorithm with Belief Propagation
Original Image
Degraded Image
MSE: 1512
MSE: 1529
RC2005 (19 July, 2005, Sendai)

23. Exact Results of Gaussian Graphical Model
Multi-dimensional Gauss integral formula
RC2005 (19 July, 2005, Sendai)

24. Comparison of Belief Propagation with Exact Results in Gaussian Graphical Model
MSE
Belief Propagation
327
36.302
Exact
315
37.919
Finally, we show only the results for the gray-level image restoration.
For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.
MSE
Belief Propagation
260
33.998
Exact
236
34.975
RC2005 (19 July, 2005, Sendai)

25. Image Restoration by Gaussian Graphical Model
Original Image
Degraded Image
Belief Propagation
Exact
Finally, we show only the results for the gray-level image restoration.
For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.
MSE: 1512
MSE: 325
MSE:315
Lowpass Filter
Wiener Filter
Median Filter
MSE: 411
MSE: 545
MSE: 447
RC2005 (19 July, 2005, Sendai)

26. Image Restoration by Gaussian Graphical Model
Original Image
Degraded Image
Belief Propagation
Exact
Finally, we show only the results for the gray-level image restoration.
For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.
MSE: 1529
MSE: 260
MSE236
Lowpass Filter
Wiener Filter
Median Filter
MSE: 224
MSE: 372
MSE: 244
RC2005 (19 July, 2005, Sendai)

27. Extension of Belief Propagation
Generalized Belief Propagation
J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).
Generalized belief propagation is equivalent to the cluster variation method in statistical mechanics
R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951).
T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957).
RC2005 (19 July, 2005, Sendai)

28. Image Restoration by Gaussian Graphical Model
MSE
Belief Propagation
327
36.302
Generalized Belief Propagation
315
37.909
Exact
37.919
Finally, we show only the results for the gray-level image restoration.
For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.
MSE
Belief Propagation
260
33.998
Generalized Belief Propagation
236
34.971
Exact
34.975
RC2005 (19 July, 2005, Sendai)

29. Image Restoration by Gaussian Graphical Model and Conventional Filters
MSE
Belief Propagation
327
Lowpass Filter
(3x3)
388
(5x5)
413
Generalized Belief Propagation
315
Median Filter
486
445
Exact
Wiener Filter
864
548
Finally, we show only the results for the gray-level image restoration.
For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.
GBP
(3x3) Lowpass
(5x5) Median
(5x5) Wiener
RC2005 (19 July, 2005, Sendai)

30. Image Restoration by Gaussian Graphical Model and Conventional Filters
MSE
Belief Propagation
260
Lowpass Filter
(3x3)
241
(5x5)
224
Generalized Belief Propagation
236
Median Filter
331
244
Exact
Wiener Filter
703
372
Finally, we show only the results for the gray-level image restoration.
For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.
GBP
(5x5) Lowpass
(5x5) Median
(5x5) Wiener
RC2005 (19 July, 2005, Sendai)

31. Contents Introduction Belief Propagation
Bayesian Image Analysis and Gaussian Graphical Model
Image Segmentation
Concluding Remarks
RC2005 (19 July, 2005, Sendai)

32. Image Segmentation by Gauss Mixture Model
RC2005 (19 July, 2005, Sendai)

33. Image Segmentation by Combining Gauss Mixture Model with Potts Model
Belief Propagation
Potts Model
RC2005 (19 July, 2005, Sendai)

34. Image Segmentation Belief Propagation Original Image Histogram
Gauss Mixture
Model
Gauss Mixture
Model and
Potts Model
Histogram
Belief
Propagation
RC2005 (19 July, 2005, Sendai)

35. Motion Detection a Segmentation b Detection AND c Segmentation
Gauss Mixture Model and Potts Model with Belief Propagation
RC2005 (19 July, 2005, Sendai)

36. Contents Introduction Belief Propagation
Bayesian Image Analysis and Gaussian Graphical Model
Image Segmentation
Concluding Remarks
RC2005 (19 July, 2005, Sendai)

37. Summary Formulation of belief propagation
Accuracy of belief propagation in Bayesian image analysis by means of Gaussian graphical model (Comparison between the belief propagation and exact calculation)
Some applications of Bayesian image analysis and belief propagation
RC2005 (19 July, 2005, Sendai)

38. Related Problem Statistical Performance Spin Glass Theory
H. Nishimori: Statistical Physics of Spin Glasses and Information Processing: An Introduction, Oxford University Press, Oxford, 2001.
RC2005 (19 July, 2005, Sendai)

39. 確率的情報処理の動向 田中和之・樺島祥介編著, “ミニ特集/ベイズ統計・統計力学と情報処理”, 計測と制御 2003年8月号．
田中和之，田中利幸，渡辺治 他著，“連載/確率的情報処理と統計力学 ～様々なアプローチとそのチュートリアル～”，数理科学2004年11月号から開始．
田中和之，岡田真人，堀口剛 他著，“小特集/確率を手なづける秘伝の計算技法 ～古くて新しい確率・統計モデルのパラダイム～”，電子情報通信学会誌 2005年9月号
RC2005 (19 July, 2005, Sendai)