Presentation is loading. Please wait.

Presentation is loading. Please wait.

29 June, 2006 Kyoto University 1 画像処理における確率伝搬法と EM アルゴリズムの統計的性能評価 東北大学大学院情報科学研究科田中和之 Reference 田中和之 : ガウシアングラフィカルモデルにもとづく確率的情報処理におけ.

Published byElmer Ross Modified over 8 years ago

Similar presentations

Presentation on theme: "29 June, 2006 Kyoto University 1 画像処理における確率伝搬法と EM アルゴリズムの統計的性能評価 東北大学大学院情報科学研究科田中和之 Reference 田中和之 : ガウシアングラフィカルモデルにもとづく確率的情報処理におけ."— Presentation transcript:

1 29 June, 2006 Kyoto University 1 画像処理における確率伝搬法と EM アルゴリズムの統計的性能評価 東北大学大学院情報科学研究科田中和之 http://www.smapip.is.tohoku.ac.jp/~kazu/ Reference 田中和之 : ガウシアングラフィカルモデルにもとづく確率的情報処理におけ る一般化された信念伝搬法, 電子情報通信学会論文誌 (D-II), Vol.J88-D-II, No.12, pp.2368-2379, 2005 共同研究者 : D. M. Titterington (University of Glasgow) 皆川まりか ( 東北大 )

2 29 June, 2006 Kyoto University 2 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

3 29 June, 2006Kyoto University3 MRF, Belief Propagation and Statistical Performance Geman and Geman (1986): IEEE Transactions on PAMI Image Processing for Markov Random Fields (MRF) (Simulated Annealing, Line Fields) Nishimori and Wong (1999): Physical Review E Statistical Performance Estimation for MRF (Infinite Range Model and Replica Theory) (Infinite Range Model and Replica Theory) Tanaka and Morita (1995): Physics Letters A Cluster Variation Method for MRF in Image Processing Cluster Variation Method (CVM) = Generalized Belief Propagation (GBP) Is it possible to estimate the performance of loopy belief propagation statistically?

4 29 June, 2006 Kyoto University 4 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

5 29 June, 2006 Kyoto University 5 Bayesian Image Restoration Original Image Degraded Image transmission Noise

6 29 June, 2006Kyoto University6 Bayes Formula and Probabilistic Image Processing Original ImageDegraded Image Prior Probability Posterior Probability Degradation Process Pixel

7 29 June, 2006 Kyoto University 7 Prior Probability in Probabilistic Image Processing Samples are generated by MCMC. Markov Chain Monte Carlo Method

8 29 June, 2006Kyoto University8 Degradation Process Additive White Gaussian Noise Histogram of Gaussian Random Numbers

9 29 June, 2006 Kyoto University 9 Degradation Process and Prior Degradation Process Prior Probability Density Function Posterior Probability Density Function Multi-Dimensional Gaussian Integral Formula

10 29 June, 2006Kyoto University10 Statistical Performance by Sample Average Prior Probability Degradation Process Posterior Probability

11 29 June, 2006Kyoto University11 Statistical Performance Analysis Prior Probability Degradation Process Posterior Probability

12 29 June, 2006Kyoto University12 Statistical Performance Analysis Multi-Dimentional Gaussian Integral Formula Nishimori (2000)

13 29 June, 2006 Kyoto University 13 Probabilistic Image Processing Marginalized Marginal Likelihood Posterior Probability Density Function

14 29 June, 2006 Kyoto University 14 Marginal Likelihood in Probabilistic Image Processing Marginalized Marginal Likelihood

15 29 June, 2006 Kyoto University 15 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Marginal Likelihood Iterate the following EM-steps until convergence: EM Algorithm Q-Function A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).

16 29 June, 2006 Kyoto University 16 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Marginal Likelihood Incomplete Data Equivalent Q-Function Pixel

17 29 June, 2006 Kyoto University 17 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

18 29 June, 2006 Kyoto University 18 Statistical Behaviour of EM (Expectation Maximization) Algorithm Numerical Experiments for Standard Image Statistical Behaviour of EM Algorithm

19 29 June, 2006 Kyoto University 19 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

20 29 June, 2006Kyoto University20 Belief Propagation and Markov Random Field ２ １ ３ ４ ５ Graphical Model with Cycles ２ １ ３ ４ ５ Fixed Point Equation Marginal Probability

21 29 June, 2006Kyoto University21 Gaussian Graphical Model and Loopy Belief Propagation Y. Weiss and W. T. Freeman, Correctness of belief propagation in Gaussian graphical models of arbitrary topology, Neural Computation, 13, 2173 (2001). K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing, J. Phys. A, Math. & Gen., 37, 8675 (2004). Dynamics of Algorithm in LBP? Loopy Belief Propagation for Gaussian Graphical Model Statistical Analysis

22 29 June, 2006Kyoto University22 Kullback-Leibler Divergence of Gaussian Graphical Model Entropy Term

23 29 June, 2006Kyoto University23 Loopy Belief Propagation Trial Function Tractable Form

24 29 June, 2006Kyoto University24 Loopy Belief Propagation Trial Function Marginal Distribution of GGM is also GGM

25 29 June, 2006Kyoto University25 Loopy Belief Propagation Bethe Free Energy in GGM

26 29 June, 2006Kyoto University26 Loopy Belief Propagation V ii and V ij do not depend on pixel i and link ij ２ １ ３ ４ ５

27 29 June, 2006 Kyoto University 27 Iteration Procedure Fixed Point Equation Iteration

28 29 June, 2006 Kyoto University 28 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Loopy Belief Propagation Exact

29 29 June, 2006 Kyoto University 29 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

30 29 June, 2006 Kyoto University 30 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Statistical Behaviour of EM Algorithm Numerical Experiments for Standard Image

31 29 June, 2006 Kyoto University 31 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

32 29 June, 2006Kyoto University32 Generalized Belief Propagation Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms J. S. Yedidia, W. T. Freeman and Y. Weiss: Transactions on Information Theory 2005. Generalized Belief Propagation for Gaussian Graphical Model K. Tanaka: IEICE Transactions on Information and Systems 2005.

33 29 June, 2006Kyoto University33 Generalized Belief Propagation Cluster: Set of nodes 12 34 12 34 1 3 2 4 Example: System consisting of 4 nodes Every subcluster of the element of B does not belong to B.

34 29 June, 2006Kyoto University34 Selection of B in LBP and GBP 123 456 789 12 45 1 4 2 5 23 56 3 6 78 4 7 5 8 89 6 9 12 4 5 23 5 6 45 7 8 56 8 9 LBP (Bethe Approx. ） GBP (Square Approx. in CVM)

35 29 June, 2006Kyoto University35 Selection of B and C in Loopy Belief Propagation LBP (Bethe Approx. ） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters

36 29 June, 2006Kyoto University36 Selection of B and C in Generalized Belief Propagation GBP (Square Approximation in CVM ） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters

37 29 June, 2006Kyoto University37 Generalized Belief Propagation Trial Function Marginal Distribution of GGM is also GGM

38 29 June, 2006Kyoto University38 Generalized Belief Propagation ２ １ ３ ４ ５

39 29 June, 2006 Kyoto University 39 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Loopy Belief Propagation Exact Generalized Belief Propagation

40 29 June, 2006 Kyoto University 40 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

41 29 June, 2006 Kyoto University 41 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

42 29 June, 2006 Kyoto University 42 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Statistical Behaviour of EM Algorithm Numerical Experiments for Standard Image

43 29 June, 2006Kyoto University43 Image Restoration by Gaussian Graphical Model Original Image MSE:314MSE:328 MSE:604 MSE: 1511 Degraded Image LBP Mean Field Method Exact SolutionGBP MSE: 314 TAP MSE:318

44 29 June, 2006Kyoto University44 Image Restoration by Gaussian Graphical Model Original Image MSE:236MSE:260 MSE: 565 MSE: 1529 Degraded Image BP Mean Field Method Exact SolutionGBP MSE:236 TAP MSE:248

45 29 June, 2006Kyoto University45 Image Restoration by Gaussian Graphical Model MSE MF604 0.00026 4 27.150-5.1335 LBP328 0.00060 0 36.328-5.1916 TAP318 0.00065 4 37.035-5.2040 GBP314 0.00071 3 37.610-5.2126 Exact314 0.00071 3 37.618-5.2158 MSE MF565 0.00029 3 26.353 - 5.09121 LBP260 0.00057 4 33.998 - 5.15241 TAP248 0.00061 0 34.475 - 5.16297 GBP236 0.00065 2 34.971 - 5.17256 Exact236 0.00065 2 34.975 - 5.17528

46 29 June, 2006Kyoto University46 Image Restoration by Gaussian Graphical Model and Conventional Filters MSE MF604 Lowpass Filter (3x3)386 LBP328(5x5)405 TAP318 Median Filter (3x3)491 GBP314(5x5)448 Exact314 Wiener Filter (3x3)863 (5x5)551 GBP (3x3) Lowpass (5x5) Median (5x5) Wiener

47 29 June, 2006Kyoto University47 Image Restoration by Gaussian Graphical Model and Conventional Filters MSE MF565 Lowpass Filter (3x3)241 LBP260(5x5)224 TAP248 Median Filter (3x3)331 GBP236(5x5)244 Exact236 Wiener Filter (3x3)703 (5x5)372 GBP (5x5) Lowpass (5x5) Median (5x5) Wiener

48 29 June, 2006 Kyoto University 48 Contents 1. Introduction 2. Gaussian Graphical Model and EM Algorithm 3. Loopy Belief Propagation 4. Generalized Belief Propagation 5. Concluding Remarks

49 29 June, 2006Kyoto University49 Summary Statistical Analysis of EM Algorithm in Generalized Belief Propagation for Gaussian Graphical Model Future Problems General Scheme of Statistical Analysis for EM Algorithm with Generalized Belief Propagation. CVM for spin glass models may be useful.

50 29 June, 2006Kyoto University50 Markov Chain Monte Carlo Method x(t)x(t)x(t+1) w(x(t+1)|x(t))

51 29 June, 2006Kyoto University51 Frequency XiXi Marginal Probabilities can be estimated from histograms. Markov Chain Monte Carlo Method

52 29 June, 2006Kyoto University52 Markov Chain Monte Carlo Method MCMC (τ=50) MCMC (τ=1) Exact Non-Synchronized Update Numerical Experiments for Standard Image

Download ppt "29 June, 2006 Kyoto University 1 画像処理における確率伝搬法と EM アルゴリズムの統計的性能評価 東北大学大学院情報科学研究科田中和之 Reference 田中和之 : ガウシアングラフィカルモデルにもとづく確率的情報処理におけ."

Similar presentations

Exact Inference in Bayes Nets

Exact Inference in Bayes Nets
Loopy Belief Propagation a summary. What is inference? Given: –Observabled variables Y –Hidden variables X –Some model of P(X,Y) We want to make some.

Loopy Belief Propagation a summary. What is inference? Given: –Observabled variables Y –Hidden variables X –Some model of P(X,Y) We want to make some.
Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.

Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.
3 March, 2003University of Glasgow1 Statistical-Mechanical Approach to Probabilistic Inference --- Cluster Variation Method and Generalized Loopy Belief.

3 March, 2003University of Glasgow1 Statistical-Mechanical Approach to Probabilistic Inference --- Cluster Variation Method and Generalized Loopy Belief.
Belief Propagation by Jakob Metzler. Outline Motivation Pearl’s BP Algorithm Turbo Codes Generalized Belief Propagation Free Energies.

Belief Propagation by Jakob Metzler. Outline Motivation Pearl’s BP Algorithm Turbo Codes Generalized Belief Propagation Free Energies.
Belief Propagation on Markov Random Fields Aggeliki Tsoli.

Belief Propagation on Markov Random Fields Aggeliki Tsoli.
1 Bayesian Image Modeling by Generalized Sparse Markov Random Fields and Loopy Belief Propagation Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan.

1 Bayesian Image Modeling by Generalized Sparse Markov Random Fields and Loopy Belief Propagation Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan.
Understanding Belief Propagation and its Applications Dan Yuan June 2004.

Understanding Belief Propagation and its Applications Dan Yuan June 2004.
24 November, 2011National Tsin Hua University, Taiwan1 Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing.

24 November, 2011National Tsin Hua University, Taiwan1 Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing.
Some Surprises in the Theory of Generalized Belief Propagation Jonathan Yedidia Mitsubishi Electric Research Labs (MERL) Collaborators: Bill Freeman (MIT)

Some Surprises in the Theory of Generalized Belief Propagation Jonathan Yedidia Mitsubishi Electric Research Labs (MERL) Collaborators: Bill Freeman (MIT)
1 物理フラクチュオマティクス論 Physical Fluctuomatics 応用確率過程論 Applied Stochastic Process 第 5 回グラフィカルモデルによる確率的情報処理 5th Probabilistic information processing by means of.

1 物理フラクチュオマティクス論 Physical Fluctuomatics 応用確率過程論 Applied Stochastic Process 第 5 回グラフィカルモデルによる確率的情報処理 5th Probabilistic information processing by means of.
Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Appendix Kazuyuki Tanaka Graduate School of Information.

Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Appendix Kazuyuki Tanaka Graduate School of Information.
1 October, 2007 ALT&DS2007 (Sendai, Japan ） 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate School of Information.

1 October, 2007 ALT&DS2007 (Sendai, Japan ） 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate School of Information.
Introduction to MCMC and BUGS. Computational problems More parameters -> even more parameter combinations Exact computation and grid approximation become.

Introduction to MCMC and BUGS. Computational problems More parameters -> even more parameter combinations Exact computation and grid approximation become.
1 Physical Fluctuomatics 5th and 6th Probabilistic information processing by Gaussian graphical model Kazuyuki Tanaka Graduate School of Information Sciences,

1 Physical Fluctuomatics 5th and 6th Probabilistic information processing by Gaussian graphical model Kazuyuki Tanaka Graduate School of Information Sciences,
3 September, 2009 SSP2009, Cardiff, UK 1 Probabilistic Image Processing by Extended Gauss-Markov Random Fields Kazuyuki Tanaka Kazuyuki Tanaka, Muneki.

3 September, 2009 SSP2009, Cardiff, UK 1 Probabilistic Image Processing by Extended Gauss-Markov Random Fields Kazuyuki Tanaka Kazuyuki Tanaka, Muneki.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki.

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki.
Markov Random Fields Probabilistic Models for Images

Markov Random Fields Probabilistic Models for Images
28 February, 2003University of Glasgow1 Cluster Variation Method and Probabilistic Image Processing -- Loopy Belief Propagation -- Kazuyuki Tanaka Graduate.

28 February, 2003University of Glasgow1 Cluster Variation Method and Probabilistic Image Processing -- Loopy Belief Propagation -- Kazuyuki Tanaka Graduate.
Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 1st Review of probabilistic information processing Kazuyuki.

Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 1st Review of probabilistic information processing Kazuyuki.

Similar presentations

Ads by Google