1. Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe Theoretical Analysis of Accuracy of Gaussian Belief Propagation

2. Background Belief propagation (BP) The algorithm which computes marginal distributions efficiently Marginal distribution: requires huge computational cost.

3. Variety of Research Areas (i) Probabilistic inference for AI (ii) Error correcting code (LDPC, Turbo codes) (iii) Code division multiple access (CDMA) ex. (iv) Probabilistic image processing 000101 000111000101 correcting noise degrade image restored image restore

4. Properties of BP & Loopy BP (LBP) Tree-structured target distribution Exact marginal probabilities Loop-structured target distribution Convergence? Approximate marginal probabilities Y. Weiss,”Correctness of belief propagation in graphical models with arbitrary topology”, Neural Computation 13(10), 2173-2200, 2001. T. Heskes, ”On the Uniqueness of Loopy Belief Propagation Fixed Points”, Neural Computation 16(11), 2379-2414, 2004. Ex.

5. Purpose We analytically clarify the accuracy of LBP when the target distribution is a Gaussian distribution. What is the conditions for LBP convergence? How close is the LBP solution to the true marginal distributions? K. Tanaka, H. Shouno, M. Okada, “Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing”, J.phys. A, Math. Gen., vol.37, no.36, pp.8675-8696, 2004. In Probabilistic image processing

6. Table of Contents ・ BP ＆ LBP ・ Gaussian Distribution ・ Main Results (i) Single Loop (ii) Graphs with Multi-loops ・ Conclusion

7. Graphical Models Target distribution Marginal distribution

8. BP ＆ LBP Marginal distribution

9. How are messages decided? Messages are decided by the fixed-points of a message update rule: If it converges a fixed-point

10. Gaussian Distribution Messages: Update rule: Target distribution:( Inverse covariance matrix)

11. Fixed-Points of Messages Single loop When a Gaussian distribution forms a single loop, the fixed-points of messages are given by Theorem1 where are the cofactors.

12. LBP Solution Theorem 2 The solution of LBP is given by where

13. Intuitive Understanding LBP Solution True Loop Tree Loopy Belief PropagationBelief Propagation

14. Accuracy of LBP The Kullback-Leibler (KL) distances are calculated as whereis given by Solution of LBP True marginal density Convergence condition issince Theorem 3

15. Graphs with Multi-Loops Multi-loops How about the graphs having arbitrary structures? We clarify the LBP solution at small covariances. We derive the expansionsw. r. t. where inverse covariance matrix is

16. A Fixed-Point of Inverse Variances A fixed-point of inverse variances satisfies the following system of equations: The solution of the system is expanded as Theorem 4

17. Comparison with true inverse variances Expansions of LBP solution are True inverse variances are

18. Accuracy of LBP Theorem 5 The Kullback-Leibler (KL) distances are expanded as Solution of LBP True marginal density whereare

19. Conclusion We analytically clarified the accuracy of LBP in a Gaussian distribution. (i) For a single loop, we revealed the parameter that determines the accuracy of LBP and the condition that tells us when LBP converges. (ii) For arbitrary structures, we revealed the expansions of LBP solution at small covariances and the accuracy. These fundamental results contribute to understanding the theoretical properties underlying LBP.