3 March, 2003University of Glasgow1 Statistical-Mechanical Approach to Probabilistic Inference --- Cluster Variation Method and Generalized Loopy Belief Propagation --- Kazuyuki Tanaka Graduate School of Information Sciences Tohoku University, Japan Kazuyuki Tanaka Graduate School of Information Sciences Tohoku University, Japan

3 March, 2003University of Glasgow2 Contents 1. Introduction 2. Probabilistic Inference 3. Cluster Variation Method 4. Generalized Loopy Belief Propagation 5. Linear Response 6. Numerical Experiments 7. Concluding Remarks

3 March, 2003University of Glasgow3 Introduction Probabilistic Inference and Belief Propagation Probabilistic InferenceProbabilistic Model Bayes Formula Belief Marginal Probability Belief Propagation Probabilistic models on tree-like networks with no loops =>Exact Results Probabilistic models on networks with some loops =>Good Approximation Generalization

3 March, 2003University of Glasgow4 Introduction Statistical Mechanics and Belief Propagation Belief Propagation Probabilistic model with no loop Probabilistic model with some loops (Lauritzen, Pearl) Probabilistic model with no loop Transfer Matrix Recursion Formula for Beliefs and Messages Probabilistic model with some loops Bethe/Kikuchi Method Cluster Variation Method Transfer Matrix = Belief Propagation

3 March, 2003University of Glasgow5 Introduction Purpose Review of generalized loopy belief propagation and cluster variation method. Calculation of correlations between any pair of nodes by combining the cluster variation method with the linear response theory

3 March, 2003University of Glasgow6 Probabilistic Inference

3 March, 2003University of Glasgow7 Probabilistic Inference Probabilistic Inference and Probabilistic Model

3 March, 2003University of Glasgow8 Probabilistic Inference Probabilistic Inference and Probabilistic Model

3 March, 2003University of Glasgow9 Probabilistic Inference Probabilistic Model and Beliefs Statistics: Marginal Probability Statistical Mechanics: One-body distribution Probabilistic Inference: Belief

3 March, 2003University of Glasgow10 Cluster Variation Method Probabilistic Inference and Probabilistic Model

3 March, 2003University of Glasgow11 Cluster Variation Method

3 March, 2003University of Glasgow12 Generalized Loopy Belief Propagation Extreme Condition of Kullback-Leibler Divergence Message Passing Algorithm

3 March, 2003University of Glasgow13 Generalized Loopy Belief Propagation Expression of Marginal Probability in terms of Messages

3 March, 2003University of Glasgow14 Generalized Loopy Belief Propagation

3 March, 2003University of Glasgow15 Linear Response

3 March, 2003University of Glasgow16 Numerical Experiments Beliefs Cluster Variation Method Exact

3 March, 2003University of Glasgow17 Numerical Experiments Correlation Functions Cluster Variation Method Exact

3 March, 2003University of Glasgow18 Concluding Remarks Summary Review of Cluster Variation Method and Loopy Belief Propagation Calculation of Correlation Function by means of Linear Response Theory and Cluster Variation Method Future Problem Learning of Hyperparameters by means of Maximum Likelihood Estimation