1. Algorithms and Theory of
Machine Learning
Algorithms and Theory of
Approximate Inference
Eric Xing
Lecture 15, August 15, 2010
Eric Xing
© Eric CMU,
Reading:

2. Inference Problems Compute the likelihood of observed data
Compute the marginal distribution over a particular subset of nodes
Compute the conditional distribution for disjoint subsets A and B
Compute a mode of the density
Brute-force methods are intractable
Recursive message passing is efficient for simple graphical models, like trees
Sum-product/max-product
Junction
Eric Xing
© Eric CMU, 2

3. Inference in GM ... A HMM A general BN x2 x3 x1 xT y2 y3 y1 yT B A D C
Eric Xing
© Eric CMU,

4. (Forward-backward , Max-product /BP, Junction Tree)
Inference Problems
Compute the likelihood of observed data
Compute the marginal distribution over a particular subset of nodes
Compute the conditional distribution for disjoint subsets A and B
Compute a mode of the density
Methods we have
Message Passing
(Forward-backward , Max-product /BP, Junction Tree)
Brute-force methods are intractable
Recursive message passing is efficient for simple graphical models, like trees
Sum-product/max-product
Junction
Brute force
Elimination
Individual computations independent
Sharing intermediate terms
Eric Xing
© Eric CMU, 4

5. Recall forward-backward on HMM
Forward algorithm
Backward algorithm A x2 x3 x1 xT y2 y3 y1 yT
...
Eric Xing
© Eric CMU,

6. Message passing for trees
Let mij(xi) denote the factor resulting from eliminating variables from bellow up to i, which is a function of xi:
This is reminiscent of a message sent from j to i. i j
A mathematical summary of the elimination algorithm
mij(xi) represents a "belief" of xi from xj! k l
Eric Xing
© Eric CMU,

7. The General Sum-Product Algorithm
Tree-structured GMs
Message Passing on Trees:
On trees, converge to a unique fixed point after a finite number of iterations
Eric Xing
© Eric CMU,

8. Junction Tree Revisited
General Algorithm on Graphs with Cycles
Steps:
=> Triangularization
=> Construct JTs
=> Message Passing on Clique Trees B C S
Eric Xing
© Eric CMU,

9. Local Consistency
Given a set of functions associated with the cliques and separator sets
They are locally consistent if:
For junction trees, local consistency is equivalent to global consistency!
Eric Xing
© Eric CMU,

10. An Ising model on 2-D image
Nodes encode hidden information (patch-identity).
They receive local information from the image (brightness, color).
Information is propagated though the graph over its edges.
Edges encode ‘compatibility’ between nodes. ?
air or water ?
Eric Xing
© Eric CMU,

11. Why Approximate Inference?
Why can’t we just run junction tree on this graph?
If NxN grid, tree width at least N
N can be a huge number(~1000s of pixels)
If N~O(1000), we have a clique with 2100 entries
Eric Xing
© Eric CMU,

12. Solution 1: Belief Propagation on loopy graphsk k
BP Message-update Rules
May not converge or converge to a wrong solution
Mki i k k j k i k k
external evidence
Compatibilities (interactions)
Eric Xing
© Eric CMU,

13. Recall BP on trees k k i k k k j i k k BP Message-update Rules
BP on trees always converges to exact marginals
Mki i k k j k i k k
external evidence
Compatibilities (interactions)
Eric Xing
© Eric CMU,

14. Solution 2: The naive mean field approximation
Approximate p(X) by fully factorized q(X)=Piqi(Xi)
For Boltzmann distribution p(X)=exp{åi < j qijXiXj+qioXi}/Z :
mean field equation: Xi
Áxjñqj resembles a “message” sent from node j to i
{áxjñqj : j Î Ni} forms the “mean field” applied to Xi from its neighborhood Xi
To draw an analogy between our algorithm and the naïve mean field method, let’s briefly review the classical mean field approximation to the Boltzmann distribution, which depends on pairwise interactions between random variables. In a Gibbs sampling algorithm that asymptotically converges to the true distribution, we iteratively sample each variable using a predictive distribution that conditioned on the previously sampled values of the neighbored variables.
The naïve mean field method approximate the true distribution with a fully factored distribution, which is a product of singleton marginals on every node. Simple algebra shows that each singleton marginals can be expressed as a conditional distribution of the relevant node given the expectation of all its neighbors, and this distribution reuses the set of coupling weights of the original model. This is very similar to the Gibbs sampler scheme expect that the sampled values are now replaced by expectations.
We can view the expectation of a neighboring node as a message, and the messages from all neighbors as the mean fields.
Eric Xing
© Eric CMU,

15. Gibbs predictive distribution:
Recall Gibbs sampling
Approximate p(X) by fully factorized q(X)=Piqi(Xi)
For Boltzmann distribution p(X)=exp{åi < j qijXiXj+qioXi}/Z :
Gibbs predictive distribution: Xi Xi
To draw an analogy between our algorithm and the naïve mean field method, let’s briefly review the classical mean field approximation to the Boltzmann distribution, which depends on pairwise interactions between random variables. In a Gibbs sampling algorithm that asymptotically converges to the true distribution, we iteratively sample each variable using a predictive distribution that conditioned on the previously sampled values of the neighbored variables.
The naïve mean field method approximate the true distribution with a fully factored distribution, which is a product of singleton marginals on every node. Simple algebra shows that each singleton marginals can be expressed as a conditional distribution of the relevant node given the expectation of all its neighbors, and this distribution reuses the set of coupling weights of the original model. This is very similar to the Gibbs sampler scheme expect that the sampled values are now replaced by expectations.
We can view the expectation of a neighboring node as a message, and the messages from all neighbors as the mean fields.
Eric Xing
© Eric CMU,

16. Summary So Far
Exact inference methods are limited to tree-structured graphs
Junction Tree methods is exponentially expensive to the tree-width
Message Passing methods can be applied for loopy graphs, but lack of analysis!
Mean-field is convergent, but can have local optimal
Where do these two algorithm come from? Do they make sense?
Eric Xing
© Eric CMU,

17. Next Step … Develop a general theory of variational inference
Introduce some approximate inference methods
Provide deep understandings to some popular methods
Eric Xing
© Eric CMU,

18. Exponential Family GMs
Canonical Parameterization
Effective canonical parameters
Regular family:
Minimal representation:
if there does not exist a nonzero vector such that is a constant
Canonical Parameters
Sufficient Statistics
Log-normalization Function
Eric Xing
© Eric CMU,

19. Examples Ising Model (binary r.v.: {-1, +1}) Gaussian MRF Eric Xing
© Eric CMU,

20. Mean Parameterization
The mean parameter associated with a sufficient statistic
is defined as
Realizable mean parameter set
A convex subset of
Convex hull for discrete case
Convex polytope when is finite
Eric Xing
© Eric CMU,

21. Convex Polytope Convex hull representation
Half-plane based representation
Minkowski-Weyl Theorem:
any polytope can be characterized by a finite collection of linear inequality constraints
Eric Xing
© Eric CMU,

22. Example Two-node Ising Model Convex hull representation
Half-plane representation
Probability Theory:
Eric Xing
© Eric CMU,

23. Marginal Polytope Canonical Parameterization Mean parameterization
Marginal distributions over nodes and edges
Marginal Polytope
Eric Xing
© Eric CMU,

24. Conjugate Duality Duality between MLE and Max-Ent:
For all , a unique canonical parameter satisfying
The log-partition function has the variational form
For all , the supremum in (*) is attained uniquely at specified by the moment-matching conditions
Bijection for minimal exponential family
Eric Xing
© Eric CMU,

25. Roles of Mean Parameters
Forward Mapping:
From to the mean parameters
A fundamental class of inference problems in exponential family models
Backward Mapping:
Parameter estimation to learn the unknown
Eric Xing
© Eric CMU,

26. Example Bernoulli If Reverse mapping: Unique!
No gradient stationary point in the Opt. problem (**)
Unique!
Eric Xing
© Eric CMU,

27. Variational Inference In General
An umbrella term that refers to various mathematical tools for optimization-based formulations of problems, as well as associated techniques for their solution
General idea:
Express a quantity of interest as the solution of an optimization problem
The optimization problem can be relaxed in various ways
Approximate the functions to be optimized
Approximate the set over which the optimization takes place
Goes in parallel with MCMC
Eric Xing
© Eric CMU,

28. A Tree-Based Outer-Bound to a
Local Consistent (Pseudo-) Marginal Polytope
normalization
marginalization
Relation to
holds for any graph
holds for tree-structured graphs
Both convex hull and intersection of half-spaces representation are hard to manipulate in general
Eric Xing
© Eric CMU, 28

29. A Example A three node graph (binary r.v.) For any , we have
For , we have
an exercise? 1 3 2
Eric Xing
© Eric CMU,

30. Bethe Entropy Approximation
Approximate the negative entropy , which doesn’t has a closed-form in general graph.
Entropy on tree (Marginals)
recall:
entropy
Bethe entropy approximation (Pseudo-marginals)
Eric Xing
© Eric CMU,

31. Bethe Variational Problem (BVP)
We already have:
a convex (polyhedral) outer bound
the Bethe approximate entropy
Combining the two ingredients, we have
a simple structured problem (differentiable & constraint set is a simple polytope)
Max-product is the solver!
Eric Xing
© Eric CMU,
Nobel Prize in Physics (1967)

32. Connection to Sum-Product Alg.
Lagrangian method for BVP:
Sum-product and Bethe Variational (Yedidia et al., 2002)
For any graph G, any fixed point of the sum-product updates specifies a pair of such that
For a tree-structured MRF, the solution is unique, where correspond to the exact singleton and pairwise marginal distributions of the MRF, and the optimal value of BVP is equal to
Eric Xing
© Eric CMU,

33. Proof
Eric Xing
© Eric CMU,

34. Discussions
The connection provides a principled basis for applying the sum-product algorithm for loopy graphs
However,
this connection provides no guarantees on the convergence of the sum-product alg. on loopy graphs
the Bethe variational problem is usually non-convex. Therefore, there are no guarantees on the global optimum
Generally, there are no guarantees that is a lower bound of
However, however
the connection and understanding suggest a number of avenues for improving upon the ordinary sum-product alg., via progressively better approximations to the entropy function and outer bounds on the marginal polytope!
Eric Xing
© Eric CMU,

35. Inexactness of Bethe and Sum-Product
From Bethe entropy approximation
Example
From pseudo-marginal outer bound
strict inclusion 3 2 1 4
Eric Xing
© Eric CMU,

36. Summary of LBP
Variational methods in general turn inference into an optimization problem
However, both the objective function and constraint set are hard to deal with
Bethe variational approximation is a tree-based approximation to both objective function and marginal polytope
Belief propagation is a Lagrangian-based solver for BVP
Generalized BP extends BP to solve the generalized hyper-tree based variational approximation problem
Eric Xing
© Eric CMU,

37. Tractable Subgraph
Given a GM with a graph G, a subgraph F is tractable if
We can perform exact inference on it
Example:
Eric Xing
© Eric CMU,

38. Mean Parameterization
For an exponential family GM defined with graph G and sufficient statistics , the realizable mean parameter set
For a given tractable subgraph F, a subset of mean parameters is of interest
Inner Approximation
Eric Xing
© Eric CMU,

39. Optimizing a Lower Bound
Any mean parameter yields a lower bound on the log-partition function
Moreover, equality holds iff and are dually coupled, i.e.,
Proof Idea: (Jensen’s Inequality)
Optimizing the lower bound gives
This is an inference!
Eric Xing
© Eric CMU,

40. Mean Field Methods In General
However, the lower bound can’t explicitly evaluated in general
Because the dual function typically lacks an explicit form
Mean Field Methods
Approximate the lower bound
Approximate the realizable mean parameter set
The MF optimization problem
Still a lower bound?
Eric Xing
© Eric CMU,

41. KL-divergence Kullback-Leibler Divergence
For two exponential family distributions with the same STs:
Primal Form
Mixed Form
Dual Form
Eric Xing
© Eric CMU,

42. Mean Field and KL-divergence
Optimizing a lower bound
Equivalent to minimize a KL-divergence
Therefore, we are doing minimization
Eric Xing
© Eric CMU,

43. Naïve Mean Field Fully factorized variational distribution Eric Xing
© Eric CMU,

44. Naïve Mean Field for Ising Model
Sufficient statistics and Mean Parameters
Naïve Mean Field
Realizable mean parameter subset
Entropy
Optimization Problem
Eric Xing
© Eric CMU,

45. Naïve Mean Field for Ising Model
Optimization Problem
Update Rule
resembles “message” sent from node to
forms the “mean field” applied to from its neighborhood
Eric Xing
© Eric CMU,

46. Non-Convexity of Mean Field
Mean field optimization is always non-convex for any exponential family in which the state space is finite
Finite convex hull
contains all the extreme points
If is a convex set, then
Mean field has been used successfully
Eric Xing
© Eric CMU,

47. Structured Mean Field
Mean field theory is general to any tractable sub-graphs
Naïve mean field is based on the fully unconnected sub-graph
Variants based on structured sub-graphs can be derived
Eric Xing
© Eric CMU, 47

48. Topic models Σ μ Σ* Approximate the Integral g g β z z Φ* β w wμ* Σ* Φ* β
Approximate the Integral g β z w
Approximate the Posterior
μ*,Σ*,φ1:n*
Optimization
Problem
Solve
Eric Xing
© Eric CMU,

49. Variational Inference With no Tears [Ahmed and Xing, 2006, Xing et al 2003]β g z e μ Σ
Fully Factored Distribution
Fixed Point Equations μ Σ g β z e
Laplace approximation
Eric Xing
© Eric CMU,

50. Summary of GMF
Message-passing algorithms (e.g., belief propagation, mean field) are solving approximate versions of exact variational principle in exponential families
There are two distinct components to approximations:
Can use either inner or outer bounds to
Various approximation to the entropy function
BP: polyhedral outer bound and non-convex Bethe approximation
MF: non-convex inner bound and exact form of entropy
Kikuchi: tighter polyhedral outer bound and better entropy approximation
Eric Xing
© Eric CMU,
© Eric CMU,