1. Collapsed Variational Dirichlet Process Mixture Models
Kenichi Kurihara, Max Welling and Yee W. Teh
Published on IJCAI 07
Discussion led by
Qi An

2. Outline Introduction Four approximations to DP
Variational Bayesian Inference
Optimal cluster label reordering
Experimental results
Discussion

3. Introduction
DP is suitable for many density estimation and data clustering applications.
Gibbs sampling solution is not efficient enough to scale up to the large scale problems.
Truncated stick-breaking approximation is formulated in the space of explicit, non-exchangeable cluster labels.

4. Introduction This paper
propose an improved VB algorithm based on integrating out mixture weights
compare the stick-breaking representation against the finite symmetric Dirichlet approximation
maintain optimal ordering of cluster labels in the stick-breaking VB algorithm

5. Approximations to DP Truncated stick-breaking representation
The joint distribution can be expressed as:

6. Approximations to DP Finite symmetric Dirichlet approximation
The joint distribution can be expressed as:
The essential difference from TSB representation is that the cluster labels remain interchangeable under this formulation.

7. Dirichlet process is most naturally defined on a partition space while both TSB and FSD are defined over the cluster label space. Moreover, TSB and FSD also live in different spaces

8. Marginalization
In variational Bayesian approximation, we assume a factorized form for the posterior distribution.
However it is not a good assumption since changes in π will have a considerable impact on z.
If we can integrate out π , the joint distribution is given by
For the TSB representation:
For the FSD representation: α

9. VB inference
We can then apply the VB inference on the four approximations
The lower bound is given by
The approximated posterior distribution for TSB and FSD are
Depending on marginalization or not, v and π may be integrated out.

10. Gaussian approximation
For collapsed approximations, the computation for q(zij) seems intractable due to the exponentially large space of assignments for all other {zij}.
With central limit theory and Taylor expansion, the expectation over zij will be approximated with those expectations

11. Optimal cluster label reordering
For FSB representation, the prior assumes a certain ordering of the clusters.
The authors claims the optimal relabelling of the clusters is given by ordering the cluster sizes in decreasing order.

12. Experimental results

13. Experimental results

14. Discussion
There is very little difference between variational Bayesian inference in the reordered stick-breaking representation and the finite mixture model with symmetric Dirichlet priors.
Label reordering is important for the stick-breaking representation
Variational approximation are much more efficient computationally than Gibbs sampling, with almost no loss in accuracy