1. Variational Bayes Model Selection for Mixture Distribution
Authors: Adrian Corduneanu & Christopher M. Bishop
Presented by Shihao Ji
Duke University Machine Learning Group
Jan. 20, 2006

2. Outline Introduction – model selection
Automatic Relevance Determination (ARD)
Experimental Results
Application to HMMs

3. Introduction Cross validation Bayesian approaches
MCMC and Laplace approximation
(Traditional) variational method
(Type II) variational method

4. Automatic Relevance Determination (ARD)
relevance vector regression
Given a dataset , we assume is Gaussian
Likelihood:
Prior:
Posterior:
Determination of hyperparameters:
Type II ML

5. Automatic Relevance Determination (ARD)
mixture of Gaussian
Given an observed dataset , we assume each data point is drawn
independently from a mixture of Gaussian density
Likelihood:
Prior:
Posterior: VB
Determination of mixing coefficients:
Type II ML

6. Automatic Relevance Determination (ARD)
model selection
Bayesian method: ,
Component elimination: if ,
i.e.,

7. Experimental Results Bayesian method vs. cross-validation
600 points drawn from a mixture of 5 Gaussians.

8. Experimental Results Component elimination
Initially the model had 15 mixtures, finally was pruned down to 3 mixtures

9. Experimental Results

10. Automatic Relevance Determination (ARD)
hidden Markov model
Given an observed dataset , we assume each data sequence is
generated independently from an HMM
Likelihood:
Prior:
Posterior: VB
Determination of p and A:
Type II ML

11. Automatic Relevance Determination (ARD)
model selection
Bayesian method: ,
State elimination: if ,
Define -- visiting frequency
where

12. Experimental Results (1)

14. Experimental Results (2)

16. Experimental Results (3)

18. Questions?