Variational Bayes Model Selection for Mixture Distribution Authors: Adrian Corduneanu & Christopher M. Bishop Presented by Shihao Ji Duke University Machine Learning Group Jan. 20, 2006

Outline Introduction – model selection Automatic Relevance Determination (ARD) Experimental Results Application to HMMs

Introduction Cross validation Bayesian approaches MCMC and Laplace approximation (Traditional) variational method (Type II) variational method

Automatic Relevance Determination (ARD) relevance vector regression Given a dataset , we assume is Gaussian Likelihood: Prior: Posterior: Determination of hyperparameters: Type II ML

Automatic Relevance Determination (ARD) mixture of Gaussian Given an observed dataset , we assume each data point is drawn independently from a mixture of Gaussian density Likelihood: Prior: Posterior: VB Determination of mixing coefficients: Type II ML

Automatic Relevance Determination (ARD) model selection Bayesian method: , Component elimination: if , i.e.,

Experimental Results Bayesian method vs. cross-validation 600 points drawn from a mixture of 5 Gaussians.

Experimental Results Component elimination Initially the model had 15 mixtures, finally was pruned down to 3 mixtures

Experimental Results

Automatic Relevance Determination (ARD) hidden Markov model Given an observed dataset , we assume each data sequence is generated independently from an HMM Likelihood: Prior: Posterior: VB Determination of p and A: Type II ML

Automatic Relevance Determination (ARD) model selection Bayesian method: , State elimination: if , Define -- visiting frequency where

Experimental Results (1)

Experimental Results (2)

Experimental Results (3)

Questions?