Generalization Performance of Exchange Monte Carlo Method for Normal Mixture Models Kenji Nagata, Sumio Watanabe Tokyo Institute of Technology

Contents  Background Normal Mixture Models Bayesian Learning MCMC method  Proposed method Exchange Monte Carlo method Application to Bayesian Learning  Experiment and Discussion  Conclusion

Background: Normal Mixture Models Normal mixture model is widely used in pattern recognition, data clustering and many other applications. :number of component :dimension of data parameter: A normal mixture model is a learning machine which estimates a target probability density by sum of normal distributions.

Posterior distribution: Marginal likelihood: Predictive distribution: Empirical Kullback information: Background: Bayesian Learning Because of difficulty of analytical calculation, the Markov Chain Monte Carlo (MCMC) method is widely used.

The algorithm to generate the sample sequence which converges to the target distribution. Background: MCMC method set with probability, set with probability. set as the next position. Huge computational cost!!

Purpose  We propose that the exchange MC method is appropriate for Bayesian learning in hierarchical learning machines.  We clarify its effectiveness by experimental result.

Contents  Background Normal Mixture Models Bayesian Learning MCMC method  Proposed method Exchange Monte Carlo method Application to Bayesian Learning  Experiment and Discussion  Conclusion

Exchange Monte Carlo method [Hukushima,96] We consider to obtain the sample sequence from the following simultaneous distribution. ＜ Algorithm ＞ 1.Each sequence is obtained from each target distribution by using the Metropolis algorithm independently and simultaneously for a few iterations. 2.Exchange of two positions, and, is tried and accepted with the following probability, The following two steps are performed alternately:

Exchange Monte Carlo method [Hukushima,96] ＜ Exchange Monte Carlo method ＞

Application to Bayesian learning (prior) (posterior) :standard normal distribution

Contents  Background Normal Mixture Models Bayesian Learning MCMC method  Proposed method Exchange Monte Carlo method Application to Bayesian Learning  Experiment and Discussion  Conclusion

Experimental Settings dimension of data: 2 components number of training data: 5 components : uniform distribution [0:1] : standard normal distribution

Experimental Settings 1.Exchange Monte Carlo method (EMC) Monte Carlo (MC) Step Sample for expectation

Experimental Settings 2. Conventional Metropolis algorithm (CM) MC Step Sample for expectation

Experimental Settings 3. Parallel Metropolis algorithm (PM) MC Step Sample for expectation

（ ） Experimental Settings Initial value: random sampling from the prior distribution For calculating the expectation, we use the last 50% of the sample sequence. （ otherwise ）

Experimental result (histogram) Marginal distribution of parameter 1.EMC 2.CM3.PM MC step:3200 The algorithm CM cannot approximate the Bayesian posterior distribution. True marginal distribution has two peaks around 0 and 0.5.

Experimental result (generalization error) Convergence of the generalization error 1.EMC 2.CM3.PM test data: EMC provides smaller generalization error than CM. MC step: from 100 to 3200 CM EMC

Contents  Background Singular Learning Machine Bayesian Learning MCMC method  Proposed method Exchange Monte Carlo method Application to Bayesian Learning  Experiment and Discussion  Conclusion

Conclusion  We proposed that the exchange MC method is appropriate for the Bayesian learning in hierarchical learning machines.  We clarified its effectiveness by the simulation of Bayesian learning in normal mixture models.  By experimental results, The exchange MC method approximates the Bayesian posterior distribution more accurately than the Metropolis algorithm. The exchange MC method provides better generalization performance than the Metropolis algorithm.