A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models Jeff A. Bilmes International Computer Science Institute Berkeley CA, and Computer Science Division Department of Electrical Engineering and Computer Science U.C. Berkeley April 1998 Presenter : Hsu Ting-Wei

Outline Abstract Maximum-likelihood Basic EM Finding Maximum Likelihood Mixture Densities Parameters via EM Learning the parameters of an HMM, EM, and the Baum-Welch algorithm 2019/9/8 NTNU Speech Lab

Abstract Using Expectation-Maximization (EM) algorithm to solve the maximum-likelihood (ML) parameter estimation problem EM parameter estimation procedure for two applications: Finding the parameters of a mixture of Gaussian densities Finding the parameters of a hidden Markov model (HMM) (i.e., the Baum-Welch algorithm) for both discrete and Gaussian mixture observation models Trying to emphasize intuition rather than mathematical rigor 2019/9/8 NTNU Speech Lab

Maximum-likelihood likelihood function density function i.i.d. incomplete data easy to count We can set the derivative of to zero to find the max value. But if the equation can’t be solved by this method ,we should take EM algorithm. max value 2019/9/8 NTNU Speech Lab

Basic EM Two main applications of the EM algorithm: When the data indeed has missing values, due to problems with or limitations of the observation process. The second occurs when optimizing the likelihood function is analytically intractable but when the likelihood function can be simplified by assuming the existence of and values for additional but missing (or hidden) parameters. more common in the computational pattern recognition community 2019/9/8 NTNU Speech Lab

Basic EM (cont.) = complete-data likelihood function joint density function = complete data set observed values missing, unknow values, constant random variable incomplete-data likelihood function : 2019/9/8 NTNU Speech Lab

Basic EM (cont.) The EM algorithm first finds the expected value of the complete-data log-likelihood function E-step : evaluation of expectation new parameter current parameter constant random variable and governed by the distribution Recall: space of values y 2019/9/8 NTNU Speech Lab

Basic EM (cont.) M-Step : maximize the expectation These two step are repeated as necessary. Each iteration is guaranteed to increase the log likelihood and the algorithm is guaranteed to converge to a local maximum of the likelihood function. 2019/9/8 NTNU Speech Lab

目標函數最大化 目標函數 目前模型參數 2019/9/8 NTNU Speech Lab

找輔助函數的一般式 目標函數 輔助函數 2019/9/8 NTNU Speech Lab

在 找”最佳的”輔助函數 目標函數 “最佳的”輔助函數 2019/9/8 NTNU Speech Lab

對輔助函數求全域最大值 目標函數 2019/9/8 NTNU Speech Lab

對輔助函數求全域最大值 (Cont.) 目標函數 2019/9/8 NTNU Speech Lab

重覆剛才的步驟 目標函數 2019/9/8 NTNU Speech Lab

在 找”最佳的”輔助函數 目標函數 “最佳的”輔助函數 2019/9/8 NTNU Speech Lab

對輔助函數求全域最大值 目標函數 2019/9/8 NTNU Speech Lab

Finding Maximum Likelihood Mixture Densities Parameters via EM probabilistic model: density function M component densities mixed together with M mixing coefficients . incomplete-data log-likelihood expression : difficult to optimize because it contains the log of the sum. 2019/9/8 NTNU Speech Lab

Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.) If we know the values of y , the likelihood becomes: If we do not know the values of y ,and y is a random vector : 2019/9/8 NTNU Speech Lab

Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.) E-step : =1 2019/9/8 NTNU Speech Lab

Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.) E-step : M-step: add Lagrange multiplier and 2019/9/8 NTNU Speech Lab

Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.) E-step : M-step: Recall: (*) derivate (*) 將 代回(*)，整理如下… 2019/9/8 NTNU Speech Lab

Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.) E-step : M-step: (*) derivate (*) 2019/9/8 NTNU Speech Lab

Learning the parameters of an HMM, EM, and the Baum-Welch algorithm A Hidden Markov Model is a probabilistic model of the joint probability of a collection of random variables The model is and Two assumption First-order assumption Output independent assumption Three basic problems continuous or discrete observations “hidden” and discrete 2019/9/8 NTNU Speech Lab

Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.) Estimation formula using the Q function. incomplete-data likelihood function E-step: complete-data likelihood function discrete We know 2019/9/8 NTNU Speech Lab

Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.) M-step in discrete: add Lagrange multiplier and 2019/9/8 NTNU Speech Lab

Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.) M-step in discrete: add Lagrange multiplier and 2019/9/8 NTNU Speech Lab

Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.) M-step in discrete: add Lagrange multiplier and 2019/9/8 NTNU Speech Lab

Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.) M-step in continuous: the mixture component for each state at each time 2019/9/8 NTNU Speech Lab