Presentation is loading. Please wait.

Presentation is loading. Please wait.

A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models Jeff A. Bilmes International.

Similar presentations


Presentation on theme: "A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models Jeff A. Bilmes International."— Presentation transcript:

1 A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models Jeff A. Bilmes International Computer Science Institute Berkeley CA, and Computer Science Division Department of Electrical Engineering and Computer Science U.C. Berkeley April 1998 Hsu Ting-Wei ‘s slide Presented by Patty Liu

2 Outline Abstract Maximum-likelihood Basic EM
Finding Maximum Likelihood Mixture Densities Parameters via EM Learning the parameters of an HMM, EM, and the Baum-Welch algorithm 2019/7/24 NTNU Speech Lab

3 Abstract Using Expectation-Maximization (EM) algorithm to solve the maximum-likelihood (ML) parameter estimation problem EM parameter estimation procedure for two applications: Finding the parameters of a mixture of Gaussian densities Finding the parameters of a hidden Markov model (HMM) (i.e., the Baum-Welch algorithm) for both discrete and Gaussian mixture observation models Trying to emphasize intuition rather than mathematical rigor 2019/7/24 NTNU Speech Lab

4 Maximum-likelihood likelihood function density function i.i.d.
incomplete data easy to count We can set the derivative of to zero to find the max value. But if the equation can’t be solved by this method ,we should take EM algorithm. max value 2019/7/24 NTNU Speech Lab

5 Basic EM Two main applications of the EM algorithm:
When the data indeed has missing values, due to problems with or limitations of the observation process. The second occurs when optimizing the likelihood function is analytically intractable but when the likelihood function can be simplified by assuming the existence of and values for additional but missing (or hidden) parameters. more common in the computational pattern recognition community 2019/7/24 NTNU Speech Lab

6 Basic EM (cont.) = complete-data likelihood function
joint density function = complete data set observed values missing, unknow values, constant random variable incomplete-data likelihood function : 2019/7/24 NTNU Speech Lab

7 Basic EM (cont.) The EM algorithm first finds the expected value of the complete-data log-likelihood function E-step : evaluation of expectation new parameter current parameter constant random variable and governed by the distribution Recall: space of values y 2019/7/24 NTNU Speech Lab

8 Basic EM (cont.) M-Step : maximize the expectation
These two step are repeated as necessary. Each iteration is guaranteed to increase the log likelihood and the algorithm is guaranteed to converge to a local maximum of the likelihood function. 2019/7/24 NTNU Speech Lab

9 目標函數最大化 目標函數 目前模型參數 2019/7/24 NTNU Speech Lab

10 找輔助函數的一般式 目標函數 輔助函數 2019/7/24 NTNU Speech Lab

11 在 找”最佳的”輔助函數 目標函數 “最佳的”輔助函數 2019/7/24 NTNU Speech Lab

12 對輔助函數求全域最大值 目標函數 2019/7/24 NTNU Speech Lab

13 對輔助函數求全域最大值 (Cont.) 目標函數 2019/7/24 NTNU Speech Lab

14 重覆剛才的步驟 目標函數 2019/7/24 NTNU Speech Lab

15 在 找”最佳的”輔助函數 目標函數 “最佳的”輔助函數 2019/7/24 NTNU Speech Lab

16 對輔助函數求全域最大值 目標函數 2019/7/24 NTNU Speech Lab

17 Finding Maximum Likelihood Mixture Densities Parameters via EM
probabilistic model: density function M component densities mixed together with M mixing coefficients . incomplete-data log-likelihood expression : difficult to optimize because it contains the log of the sum. 2019/7/24 NTNU Speech Lab

18 Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.)
If we know the values of y , the likelihood becomes: If we do not know the values of y ,and y is a random vector : 2019/7/24 NTNU Speech Lab

19 Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.)
E-step : =1 2019/7/24 NTNU Speech Lab

20 Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.)
E-step : M-step: add Lagrange multiplier and 2019/7/24 NTNU Speech Lab

21 Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.)
E-step : M-step: Recall: (*) derivate (*) 將 代回(*),整理如下… 2019/7/24 NTNU Speech Lab

22 Finding Maximum Likelihood Mixture Densities Parameters via EM (cont.)
E-step : M-step: (*) derivate (*) 2019/7/24 NTNU Speech Lab

23 Learning the parameters of an HMM, EM, and the Baum-Welch algorithm
A Hidden Markov Model is a probabilistic model of the joint probability of a collection of random variables The model is and Two assumption First-order assumption Output independent assumption Three basic problems continuous or discrete observations “hidden” and discrete 2019/7/24 NTNU Speech Lab

24 Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.)
Estimation formula using the Q function. incomplete-data likelihood function E-step: complete-data likelihood function discrete We know 2019/7/24 NTNU Speech Lab

25 Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.)
M-step in discrete: add Lagrange multiplier and 2019/7/24 NTNU Speech Lab

26 Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.)
M-step in discrete: add Lagrange multiplier and 2019/7/24 NTNU Speech Lab

27 Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.)
M-step in discrete: add Lagrange multiplier and 2019/7/24 NTNU Speech Lab

28 Learning the parameters of an HMM, EM, and the Baum-Welch algorithm (cont.)
M-step in continuous: the mixture component for each state at each time 2019/7/24 NTNU Speech Lab


Download ppt "A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models Jeff A. Bilmes International."

Similar presentations


Ads by Google