1. Expectation Maximization for GMM Comp344 Tutorial Kai Zhang

2. GMM Model the data distribution by a combination of Gaussian functions Given a set of sample points, how to estimate the parameters of the GMM?

3. EM Basic Idea Given data X, and initial parameter Θ t Assume a hidden variable Y 1. Study how Y is distributed based on current knowledge (X and Θ t ), i.e., p(Y|X, Θ t ) Compute the expectation of the joint data likelihood under this distribution (called Q function) 2. Maximize this expectation w.r.t. the to-be-determined parameter Θ t+1 Iterate step 1 and 2 until convergence

4. EM with GMM In the context of GMM X: data points Y: which Gaussian creates which data points Θ :parameters of the mixture model Constraint: P k ’s must sum up to 1, so that p(x) is a pdf

5. How to write the Q function under GMM setting Likelihood of a data set is the multiplication of all the sample likelihood, so

6. The Q function specific for GMM is Plug in the definition of p(x| Θ k ), compute derivative w.r.t. the parameters, we obtain the iteration procedures E step M step

7. Posteriors Intuitive meaning of The posterior probability that x i is created by the kth Gaussian component (soft membership) The meaning of Note that it is the summation of all having the same k So it means the strength of the kth Gaussian component

8. Comments GMM can be deemed as performing a density estimation, in the form of a combination of a number of Gaussian functions clustering, where clusters correspond to the Gaussian component, and cluster assignment can be achieved through the bayes rule GMM produces exactly what are needed in the Bayes decision rule: prior probability and class conditional probability So GMM+Bayes rule can compute posterior probability, hence solving clustering problem

9. Illustration Class/points Conditional Prob X1(i=1)X 2 (i=2)………… Class1,k=1 (P 1 ) P 11 =P(x 1 |k=1) P 21 =P(x 2 |k=1) …… Each row sum up to 1 (a Gaussian curve) Class2,k=2 (P 2 ) P 12 = p(x 1 |k=2) P 22 = P(x 2 |k=2) Condition: P 1 + P 2 =1 Each column can be used to compute the posterior probability

10. Illustration Conditional probability x1x2x3x4x5 c1P 1|1 =0.35P 2|1 =0.35P 3|1 =0.1P 41 =0.1P 51 =0.1 c2P 1|2 =0.05P 2|2 =0.05P 3|2 =0.3P 4|2 =0.3P 5|2 =0.3 Posterior probability x1x2x3x4x5 c114/17 2/11 c23/17 3/119/11 class Prior probability c1P 1 =2/5 c2P 2 =3/5 class (updated) Prior Probability c1 (28/17+6/11 )/5 c2 (6/17+21/11 )/5 (Updated) Conditional Probability Estimate the mean and covariance c1 X1(14/17),X2(14/17),X3(2/11),X4(2/11),X5(2/11) c2 X1(4/17),X2(4/17),X3(9/11),X4(9/11),X5(9/11)

11. Initialization Perform an initial clustering and divide the data into m clusters (e.g., simply cut one dimension into m segments) For the kth cluster Its mean is the kth Gaussian component mean (μ k ) Its covariance is the kth Gaussian component covariance (Σ k ) The portion of samples is the Prior for the kth Gaussian component (p k)

12. EM iterations

13. Applications, image segmentation