1. Lecture 11: Mixture of Gaussians
CS480/680: Intro to ML
Lecture 11: Mixture of Gaussians
11/1/18
Yao-Liang Yu

2. Recap: Gaussian distribution
11/1/18
Yao-Liang Yu

3. Multi-modality
11/1/18
Yao-Liang Yu

4. Mixture models Where did we see a similar idea? # of components
parameters
mixing distr.
k-th component distr.
Where did we see a similar idea?
11/1/18
Yao-Liang Yu

5. Example: Gaussian Mixture Models
11/1/18
Yao-Liang Yu

6. Universality
Theorem. GMM with sufficiently many components can approximate any probability density function on Rd.
How many is many?
Nothing special about Gaussian here, except computationally (later).
11/1/18
Yao-Liang Yu

7. Example: Mixture of Experts
mixing distr.
k-th component distr.
11/1/18
Yao-Liang Yu

8. Inference problem Given iid sample X1, X2, …, Xn from p(x|θ)
latent (unobserved)
Given iid sample X1, X2, …, Xn from p(x|θ)
Need to estimate θ
Maximum likelihood is NP-hard…
11/1/18
Yao-Liang Yu

9. Soft clustering
(Stauffer & Grimson, CVPR’98)
11/1/18
Yao-Liang Yu

10. Bigger issue: identifiability?
Is this factorization even unique?
Yes, for GMMs!
11/1/18
Yao-Liang Yu

11. Variational form of Max Likelihood
neg. entropy
11/1/18
Yao-Liang Yu

12. KL divergence Both p and q are nonnegative and sum to 1
Jensen’s inequality
E(log(X)) <= log(E(X))
Both p and q are nonnegative and sum to 1
Equality holds iff p == q
Measures difference between distributions; asymmetric
11/1/18
Yao-Liang Yu

13. The EM algorithm Fix q, solve θ Fix θ, solve q often closed-form
11/1/18
Yao-Liang Yu

14. EM for GMM: step 1
11/1/18
Yao-Liang Yu

15. Just in case
11/1/18
Yao-Liang Yu

16. EM for GMM: step 2
posterior
prior
likelihood
11/1/18
Yao-Liang Yu

17. Example
11/1/18
Yao-Liang Yu

18. Other uses of EM Simplify computation Missing data
t-distribution as a Gaussian scale-mixture
Missing data
11/1/18
Yao-Liang Yu

19. Questions?
11/1/18
Yao-Liang Yu