1. Probabilistic Models with Latent Variables
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

2. Density Estimation Problem
Learning from unlabeled data 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑁
Unsupervised learning, density estimation
Empirical distribution typically has multiple modes
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

3. Density Estimation Problem
From
From
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

4. Density Estimation Problem
Conv. composition of unimodal pdf’s: multimodal pdf
𝑓 𝑥 = 𝑘 𝑤 𝑘 𝑓 𝑘 (𝑥) where 𝑘 𝑤 𝑘 =1
Physical interpretation
Sub populations
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

5. Latent Variables Introduce new variable 𝑍 𝑖 for each 𝑋 𝑖
Latent / hidden: not observed in the data
Probabilistic interpretation
Mixing weights: 𝑤 𝑘 ≡𝑝( 𝑧 𝑖 =𝑘)
Mixture densities: 𝑓 𝑘 (𝑥)≡𝑝(𝑥| 𝑧 𝑖 =𝑘)
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

6. Generative Mixture Model
For 𝑖=1,…,𝑁
𝑍 𝑖 ~𝑖𝑖𝑑 𝑀𝑢𝑙𝑡
𝑋 𝑖 ~𝑖𝑖𝑑 𝑝(𝑥| 𝑧 𝑖 )
𝑃( 𝑥 𝑖 , 𝑧 𝑖 )=𝑝 𝑧 𝑖 𝑝( 𝑥 𝑖 | 𝑧 𝑖 )
𝑃 𝑥 𝑖 = 𝑘 𝑝( 𝑥 𝑖 , 𝑧 𝑖 ) recovers mixture distribution
𝑍 𝑖
𝑋 𝑖 𝑁
Plate Notation
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

7. Tasks in a Mixture Model
Inference
𝑃 𝑧 𝑥 = 𝑖 𝑃( 𝑧 𝑖 | 𝑥 𝑖 )
Parameter Estimation
Find parameters that e.g. maximize likelihood
Does not decouple according to classes
Non convex, many local minima
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

8. Example: Gaussian Mixture Model
For 𝑖=1,…,𝑁
𝑍 𝑖 ~𝑖𝑖𝑑 𝑀𝑢𝑙𝑡(𝜋)
𝑋 𝑖 | 𝑍 𝑖 =𝑘~𝑖𝑖𝑑 𝑁(𝑥| 𝜇 𝑘 ,Σ)
Inference
𝑃( 𝑧 𝑖 =𝑘| 𝑥 𝑖 ;𝜇,Σ)
Soft-max function
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

9. Example: Gaussian Mixture Model
Loglikelihood
Which training instance comes from which component?
𝑙 𝜃 = 𝑖 log 𝑝 𝑥 𝑖 = 𝑖 log 𝑘 𝑝 𝑧 𝑖 =𝑘 𝑝( 𝑥 𝑖 | 𝑧 𝑖 =𝑘)
No closed form solution for maximizing 𝑙 𝜃
Possibility 1: Gradient descent etc
Possibility 2: Expectation Maximization
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

10. Expectation Maximization Algorithm
Observation: Know values of 𝑍 𝑖 ⇒ easy to maximize
Key idea: iterative updates
Given parameter estimates, “infer” all 𝑍 𝑖 variables
Given inferred 𝑍 𝑖 variables, maximize wrt parameters
Questions
Does this converge?
What does this maximize?
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

11. Expectation Maximization Algorithm
Complete loglikelihood
𝑙 𝑐 𝜃 = 𝑖 log 𝑝 𝑥 𝑖 , 𝑧 𝑖 = 𝑖 log 𝑝 𝑧 𝑖 𝑝( 𝑥 𝑖 | 𝑧 𝑖 )
Problem: 𝑧 𝑖 not known
Possible solution: Replace w/ conditional expectation
Expected complete loglikelihood
𝑄 𝜃, 𝜃 𝑜𝑙𝑑 =𝐸[ 𝑖 log 𝑝 𝑥 𝑖 , 𝑧 𝑖 ]
Wrt 𝑝(𝑧|𝑥, 𝜃 𝑜𝑙𝑑 ) where 𝜃 𝑜𝑙𝑑 are the current parameters
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

12. Expectation Maximization Algorithm
𝑄 𝜃, 𝜃 𝑜𝑙𝑑 =𝐸[ 𝑖 log 𝑝 𝑥 𝑖 , 𝑧 𝑖 ]
= 𝑖 𝑘 𝐸 𝐼 𝑧 𝑖 =𝑘 log [ 𝜋 𝑘 𝑝( 𝑥 𝑖 | 𝜃 𝑘 )]
= 𝑖 𝑘 𝑝( 𝑧 𝑖 =𝑘| 𝑥 𝑖 , 𝜃 𝑜𝑙𝑑 ) log [ 𝜋 𝑘 𝑝( 𝑥 𝑖 | 𝜃 𝑘 )]
= 𝑖 𝑘 𝛾 𝑖𝑘 log 𝜋 𝑘 + 𝑖 𝑘 𝛾 𝑖𝑘 log 𝑝( 𝑥 𝑖 | 𝜃 𝑘 )
Where 𝛾 𝑖𝑘 =𝑝( 𝑧 𝑖 =𝑘| 𝑥 𝑖 , 𝜃 𝑜𝑙𝑑 )
Compare with likelihood for generative classifier
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

13. Expectation Maximization Algorithm
Expectation Step
Update 𝛾 𝑖𝑘 based on current parameters
𝛾 𝑖𝑘 = 𝜋 𝑘 𝑝 𝑥 𝑖 𝜃 𝑜𝑙𝑑, 𝑘 𝑘 𝜋 𝑘 𝑝 𝑥 𝑖 𝜃 𝑜𝑙𝑑, 𝑘
Maximization Step
Maximize 𝑄 𝜃, 𝜃 𝑜𝑙𝑑 wrt parameters
Overall algorithm
Initialize all latent variables
Iterate until convergence
M Step
E Step
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

14. Example: EM for GMM E Step remains the step for all mixture models
M Step
𝜋 𝑘 = 𝑖 𝛾 𝑖𝑘 𝑁 = 𝛾 𝑘 𝑁
𝜇 𝑘 = 𝑖 𝛾 𝑖𝑘 𝑥 𝑖 𝛾 𝑘
Σ=?
Compare with generative classifier
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

15. Analysis of EM Algorithm
Expected complete LL is a lower bound on LL
EM iteratively maximizes this lower bound
Converges to a local maximum of the loglikelihood
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

16. Bayesian / MAP Estimation
EM overfits
Possible to perform MAP instead of MLE in M-step
EM is partially Bayesian
Posterior distribution over latent variables
Point estimate over parameters
Fully Bayesian approach is called Variational Bayes
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

17. (Lloyd’s) K Means Algorithm
Hard EM for Gaussian Mixture Model
Point estimate of parameters (as usual)
Point estimate of latent variables
Spherical Gaussian mixture components
𝑧 𝑖 ∗ = arg max k 𝑝( 𝑧 𝑖 =𝑘| 𝑥 𝑖 ,𝜃) = arg min 𝑘 𝑥 𝑖 − 𝜇 𝑘
Where 𝜇 𝑘 = 𝑖: 𝑧 𝑖 =𝑘 𝑥 𝑖 𝑁
Most popular “hard” clustering algorithm
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

18. K Means Problem
Given { 𝑥 𝑖 }, find k “means” ( 𝜇 1 ∗ ,…, 𝜇 𝑘 ∗ ) and data assignments ( 𝑧 1 ∗ ,…, 𝑧 𝑁 ∗ ) such that
𝜇 ∗ , 𝑧 ∗ = arg min 𝜇,𝑧 𝑖 𝑥 𝑖 −𝜇 𝑧 𝑖
Note: 𝑧 𝑖 is k-dimensional binary vector
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

19. Model selection: Choosing K for GMM
Cross validation
Plot likelihood on training set and validation set for increasing values of k
Likelihood on training set keeps improving
Likelihood on validation set drops after “optimal” k
Does not work for k-means! Why?
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

20. Principal Component Analysis: Motivation
Dimensionality reduction
Reduces #parameters to estimate
Data often resides in much lower dimension, e.g., on a line in a 3D space
Provides “understanding”
Mixture models very restricted
Latent variables restricted to small discrete set
Can we “relax” the latent variable?
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

21. Classical PCA: Motivation
Revisit K-means
min 𝑊,𝑍 𝐽 𝑊,𝑍 = 𝑋−𝑊 𝑍 𝑇 2 𝐹
W: matrix containing means
Z: matrix containing cluster membership vectors
How can we relax Z and W?
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

22. Classical PCA: Problem
min 𝑊,𝑍 𝐽 𝑊,𝑍 = |𝑋−𝑊 𝑍 𝑇 | 2 𝐹
X : 𝐷×𝑁
Arbitrary Z of size 𝑁×𝐿,
Orthonormal W of size 𝐷×𝐿
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

23. Classical PCA: Optimal Solution
Empirical covariance matrix Σ = 1 𝑁 𝑖 𝑥 𝑖 𝑥 𝑖 𝑇
Scaled and centered data
𝑊 = 𝑉 𝐿 where 𝑉 𝐿 contains L Eigen vectors for the L largest Eigen values of Σ
𝑧 𝑖 = 𝑊 𝑇 𝑥 𝑖
Alternative solution via Singular Value Decomposition (SVD)
W contains the “principal components” that capture the largest variance in the data
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

24. 𝑃( 𝑥 𝑖 | 𝑧 𝑖 ) = 𝑁( 𝑥 𝑖 |𝑊 𝑧 𝑖 +𝜇, Ψ)
Probabilistic PCA
Generative model
𝑃( 𝑧 𝑖 ) = 𝑁( 𝑧 𝑖 | 𝜇 0 , Σ 0 )
𝑃( 𝑥 𝑖 | 𝑧 𝑖 ) = 𝑁( 𝑥 𝑖 |𝑊 𝑧 𝑖 +𝜇, Ψ)
Ψ forced to be diagonal
Latent linear models
Factor Analysis
Special Case: PCA with Ψ = 𝜎 2 𝐼
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

25. Visualization of Generative Process
From Bishop, PRML
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

26. Relationship with Gaussian Density
𝐶𝑜𝑣[𝑥] = 𝑊 𝑊 𝑇 +Ψ
Why does Ψ need to be restricted?
Intermediate low rank parameterization of Gaussian covariance matrix between full rank and diagonal
Compare #parameters
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

27. EM for PCA: Rod and Springs
From Bishop, PRML
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

28. Advantages of EM Simpler than gradient methods w/ constraints
Handles missing data
Easy path for handling more complex models
Not always the fastest method
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya

29. Summary of Latent Variable Models
Learning from unlabeled data
Latent variables
Discrete: Clustering / Mixture models ; GMM
Continuous: Dimensionality reduction ; PCA
Summary / “Understanding” of data
Expectation Maximization Algorithm
Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya