1. EM Algorithm
主講人：虞台文

2. Contents Introduction Example  Missing Data
Example  Mixed Attributes
Example  Mixture
Main Body
Mixture Model
EM-Algorithm on GMM

3. EM Algorithm
Introduction

4. Introduction
EM is typically used to compute maximum likelihood estimates given incomplete samples.
The EM algorithm estimates the parameters of a model iteratively.
Starting from some initial guess, each iteration consists of
an E step (Expectation step)
an M step (Maximization step)

5. Applications Filling in missing data in samples
Discovering the value of latent variables
Estimating the parameters of HMMs
Estimating parameters of finite mixtures
Unsupervised learning of clusters …

6. EM Algorithm
Example 
Missing Data

7. Univariate Normal Sample 
Sampling

8. Given x, it is a function of  and 2
Maximum Likelihood  
Sampling
We want to maximize it.
Given x, it is a function of  and 2

9. Log-Likelihood Function
Maximize
this instead
By setting
and

10. Max. the Log-Likelihood Function

11. Max. the Log-Likelihood Function

12. Miss Data
Missing data  
Sampling

13. E-Step
be the estimated parameters at the initial of the tth iterations
Let

14. E-Step
be the estimated parameters at the initial of the tth iterations
Let

15. M-Step
be the estimated parameters at the initial of the tth iterations
Let

16. Exercise n = 40 (10 data missing)
Estimate using different initial conditions.

17. Example  Mixed Attributes
EM Algorithm
Example 
Mixed Attributes

18. Multinomial Population
Sampling
N samples

19. Maximum Likelihood
Sampling
N samples

20. Maximum Likelihood
Sampling
N samples
We want to maximize it.

21. Log-Likelihood

22. Mixed Attributes
Sampling
N samples
x3 is not available

23. E-Step N samples Given (t), what can you say about x3?
Sampling
N samples
x3 is not available
Given (t), what can you say about x3?

24. M-Step

25. Exercise
Estimate  using different initial conditions?

26. EM Algorithm
Example: Mixture

27. Binomial/Poison Mixture
M : married obasong
X : # Children
Binomial/Poison Mixture
# Children n0 n1 n2 n3 n4 n5 n6
# Obasongs
Married Obasongs
Unmarried Obasongs
(No Children)

28. Binomial/Poison Mixture
M : married obasong
X : # Children
Binomial/Poison Mixture
# Children n0 n1 n2 n3 n4 n5 n6
# Obasongs
Married Obasongs
Unmarried Obasongs
(No Children)
Unobserved data:
nA : # married Ob’s
nB : # unmarried Ob’s

29. Binomial/Poison Mixture
M : married obasong
X : # Children
Binomial/Poison Mixture
# Children n0 n1 n2 n3 n4 n5 n6
# Obasongs
Complete
data n1 n2 n3 n4 n5 n6
Probability
pA, pB p1 p2 p3 p4 p5 p6

30. Binomial/Poison Mixture
# Children n0 n1 n2 n3 n4 n5 n6
# Obasongs
Complete
data n1 n2 n3 n4 n5 n6
Probability
pA, pB p1 p2 p3 p4 p5 p6

31. Complete Data Likelihood
# Children n0 n1 n2 n3 n4 n5 n6
# Obasongs
Complete
data n1 n2 n3 n4 n5 n6
Probability
pA, pB p1 p2 p3 p4 p5 p6

32. Complete Data Likelihood
# Children n0 n1 n2 n3 n4 n5 n6
# Obasongs
Complete
data n1 n2 n3 n4 n5 n6
Probability
pA, pB p1 p2 p3 p4 p5 p6

33. Log-Likelihood

34. Maximization

35. Maximization

36. E-Step
Given

37. M-Step

38. Example # Obasongs # Children t   nA nB 3,062 587 284 103 33 4 2 t   nA nB

39. EM Algorithm
Main Body

40. Maximum Likelihood 

41. Latent Variables
Incomplete Data 
Complete Data

42. Complete Data Likelihood

43. Complete Data Likelihood
A function of
latent variable Y
and parameter 
A function of
parameter 
A function of random variable Y.
The result is in term of random variable Y.
If we are given ,
Computable

44. Expectation Step Define
Let (i1) be the parameter vector obtained at the (i1)th step.
Define

45. Maximization Step Define
Let (i1) be the parameter vector obtained at the (i1)th step.
Define

46. EM Algorithm
Mixture Model

47. Mixture Models
If there is a reason to believe that a data set is comprised of several distinct populations, a mixture model can be used.
It has the following form:
with

48. Mixture Models 
Let yi{1,…, M} represents the source that generates the data.

49. Mixture Models 
Let yi{1,…, M} represents the source that generates the data.

50. Mixture Models 

51. Mixture Models

52. Given x and , the conditional density of y can be computed.
Mixture Models
Given x and , the conditional density of y can be computed.

53. Complete-Data Likelihood Function

54. Expectation
g: Guess

55. Expectation
g: Guess

56. Expectation
Zero when yi  l

57. Expectation

58. Expectation

59. Expectation 1

60. Maximization Given the initial guess g,
We want to find , to maximize the above expectation.
In fact, iteratively.

61. The GMM (Guassian Mixture Model)
Guassian model of a d-dimensional source, say j :
GMM with M sources:

62. EM Algorithm
EM-Algorithm on GMM

63. Goal
Mixture Model
subject to
To maximize:

64. Goal Mixture Model Correlated with l only. Correlated with l only.
subject to
To maximize:

65. Finding l
Due to the constraint on l’s, we introduce Lagrange Multiplier , and solve the following equation.

66. Finding l 1 N 1

67. Finding l

68. Only need to maximize
this term
Finding l
Consider GMM
unrelated

69. Finding l How? Therefore, we want to maximize: Only need to maximize
this term
Finding l
Therefore, we want to maximize:
How?
knowledge on matrix algebra is needed.
unrelated

70. Finding l
Therefore, we want to maximize:

71. Summary EM algorithm for GMM
Given an initial guess g, find new as follows
Not converge

72. Demonstration
EM algorithm for Mixture models

73. Exercises
Write a program to generate multidimensional Gaussian distribution.
Draw the distribution for 2-dim data.
Write a program to generate GMM.
Write EM-algorithm to analyze GMM data.
Study more EM-algorithm for mixture.
Find applications for EM-algorithm.

74. References
A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models (1998), Jeff Bilmes
The Expectation Maximization Algorithm: A short tutorial, Sean Borman.