1. EM Algorithm 主講人：虞台文 大同大學資工所 智慧型多媒體研究室

2. Contents Introduction Example  Missing Data Example  Mixed Attributes Example  Mixture Main Body Mixture Model EM-Algorithm on GMM

3. EM Algorithm Introduction 大同大學資工所 智慧型多媒體研究室

4. Introduction EM is typically used to compute maximum likelihood estimates given incomplete samples. The EM algorithm estimates the parameters of a model iteratively. – Starting from some initial guess, each iteration consists of an E step (Expectation step) an M step (Maximization step)

5. Applications Filling in missing data in samples Discovering the value of latent variables Estimating the parameters of HMMs Estimating parameters of finite mixtures Unsupervised learning of clusters …

6. EM Algorithm Example: Missing Data 大同大學資工所 智慧型多媒體研究室

7. Univariate Normal Sample   Sampling

8. Maximum Likelihood   Sampling Given x, it is a function of  and  2 We want to maximize it.

9. Log-Likelihood Function Maximize this instead By setting and

10. Max. the Log-Likelihood Function

12. Miss Data   Sampling Missing data

13. E-Step Let be the estimated parameters at the initial of the t th iterations

14. E-Step Let be the estimated parameters at the initial of the t th iterations

15. M-Step Let be the estimated parameters at the initial of the t th iterations

16. Exercise 375.081556 362.275902 332.612068 351.383048 304.823174 386.438672 430.079689 395.317406 369.029845 365.343938 243.548664 382.789939 374.419161 337.289831 418.928822 364.086502 343.854855 371.279406 439.241736 338.281616 454.981077 479.685107 336.634962 407.030453 297.821512 311.267105 528.267783 419.841982 392.684770 301.910093 n = 40 (10 data missing ) Estimate using different initial conditions.

17. EM Algorithm Example: Mixed Attributes 大同大學資工所 智慧型多媒體研究室

18. Multinomial Population Sampling N samples

19. Maximum Likelihood Sampling N samples

20. Maximum Likelihood Sampling N samples We want to maximize it.

21. Log-Likelihood

22. Mixed Attributes Sampling N samples x 3 is not available

23. E-Step Sampling N samples x 3 is not available Given  (t), what can you say about x 3 ?

24. M-Step

25. Exercise Estimate  using different initial conditions?

26. EM Algorithm Example: Mixture 大同大學資工所 智慧型多媒體研究室

27. Binomial/Poison Mixture # Obasongs n0n0 n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 # Children Married Obasongs Unmarried Obasongs (No Children) M : married obasong X : # Children

28. Binomial/Poison Mixture # Obasongs n0n0 n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 # Children M : married obasong X : # Children Married Obasongs Unmarried Obasongs (No Children) n A : # married Ob’s n B : # unmarried Ob’s Unobserved data:

29. Binomial/Poison Mixture # Obasongs n0n0 n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 # Children M : married obasong X : # Children Complete data n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 Probability p A, p B p1p1 p2p2 p3p3 p4p4 p5p5 p6p6

30. Binomial/Poison Mixture # Obasongs n0n0 n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 # Children Complete data n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 Probability p A, p B p1p1 p2p2 p3p3 p4p4 p5p5 p6p6

31. Complete Data Likelihood # Obasongs n0n0 n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 # Children Complete data n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 Probability p A, p B p1p1 p2p2 p3p3 p4p4 p5p5 p6p6

32. Complete Data Likelihood # Obasongs n0n0 n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 # Children Complete data n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 Probability p A, p B p1p1 p2p2 p3p3 p4p4 p5p5 p6p6

33. Log-Likelihood

34. Maximization

36. E-Step Given

37. M-Step

38. Example # Obasongs # Children 3,0625872841033342 00.750000 0.400000 2502.779 559.221 10.614179 1.035478 2503.591 558.409 20.614378 1.036013 2504.219 557.781 30.614532 1.036427 2504.705 557.295 40.614652 1.036748 2505.081 556.919 50.614744 1.036996 2505.371 556.629 t  nAnA nBnB

39. EM Algorithm Main Body 大同大學資工所 智慧型多媒體研究室

40. Maximum Likelihood 

41. Latent Variables Incomplete Data Complete Data 

42. Complete Data Likelihood Complete Data 

43. Complete Data Likelihood Complete Data A function of parameter  A function of latent variable Y and parameter  If we are given , Computable The result is in term of random variable Y. A function of random variable Y.

44. Expectation Step Let  (i  1) be the parameter vector obtained at the (i  1) th step. Define

45. Maximization Step Let  (i  1) be the parameter vector obtained at the (i  1) th step. Define

46. EM Algorithm Mixture Model 大同大學資工所 智慧型多媒體研究室

47. Mixture Models If there is a reason to believe that a data set is comprised of several distinct populations, a mixture model can be used. It has the following form: with

48. Mixture Models  Let y i  {1,…, M} represents the source that generates the data.

49. Mixture Models  Let y i  {1,…, M} represents the source that generates the data.

50. Mixture Models 

52. Given x and , the conditional density of y can be computed.

53. Complete-Data Likelihood Function 

54. Expectation  g : Guess

55. Expectation  g : Guess

56. Expectation Zero when y i  l

57. Expectation

59. 1

60. Maximization Given the initial guess  g, We want to find , to maximize the above expectation. In fact, iteratively.

61. The GMM (Guassian Mixture Model) Guassian model of a d-dimensional source, say j : GMM with M sources:

62. EM Algorithm EM-Algorithm on GMM 大同大學資工所 智慧型多媒體研究室

63. Goal Mixture Model subject to To maximize:

64. Goal Mixture Model subject to To maximize: Correlated with  l only. Correlated with  l only.

65. Finding  l Due to the constraint on  l ’s, we introduce Lagrange Multiplier, and solve the following equation.

66. Finding  l 1 N 1

68. Finding  l Only need to maximize this term Consider GMM unrelated

69. Finding  l Only need to maximize this term Therefore, we want to maximize: unrelated How? knowledge on matrix algebra is needed.

70. Finding  l Therefore, we want to maximize:

71. Summary EM algorithm for GMM Given an initial guess  g, find  new as follows Not converge

72. Demonstration EM algorithm for Mixture models

73. Exercises Write a program to generate multidimensional Gaussian distribution. Draw the distribution for 2-dim data. Write a program to generate GMM. Write EM-algorithm to analyze GMM data. Study more EM-algorithm for mixture. Find applications for EM-algorithm.