1 Expectation Maximization Algorithm José M. Bioucas-Dias Instituto Superior Técnico 2005

2 Expectation Maximization (EM) Tool EM approach: Approximate by a tractable iterative procedure Problem: Compute the MAP estimate is often very hard to compute [Dempster et. al, 77], [Little & Rubin, 87], [McLachlan and T. Krishnan, 97]

3 Complete and Missing Data Let where h is a non-invertible function such that Particular case: are termed missing data and complete data, respectively

4 A Minorization Bound for Define E is the Kullback-Leibler distance between the densities Note: Given two densities p and q Facts:

5 EM Concept

6 EM Algorithm Define the sequence Then is non-decreasing Proof Kullback Maximization

7 EM Algorithm (Cont.) Computing Q It is not necessary to compute the second mean value (it does not depend on )

8 EM Acronym E – Expectation M – Maximization

9 The EM rationale remains valid if we replace with i.e., it is not neccessary to maximaze w.r.t. It suffices to assure that increases. Generalized EM (GEM) Algorithm

10 Generalized EM (GEM) Algorithm (cont.)

11 Convergency of Assume that is continuous on both arguments then all limit points of are stationary points of and the GEM sequence converges monotonically to for some stationary point [Wu, 83].

12 References A. Dempster, N. Laird, and D. Rubin. “Maximum likelihood estimation from incomplete data via the EM algorithm.” Journal of the Royal Statistical Society B, vol. 39, pp. 1-38, R. Little and D. Rubin. Statistical Analysis with Missing Data. John Wiley & Sons, New York, 1987 G. McLachlan and T. Krishnan. The EM Algorithm and Extensions. John Wiley & Sons, New York, M. Tanner. Tools for Statistical Inference. Springer-Verlag, New York, C. Wu, “On the convergence properties of the EM algorithm,” The Annal of Statistics, vol. 11, no. 1, pp , 1983