1. Maximum Likelihood (ML), Expectation Maximization (EM)
Pieter Abbeel
UC Berkeley EECS
Many slides adapted from Thrun, Burgard and Fox, Probabilistic Robotics
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAA

2. Outline Maximum likelihood (ML) Priors, and maximum a posteriori (MAP)
Cross-validation
Expectation Maximization (EM)

3. Thumbtack Let µ = P(up), 1-µ = P(down) How to determine µ ?
Empirical estimate: 8 up, 2 down 

5. Maximum Likelihood µ = P(up), 1-µ = P(down) Observe:
Likelihood of the observation sequence depends on µ:
Maximum likelihood finds
extrema at µ = 0, µ = 1, µ = 0.8
Inspection of each extremum yields µML = 0.8

6. Maximum Likelihood
More generally, consider binary-valued random variable with µ = P(1), 1-µ = P(0), assume we observe n1 ones, and n0 zeros
Likelihood:
Derivative:
Hence we have for the extrema:
n1/(n0+n1) is the maximum
= empirical counts.

7. Log-likelihood The function
is a monotonically increasing function of x
Hence for any (positive-valued) function f:
In practice often more convenient to optimize the log- likelihood rather than the likelihood itself
Example:

8. Log-likelihood  Likelihood
Reconsider thumbtacks: 8 up, 2 down
Likelihood
Definition: A function f is concave if and only
Concave functions are generally easier to maximize then non-concave functions
log-likelihood
Not Concave
Concave

9. Concavity and Convexity
f is concave if and only
“Easy” to maximize
f is convex if and only
“Easy” to minimize x1 x2 x1 x2
¸ x2+(1-¸)x2
¸ x2+(1-¸)x2

10. ML for Multinomial Consider having received samples
\theta^{\rm ML} can be found through solving a linear system
Perhaps this derivation is easier with Lagrange multipliers; but those are to be covered later in the course only…

11. ML for Fully Observed HMM
Given samples
Dynamics model:
Observation model:
 Independent ML problems for each and each

12. ML for Exponential Distribution
Source: wikipedia
Consider having received samples
3.1, 8.2, 1.7 ll

13. ML for Exponential Distribution
Source: wikipedia
Consider having received samples

14. Uniform
Consider having received samples

15. ML for Gaussian
Consider having received samples

16. ML for Conditional Gaussian
Equivalently: More generally:

17. ML for Conditional Gaussian

18. ML for Conditional Multivariate Gaussian
To find C: notice that there is effectively twice the same equation: the equation formed by terms 1 and 4, and by 2 and 3, which are the transposes of each other. So if we can solve one of these equations, we are done. Solving is easy to (assuming \Sigma and X^\top X full rank, of course)

19. Aside: Key Identities for Derivation on Previous Slide

20. ML Estimation in Fully Observed Linear Gaussian Bayes Filter Setting
Consider the Linear Gaussian setting:
Fully observed, i.e., given
 Two separate ML estimation problems for conditional multivariate Gaussian: 1: 2:

21. Priors --- Thumbtack Let µ = P(up), 1-µ = P(down) How to determine µ ?
ML estimate: 5 up, 0 down 
Laplace estimate: add a fake count of 1 for each outcome

22. Priors --- Thumbtack Alternatively, consider $µ$ to be random variable
Prior P(µ) / µ(1-µ)
Measurements: P( x | µ )
Posterior:
Maximum A Posterior (MAP) estimation
= find µ that maximizes the posterior 

23. Priors --- Beta Distribution
Figure source: Wikipedia

24. Priors --- Dirichlet Distribution
Generalizes Beta distribution
MAP estimate corresponds to adding fake counts n1, …, nK

25. MAP for Mean of Univariate Gaussian
Assume variance known. (Can be extended to also find MAP for variance.)
Prior:

26. MAP for Univariate Conditional Linear Gaussian
Assume variance known. (Can be extended to also find MAP for variance.)
Prior:
[Interpret!]

27. MAP for Univariate Conditional Linear Gaussian: Example
for run=1:4
a = randn;
b = randn;
x = (rand(5,1) - 0.5);
y = a*x + b + randn(5,1);
X = [ones(5,1) x];
ba_ML = (X'*X)^(-1)*X'*y;
ba_MAP = (eye(2) + X'*X)^(-1)*(X'*y);
figure; plot(x, y, '.');
hold on;
plot(x, ba_ML(1) + ba_ML(2)*x, 'r-');
plot(x, ba_MAP(1) + ba_MAP(2)*x, 'k-');
plot(x, b + a*x, 'g-');
end
TRUE ---
Samples .
ML ---
MAP ---

28. Prior for Multivariate Conditional Linear Gaussian

29. Cross Validation
Choice of prior will heavily influence quality of result
Fine-tune choice of prior through cross-validation:
1. Split data into “training” set and “validation” set
2. For a range of priors,
Train: compute µMAP on training set
Cross-validate: evaluate performance on validation set by evaluating the likelihood of the validation data under µMAP just found
3. Choose prior with highest validation score
For this prior, compute µMAP on (training+validation) set
Typical training / validation splits:
1-fold: 70/30, random split
10-fold: partition into 10 sets, average performance for each of the sets being the validation set and the other 9 being the training set

30. Outline Maximum likelihood (ML) Priors, and maximum a posteriori (MAP)
Cross-validation
Expectation Maximization (EM)

31. Mixture of Gaussians Generally: Example:
ML Objective: given data z(1), …, z(m)
Setting derivatives w.r.t. µ, ¹, § equal to zero does not enable to solve for their ML estimates in closed form
We can evaluate function  we can in principle perform local optimization, see future lectures. In this lecture: “EM” algorithm, which is typically used to efficiently optimize the objective (locally)

32. Expectation Maximization (EM)
Example:
Model:
Goal:
Given data z(1), …, z(m) (but no x(i) observed)
Find maximum likelihood estimates of ¹1, ¹2
EM basic idea: if x(i) were known  two easy-to-solve separate ML problems
EM iterates over
E-step: For i=1,…,m fill in missing data x(i) according to what is most likely given the current model ¹
M-step: run ML for completed data, which gives new model ¹

33. EM Derivation EM solves a Maximum Likelihood problem of the form:
µ: parameters of the probabilistic model we try to find
x: unobserved variables
z: observed variables
Jensen’s Inequality

34. Jensen’s inequality Illustration: P(X=x1) = 1-¸, P(X=x2) = ¸ x1
E[X] = ¸ x2+(1-¸)x2
Illustration:
P(X=x1) = 1-¸,
P(X=x2) = ¸

35. EM Derivation (ctd)
Jensen’s Inequality: equality holds when is an affine
function. This is achieved for
EM Algorithm: Iterate 1. E-step: Compute 2. M-step: Compute
M-step optimization can be done efficiently in most cases
E-step is usually the more expensive step
It does not fill in the missing data x with hard values, but finds a distribution q(x)

36. EM Derivation (ctd)
M-step objective is upper- bounded by true objective
M-step objective is equal to true objective at current parameter estimate
 Improvement in true objective is at least as large as improvement in M-step objective

37. EM 1-D Example --- 2 iterations
Estimate 1-d mixture of two Gaussians with unit variance:
one parameter ¹ ; ¹1 = ¹ - 7.5, ¹2 = ¹+7.5
% true distribution: 3/4 N(-2,1) + 1/4 N(2,1)
clear; clc; close all;
offset = 5; middle = 0;
mu1 = middle-offset/2; mu2 = middle+offset/2; sigma1 = 1; sigma2 = 1;
theta1 = 1/2;
theta2 = 1/2;
x_for_plotting = middle-offset:0.01:middle+offset;
y_for_plotting = theta1*N(x_for_plotting, mu1, sigma1) ...
+ theta2*N(x_for_plotting, mu2, sigma2);
figure; plot(x_for_plotting, y_for_plotting); xlabel('x'); ylabel('probability density');
N_samples = 20;
samples1 = (randn(1, theta1*N_samples)-.5)*2*sigma1 + mu1;
samples2 = (randn(1, theta2*N_samples)-.5)*2*sigma2 + mu2;
samples = [samples1 samples2];
figure; plot(samples, zeros(size(samples)), 'x'); axis([min(samples) max(samples) 0 1]);
% parameterized distribution: 3/4 N(mu, 1) + 1/4 N(mu+offset,1)
% ML problem: find best mu
offset0 = offset+10;
% what we try to fit to the data:
xx_for_plotting = -offset0:0.1:offset0;
yy_for_plotting = theta1*N(xx_for_plotting, -offset0/2, sigma1) + theta2*N(xx_for_plotting, offset0/2, sigma2);
figure; plot(xx_for_plotting, yy_for_plotting);
% let's plot overall landscape:
mu_for_plotting = 4*x_for_plotting;
for i=1:length(mu_for_plotting)
ML_objective(i) = sum(log(theta1*N(samples, mu_for_plotting(i)-offset0/2, sigma1) ...
+ theta2*N(samples, mu_for_plotting(i)+offset0/2, sigma2)));
end
figure; plot(mu_for_plotting, ML_objective); xlabel('mu'); ylabel('log-likelihood');
%init:
mu = middle;
for em_iter = 1:2%:20
mus(em_iter) = mu;
for i=1:length(samples)
p1(i) = theta1*N(samples(i), mu-offset0/2, sigma1);
p2(i) = theta2*N(samples(i), mu+offset0/2, sigma2);
z = p1+p2;
p1 = p1./z;
p2 = p2./z;
EM_bound(i) = sum(p1.*(log(theta1)+logN(samples, mu_for_plotting(i)-offset0/2, sigma1)) ...
+ p2.*(log(theta2)+logN(samples, mu_for_plotting(i)+offset0/2, sigma2))) - sum(p1.*log(p1) + p2.*log(p2));
% figure; plot(mu_for_plotting, ML_objective); xlabel('mu'); ylabel('log-likelihood');
hold on; plot(mu_for_plotting, EM_bound, 'r'); axis([min(mu_for_plotting) max(mu_for_plotting) min(ML_objective) max(ML_objective)]);
ll(em_iter) = sum(log(theta1*N(samples, mu-offset0/2, sigma1) ...
+ theta2*N(samples, mu+offset0/2, sigma2)));
plot(mu, ll(em_iter), 'kx');
mu = (p1*(samples+offset0/2)' + p2*(samples-offset0/2)')/N_samples;
figure; plot(ll);

38. EM for Mixture of Gaussians
X ~ Multinomial Distribution, P(X=k ; µ) = µk
Z ~ N(¹k, §k)
Observed: z(1), z(2), …, z(m)

39. EM for Mixture of Gaussians
E-step:
M-step:

40. EM Run for Mixture of Gaussians
Debugging tip: check log-likelihood score for each step!
should increase, otherwise BUG!
TODO

41. ML Objective HMM Given samples Dynamics model: Observation model:
No simple decomposition into independent ML problems for each and each
No closed form solution found by setting derivatives equal to zero

42. ML Objective HMM ML objective:
Can efficiently evaluate by running (forward) filter + minor additional computation:
Forward filter: for t=0, 1, …, T-1 
We can evaluate function  we can in principle perform local optimization, see future lectures. In this lecture: “EM” algorithm, which is typically used to efficiently optimize the objective (locally)

43. EM for HMM --- M-step
µ and ° computed from “soft” counts

44. EM for HMM --- E-step No need to find conditional full joint
Run smoother to find:

45. EM Example for HMM
TODO: make this slide

46. ML Objective for Linear Gaussians
Linear Gaussian setting:
Given
ML objective:
EM-derivation: same as HMM

47. EM Derivation for Linear Gaussians
TODO type this out (perhaps)

48. EM for Linear Gaussians --- E-Step
Forward:
Backward:

49. EM for Linear Gaussians --- M-step
[Updates for A, B, C, d. TODO: Fill in once found/derived.]

50. EM for Linear Gaussians --- The Log-likelihood
When running EM, it can be good to keep track of the log- likelihood score --- it is supposed to increase every iteration

51. EM for Extended Kalman Filter Setting
As the linearization is only an approximation, when performing the updates, we might end up with parameters that result in a lower (rather than higher) log-likelihood score
 Solution: instead of updating the parameters to the newly estimated ones, interpolate between the previous parameters and the newly estimated ones. Perform a “line- search” to find the setting that achieves the highest log- likelihood score

52. EM Example for Linear Gaussians
TODO