1. Graduate School of Information Sciences, Tohoku University
Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
Phisical Fluctuomatics (Tohoku University)

2. Phisical Fluctuomatics (Tohoku University)
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4.
Phisical Fluctuomatics (Tohoku University)

3. Statistical Machine Learning and Model Selection
Inference of Probabilistic Model by using Data
Model Selection
Statistical Machine Learning
Maximum Likelihood
Complete Data and Incomplete data
EM algorithm
Extension
Implement by Belief Propagation and Markov Chain Monte Carlo method
Akaike Information Criteria, Akaike Bayes Information Criteria, etc.
Phisical Fluctuomatics (Tohoku University)

4. Maximum Likelihood Estimation
Parameter
Phisical Fluctuomatics (Tohoku University)

5. Maximum Likelihood Estimation
Parameter 1 2 3 4 5 6 7 8
Phisical Fluctuomatics (Tohoku University)

6. Maximum Likelihood Estimation
Data
Parameter 1 2 3 4 5 6 7 8
Data
Phisical Fluctuomatics (Tohoku University)

7. Maximum Likelihood Estimation
Data
Parameter 1 2 3 4 5 6 7 8
Data
Histogram
Phisical Fluctuomatics (Tohoku University)

8. Maximum Likelihood Estimation
Data
Parameter 1 2 3 4 5 6 7 8
Data
Histogram
Phisical Fluctuomatics (Tohoku University)

9. Maximum Likelihood Estimation
Data
Parameter 1 2 3 4 5 6 7 8
Data
Histogram
Phisical Fluctuomatics (Tohoku University)

10. Maximum Likelihood Estimation
Data
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.
Phisical Fluctuomatics (Tohoku University)

11. Maximum Likelihood Estimation
Data
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.
Extremum Condition
Phisical Fluctuomatics (Tohoku University)

12. Maximum Likelihood Estimation
Data
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.
Extremum Condition
Sample Deviation
Sample Average
Phisical Fluctuomatics (Tohoku University)

13. Maximum Likelihood Estimation
Data
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.
Extremum Condition
Sample Deviation
Sample Average
Histogram
Phisical Fluctuomatics (Tohoku University)

14. Phisical Fluctuomatics (Tohoku University)
Maximum Likelihood
is unknown
Hyperparameter
Data
Marginal Likelihood
Extremum Condition
Parameter
Bayes Formula
Phisical Fluctuomatics (Tohoku University)

15. Probabilistic Model for Signal Processing
Noise i fi i gi
Transmission
Source Signal
Observable Data
Bayes Formula
Phisical Fluctuomatics (Tohoku University)

16. Prior Probability for Source Signalj
E：Set of all the links
One dimensional Data
Image Data 1 2 3 4 5 1 2 3 4 6 7 8 9 21 22 23 24 5 10 25 11 12 13 14 16 17 18 19 15 20 = 1 2 X 2 3 X 3 4 X 4 5
Phisical Fluctuomatics (Tohoku University)

17. Data Generating Process
Additive White Gaussian Noise
V：Set of all the nodes
Phisical Fluctuomatics (Tohoku University)

18. Probabilistic Model for Signal Processing
Parameter
データ i fi i gi
Hyperparameter
Posterior Probability
Phisical Fluctuomatics (Tohoku University)

19. Maximum Likelihood in Signal Processing
Incomplete Data
Parameter
Data
Marginal Likelihood
Hyperparameter
Extremum Condition
Phisical Fluctuomatics (Tohoku University)

20. Maximum Likelihood and EM algorithm
Incomplete Data
Parameter
Data
Marginal Likelihood
Q function
Hyperparameter
Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood.
Extemum Condition
Phisical Fluctuomatics (Tohoku University)

21. Model Selection in One Dimensional Signal
Expectation Maximization (EM) Algorithm
127
255
100
200
Original Signal
Degraded Signal
Estimated Signal
0.04
0.03
α(t)
0.02
0.01
α(0)=0.0001, σ(0)=100
Phisical Fluctuomatics (Tohoku University)

22. Model Selection in Noise Reduction
MSE
327
36.30
Estimate of Original Image
Original Image
Degraded Image
EM algorithm and Belief Propagation
α(0)=0.0001
σ(0)=100
MSE
260
34.00
Phisical Fluctuomatics (Tohoku University)

23. Phisical Fluctuomatics (Tohoku University)
Summary
Maximum Likelihood and EM algorithm
Statistical Inference by Gaussian Graphical Model
Phisical Fluctuomatics (Tohoku University)

24. Phisical Fluctuomatics (Tohoku University)
Practice 3-1 ,　 , ,　
Let us suppose that data {gi |i=0,1,...,N-1} are generated by according to the following probability density function: ,
where
Prove that estimates for average m and variance s2 of the maximum likelihood ,
are given as
Phisical Fluctuomatics (Tohoku University)