1. Phisical Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/

2. Phisical Fluctuomatics (Tohoku University) 2 Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese), Chapter 4.

3. Phisical Fluctuomatics (Tohoku University) 3 Statistical Machine Learning and Model Selection Inference of Probabilistic Model by using Data Model Selection Statistical Machine Learning Maximum Likelihood EM algorithm Implement by Belief Propagation and Markov Chain Monte Carlo method Akaike Information Criteria, Akaike Bayes Information Criteria, etc. Extension Complete Data and Incomplete data

4. Phisical Fluctuomatics (Tohoku University) 4 Maximum Likelihood Estimation Parameter

5. Phisical Fluctuomatics (Tohoku University) 5 Maximum Likelihood Estimation Parameter 012 345 678

6. Phisical Fluctuomatics (Tohoku University) 6 Maximum Likelihood Estimation Data Parameter 012 345 678 Data

7. Phisical Fluctuomatics (Tohoku University) 7 Maximum Likelihood Estimation Data Parameter 012 345 678 Data Histogram

8. Phisical Fluctuomatics (Tohoku University) 8 Maximum Likelihood Estimation Data Parameter 012 345 678 Data Histogram

9. Phisical Fluctuomatics (Tohoku University) 9 Maximum Likelihood Estimation Data Parameter 012 345 678 Data Histogram

10. Phisical Fluctuomatics (Tohoku University) 10 Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ 2 is regarded as likelihood function for average μ and variance σ 2 when data is given.

11. Phisical Fluctuomatics (Tohoku University) 11 Maximum Likelihood Estimation Data Extremum Condition Parameter Probability density function for data with average μ and variance σ 2 is regarded as likelihood function for average μ and variance σ 2 when data is given.

12. Phisical Fluctuomatics (Tohoku University) 12 Maximum Likelihood Estimation Data Extremum Condition Sample Average Sample Deviation Parameter Probability density function for data with average μ and variance σ 2 is regarded as likelihood function for average μ and variance σ 2 when data is given.

13. Phisical Fluctuomatics (Tohoku University) 13 Maximum Likelihood Estimation Data Extremum Condition Sample Average Sample Deviation Parameter Probability density function for data with average μ and variance σ 2 is regarded as likelihood function for average μ and variance σ 2 when data is given. Histogram

14. Phisical Fluctuomatics (Tohoku University) 14 Maximum Likelihood Data Extremum Condition Hyperparameter Parameter Bayes Formula is unknown Marginal Likelihood

15. Phisical Fluctuomatics (Tohoku University) 15 Probabilistic Model for Signal Processing Source SignalObservable Data Transmission Noise i fifi i gigi Bayes Formula

16. Phisical Fluctuomatics (Tohoku University) 16 Prior Probability for Source Signal Image Data One dimensional Data 12345 12 23 X 34 45 XX = E ： Set of all the links ij 1234 6789 21222324 5 10 25 11121314 16171819 15 20

17. Phisical Fluctuomatics (Tohoku University) 17 Data Generating Process Additive White Gaussian Noise V ： Set of all the nodes

18. Phisical Fluctuomatics (Tohoku University) 18 Probabilistic Model for Signal Processing データ Hyperparameter i fifi i gigi Parameter Posterior Probability

19. Phisical Fluctuomatics (Tohoku University) 19 Maximum Likelihood in Signal Processing Data Extremum Condition Hyperparameter Parameter Incomplete Data Marginal Likelihood

20. Phisical Fluctuomatics (Tohoku University) 20 Maximum Likelihood and EM algorithm Data Extemum Condition Hyperparameter Parameter Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood. Q function Marginal Likelihood Incomplete Data

21. Phisical Fluctuomatics (Tohoku University) 21 Model Selection in One Dimensional Signal Expectation Maximization (EM) Algorithm 0 127 255 0 127 255 0 127 255 100 0 200 100 0 200 100 0 200 Original Signal Degraded Signal Estimated Signal 0.04 0.03 0.02 0.01 α(t)α(t) 0 α(0)=0.0001, σ(0)=100

22. Phisical Fluctuomatics (Tohoku University) 22 Model Selection in Noise Reduction Original Image Degraded Image EM algorithm and Belief Propagation α(0)=0.0001 σ(0)=100 Estimate of Original Image MSE 3270.00061136.30 MSE 2600.00057434.00

23. Phisical Fluctuomatics (Tohoku University) 23 Summary Maximum Likelihood and EM algorithm Statistical Inference by Gaussian Graphical Model

24. Phisical Fluctuomatics (Tohoku University) 24 Let us suppose that data {g i |i=0,1,...,N  1} are generated by according to the following probability density function:, Prove that estimates for average  and variance   of the maximum likelihood are given as, Practice 3-1,,, where