Graduate School of Information Sciences, Tohoku University Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Phisical Fluctuomatics (Tohoku University)

Phisical Fluctuomatics (Tohoku University) Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4. Phisical Fluctuomatics (Tohoku University)

Statistical Machine Learning and Model Selection Inference of Probabilistic Model by using Data Model Selection Statistical Machine Learning Maximum Likelihood Complete Data and Incomplete data EM algorithm Extension Implement by Belief Propagation and Markov Chain Monte Carlo method Akaike Information Criteria, Akaike Bayes Information Criteria, etc. Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Parameter Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Parameter 1 2 3 4 5 6 7 8 Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Data Parameter 1 2 3 4 5 6 7 8 Data Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Data Parameter 1 2 3 4 5 6 7 8 Data Histogram Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Data Parameter 1 2 3 4 5 6 7 8 Data Histogram Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Data Parameter 1 2 3 4 5 6 7 8 Data Histogram Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Sample Deviation Sample Average Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Sample Deviation Sample Average Histogram Phisical Fluctuomatics (Tohoku University)

Phisical Fluctuomatics (Tohoku University) Maximum Likelihood is unknown Hyperparameter Data Marginal Likelihood Extremum Condition Parameter Bayes Formula Phisical Fluctuomatics (Tohoku University)

Probabilistic Model for Signal Processing Noise i fi i gi Transmission Source Signal Observable Data Bayes Formula Phisical Fluctuomatics (Tohoku University)

Prior Probability for Source Signal j E：Set of all the links One dimensional Data Image Data 1 2 3 4 5 1 2 3 4 6 7 8 9 21 22 23 24 5 10 25 11 12 13 14 16 17 18 19 15 20 = 1 2 X 2 3 X 3 4 X 4 5 Phisical Fluctuomatics (Tohoku University)

Data Generating Process Additive White Gaussian Noise V：Set of all the nodes Phisical Fluctuomatics (Tohoku University)

Probabilistic Model for Signal Processing Parameter データ i fi i gi Hyperparameter Posterior Probability Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood in Signal Processing Incomplete Data Parameter Data Marginal Likelihood Hyperparameter Extremum Condition Phisical Fluctuomatics (Tohoku University)

Maximum Likelihood and EM algorithm Incomplete Data Parameter Data Marginal Likelihood Q function Hyperparameter Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood. Extemum Condition Phisical Fluctuomatics (Tohoku University)

Model Selection in One Dimensional Signal Expectation Maximization (EM) Algorithm 127 255 100 200 Original Signal Degraded Signal Estimated Signal 0.04 0.03 α(t) 0.02 0.01 α(0)=0.0001, σ(0)=100 Phisical Fluctuomatics (Tohoku University)

Model Selection in Noise Reduction MSE 327 0.000611 36.30 Estimate of Original Image Original Image Degraded Image EM algorithm and Belief Propagation α(0)=0.0001 σ(0)=100 MSE 260 0.000574 34.00 Phisical Fluctuomatics (Tohoku University)

Phisical Fluctuomatics (Tohoku University) Summary Maximum Likelihood and EM algorithm Statistical Inference by Gaussian Graphical Model Phisical Fluctuomatics (Tohoku University)

Phisical Fluctuomatics (Tohoku University) Practice 3-1 ,　 , ,　 Let us suppose that data {gi |i=0,1,...,N-1} are generated by according to the following probability density function: , where Prove that estimates for average m and variance s2 of the maximum likelihood , are given as Phisical Fluctuomatics (Tohoku University)