Phisical Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University

Phisical Fluctuomatics (Tohoku University) 2 Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese), Chapter 4.

Phisical Fluctuomatics (Tohoku University) 3 Statistical Machine Learning and Model Selection Inference of Probabilistic Model by using Data Model Selection Statistical Machine Learning Maximum Likelihood EM algorithm Implement by Belief Propagation and Markov Chain Monte Carlo method Akaike Information Criteria, Akaike Bayes Information Criteria, etc. Extension Complete Data and Incomplete data

Phisical Fluctuomatics (Tohoku University) 4 Maximum Likelihood Estimation Parameter

Phisical Fluctuomatics (Tohoku University) 5 Maximum Likelihood Estimation Parameter

Phisical Fluctuomatics (Tohoku University) 6 Maximum Likelihood Estimation Data Parameter Data

Phisical Fluctuomatics (Tohoku University) 7 Maximum Likelihood Estimation Data Parameter Data Histogram

Phisical Fluctuomatics (Tohoku University) 8 Maximum Likelihood Estimation Data Parameter Data Histogram

Phisical Fluctuomatics (Tohoku University) 9 Maximum Likelihood Estimation Data Parameter Data Histogram

Phisical Fluctuomatics (Tohoku University) 10 Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ 2 is regarded as likelihood function for average μ and variance σ 2 when data is given.

Phisical Fluctuomatics (Tohoku University) 11 Maximum Likelihood Estimation Data Extremum Condition Parameter Probability density function for data with average μ and variance σ 2 is regarded as likelihood function for average μ and variance σ 2 when data is given.

Phisical Fluctuomatics (Tohoku University) 12 Maximum Likelihood Estimation Data Extremum Condition Sample Average Sample Deviation Parameter Probability density function for data with average μ and variance σ 2 is regarded as likelihood function for average μ and variance σ 2 when data is given.

Phisical Fluctuomatics (Tohoku University) 13 Maximum Likelihood Estimation Data Extremum Condition Sample Average Sample Deviation Parameter Probability density function for data with average μ and variance σ 2 is regarded as likelihood function for average μ and variance σ 2 when data is given. Histogram

Phisical Fluctuomatics (Tohoku University) 14 Maximum Likelihood Data Extremum Condition Hyperparameter Parameter Bayes Formula is unknown Marginal Likelihood

Phisical Fluctuomatics (Tohoku University) 15 Probabilistic Model for Signal Processing Source SignalObservable Data Transmission Noise i fifi i gigi Bayes Formula

Phisical Fluctuomatics (Tohoku University) 16 Prior Probability for Source Signal Image Data One dimensional Data X XX = E ： Set of all the links ij

Phisical Fluctuomatics (Tohoku University) 17 Data Generating Process Additive White Gaussian Noise V ： Set of all the nodes

Phisical Fluctuomatics (Tohoku University) 18 Probabilistic Model for Signal Processing データ Hyperparameter i fifi i gigi Parameter Posterior Probability

Phisical Fluctuomatics (Tohoku University) 19 Maximum Likelihood in Signal Processing Data Extremum Condition Hyperparameter Parameter Incomplete Data Marginal Likelihood

Phisical Fluctuomatics (Tohoku University) 20 Maximum Likelihood and EM algorithm Data Extemum Condition Hyperparameter Parameter Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood. Q function Marginal Likelihood Incomplete Data

Phisical Fluctuomatics (Tohoku University) 21 Model Selection in One Dimensional Signal Expectation Maximization (EM) Algorithm Original Signal Degraded Signal Estimated Signal α(t)α(t) 0 α(0)=0.0001, σ(0)=100

Phisical Fluctuomatics (Tohoku University) 22 Model Selection in Noise Reduction Original Image Degraded Image EM algorithm and Belief Propagation α(0)= σ(0)=100 Estimate of Original Image MSE MSE

Phisical Fluctuomatics (Tohoku University) 23 Summary Maximum Likelihood and EM algorithm Statistical Inference by Gaussian Graphical Model

Phisical Fluctuomatics (Tohoku University) 24 Let us suppose that data {g i |i=0,1,...,N  1} are generated by according to the following probability density function:, Prove that estimates for average  and variance   of the maximum likelihood are given as, Practice 3-1,,, where