Expectation-Maximization (EM) Algorithm & Monte Carlo Sampling for Inference and Approximation.

Slides:

Advertisements

Similar presentations

Expectation-Maximization (EM) Algorithm Md. Rezaul Karim Professor Department of Statistics University of Rajshahi Bangladesh September 21, 2012.

Advertisements

EMNLP, June 2001Ted Pedersen - EM Panel1 A Gentle Introduction to the EM Algorithm Ted Pedersen Department of Computer Science University of Minnesota.

Hypothesis testing and confidence intervals by resampling by J. Kárász.

Expectation Maximization

Maximum Likelihood And Expectation Maximization Lecture Notes for CMPUT 466/551 Nilanjan Ray.

Estimation  Samples are collected to estimate characteristics of the population of particular interest. Parameter – numerical characteristic of the population.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Jensen’s Inequality (Special Case) EM Theorem.

Segmentation and Fitting Using Probabilistic Methods

Statistical Topic Modeling part 1

Hidden Variables, the EM Algorithm, and Mixtures of Gaussians Computer Vision CS 143, Brown James Hays 02/22/11 Many slides from Derek Hoiem.

Hidden Variables, the EM Algorithm, and Mixtures of Gaussians Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 03/15/12.

. Learning – EM in The ABO locus Tutorial #8 © Ilan Gronau. Based on original slides of Ydo Wexler & Dan Geiger.

Mixture Language Models and EM Algorithm

Visual Recognition Tutorial

EE-148 Expectation Maximization Markus Weber 5/11/99.

Data Mining Techniques Outline

Lecture 5: Learning models using EM

. Learning – EM in The ABO locus Tutorial #9 © Ilan Gronau.

CSE 221: Probabilistic Analysis of Computer Systems Topics covered: Statistical inference (Sec. )

Chapter 14 Simulation. Monte Carlo Process Statistical Analysis of Simulation Results Verification of the Simulation Model Computer Simulation with Excel.

Expectation-Maximization (EM) Chapter 3 (Duda et al.) – Section 3.9

. Learning – EM in The ABO locus Tutorial #9 © Ilan Gronau. Based on original slides of Ydo Wexler & Dan Geiger.

7-1 Introduction The field of statistical inference consists of those methods used to make decisions or to draw conclusions about a population. These.

CSE 221: Probabilistic Analysis of Computer Systems Topics covered: Statistical inference.

Overview G. Jogesh Babu. Probability theory Probability is all about flip of a coin Conditional probability & Bayes theorem (Bayesian analysis) Expectation,

AP STATISTICS “Do Cell Phones Distract Drivers?”.

Topic Models in Text Processing IR Group Meeting Presented by Qiaozhu Mei.

Random Sampling, Point Estimation and Maximum Likelihood.

Probabilistic Mechanism Analysis. Outline Uncertainty in mechanisms Why consider uncertainty Basics of uncertainty Probabilistic mechanism analysis Examples.

Machine Learning Lecture 23: Statistical Estimation with Sampling Iain Murray’s MLSS lecture on videolectures.net:

Cognitive Computer Vision Kingsley Sage and Hilary Buxton Prepared under ECVision Specific Action 8-3

Lecture 19: More EM Machine Learning April 15, 2010.

7-1 Introduction The field of statistical inference consists of those methods used to make decisions or to draw conclusions about a population. These.

Monte Carlo Methods Versatile methods for analyzing the behavior of some activity, plan or process that involves uncertainty.

Computer Science, Software Engineering & Robotics Workshop, FGCU, April 27-28, 2012 Fault Prediction with Particle Filters by David Hatfield mentors: Dr.

14.3 Simulation Techniques and the Monte Carlo Method simulation technique A simulation technique uses a probability experiment to mimic a real-life situation.

Mixture Models, Monte Carlo, Bayesian Updating and Dynamic Models Mike West Computing Science and Statistics, Vol. 24, pp , 1993.

7.1: What is a Sampling Distribution?!?!. Section 7.1 What Is a Sampling Distribution? After this section, you should be able to… DISTINGUISH between.

Monte Carlo Process Risk Analysis for Water Resources Planning and Management Institute for Water Resources 2008.

Chapter 7 Point Estimation of Parameters. Learning Objectives Explain the general concepts of estimating Explain important properties of point estimators.

Computer Vision Lecture 6. Probabilistic Methods in Segmentation.

Markov Chain Monte Carlo for LDA C. Andrieu, N. D. Freitas, and A. Doucet, An Introduction to MCMC for Machine Learning, R. M. Neal, Probabilistic.

ECE 8443 – Pattern Recognition Objectives: Jensen’s Inequality (Special Case) EM Theorem Proof EM Example – Missing Data Intro to Hidden Markov Models.

Week 41 How to find estimators? There are two main methods for finding estimators: 1) Method of moments. 2) The method of Maximum likelihood. Sometimes.

CSE 517 Natural Language Processing Winter 2015

1 Chapter 8: Model Inference and Averaging Presented by Hui Fang.

Discriminative Training and Machine Learning Approaches Machine Learning Lab, Dept. of CSIE, NCKU Chih-Pin Liao.

CS Statistical Machine learning Lecture 25 Yuan (Alan) Qi Purdue CS Nov

Design and Implementation of Speech Recognition Systems Fall 2014 Ming Li Special topic: the Expectation-Maximization algorithm and GMM Sep Some.

Hidden Variables, the EM Algorithm, and Mixtures of Gaussians Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 02/22/11.

Hidden Markov Models. A Hidden Markov Model consists of 1.A sequence of states {X t |t  T } = {X 1, X 2,..., X T }, and 2.A sequence of observations.

For multivariate data of a continuous nature, attention has focussed on the use of multivariate normal components because of their computational convenience.

Machine Learning Expectation Maximization and Gaussian Mixtures CSE 473 Chapter 20.3.

HW7: Evolutionarily conserved segments ENCODE region 009 (beta-globin locus) Multiple alignment of human, dog, and mouse 2 states: neutral (fast-evolving),

Computational Intelligence: Methods and Applications Lecture 26 Density estimation, Expectation Maximization. Włodzisław Duch Dept. of Informatics, UMK.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Jensen’s Inequality (Special Case) EM Theorem.

Machine Learning Expectation Maximization and Gaussian Mixtures CSE 473 Chapter 20.3.

10 October, 2007 University of Glasgow 1 EM Algorithm with Markov Chain Monte Carlo Method for Bayesian Image Analysis Kazuyuki Tanaka Graduate School.

Generalization Performance of Exchange Monte Carlo Method for Normal Mixture Models Kenji Nagata, Sumio Watanabe Tokyo Institute of Technology.

Statistical Estimation

7-1 Introduction The field of statistical inference consists of those methods used to make decisions or to draw conclusions about a population. These.

Chapter Six Normal Curves and Sampling Probability Distributions

Sampling Distributions

LECTURE 10: EXPECTATION MAXIMIZATION (EM)

Topic Models in Text Processing

Graduate School of Information Sciences, Tohoku University

Sampling Distributions (§ )

MGSE7.SP.3/MGSE7.SP.4: I can use measure of center and measures of variability for numerical data from random samples to draw informal comparative inferences.

Graduate School of Information Sciences, Tohoku University

Maximum Likelihood Estimation (MLE)

Presentation transcript:

Expectation-Maximization (EM) Algorithm & Monte Carlo Sampling for Inference and Approximation

Expectation-Maximization Algorithm “The Expectation-Maximization algorithm is a general technique for finding maximum likelyhood * solutions for probabilistic models having latent variables” (Dempster et al., 1977; McLachlan and Krishnan, 1997). Is an iterative process and consists of two steps: E-step and M-step. General purpose technique: - Needs to be adapted for each application - Versatile. Used in machine learning, computer vision, language processing....

Intro: Maximum Likelihood Estimation methods Maximum Likelihood Estimation (MLE) are methods to estimate parameters of an unknown, parameter-dependent probability density function p( x | θ ) from the observed sample (x 1,x 2,...,x n ).

- When is EM useful? - When MLE solutions are difficult or not possible to get because there are latent variables involved. - Either missing values or we decide to get aditional unkown variables for modelling simplicity.

EM summarized

Monte Carlo Sampling for Inference and Approximation - Inference – To draw conclusions from gathered data. - Monte Carlo Sampling – Broad selection of computational algorithms that rely on repeated random sampling to obtain numerical results. - For a better understanding we have prepared two very simple examples.

Rolling a dice - We know that the probability of getting a 4 is: - 1/6 (approx 17%) - Can we obtain the same result by Monte Carlo simulation? - More iterations give less error in the result!

Calculating the area of the unit circle 10 iterations of Monte Carlo Ratio: 2.4

Calculating the area of the unit circle 1000 iterations of Monte Carlo Ratio: 3.04

Calculating the area of the unit circle - 1 million iterations: - Ratio: million iterations: - Ratio: And so forth!

Application of EM -Pattern Recognition -Image Recognition -Computer vision -Maximum likelihood -Bioinformatics

Application of MC -Finance -Statistics -Molecular dynamics -Computer Graphics -Fluid mechanics