Fundamentals of estimation and detection in signals and images

Slides:



Advertisements
Similar presentations
SOME GENERAL PROBLEMS.
Advertisements

Point Estimation Notes of STAT 6205 by Dr. Fan.
CHAPTER 8 More About Estimation. 8.1 Bayesian Estimation In this chapter we introduce the concepts related to estimation and begin this by considering.
ECE 8443 – Pattern Recognition LECTURE 05: MAXIMUM LIKELIHOOD ESTIMATION Objectives: Discrete Features Maximum Likelihood Resources: D.H.S: Chapter 3 (Part.
1 12. Principles of Parameter Estimation The purpose of this lecture is to illustrate the usefulness of the various concepts introduced and studied in.
Chapter 7 Title and Outline 1 7 Sampling Distributions and Point Estimation of Parameters 7-1 Point Estimation 7-2 Sampling Distributions and the Central.
Estimation  Samples are collected to estimate characteristics of the population of particular interest. Parameter – numerical characteristic of the population.
Fundamental Performance Limits in Image Registration By Dirk Robinson and Peyman Milanfar IEEE Transactions on Image Processing Vol. 13, No. 9, 9/2004.
Chap 8: Estimation of parameters & Fitting of Probability Distributions Section 6.1: INTRODUCTION Unknown parameter(s) values must be estimated before.
STAT 497 APPLIED TIME SERIES ANALYSIS
ELEC 303 – Random Signals Lecture 18 – Statistics, Confidence Intervals Dr. Farinaz Koushanfar ECE Dept., Rice University Nov 10, 2009.
Maximum likelihood (ML) and likelihood ratio (LR) test
Maximum likelihood (ML)
AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory We seek to determine from a set of data, a set of parameters such that their values would.
Maximum likelihood (ML) and likelihood ratio (LR) test
Estimation of parameters. Maximum likelihood What has happened was most likely.
2. Point and interval estimation Introduction Properties of estimators Finite sample size Asymptotic properties Construction methods Method of moments.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
5. Estimation 5.3 Estimation of the mean K. Desch – Statistical methods of data analysis SS10 Is an efficient estimator for μ ?  depends on the distribution.
Chapter 5 Image Restoration. Preview Goal: improve an image in some predefined sense. Image enhancement: subjective process Image restoration: objective.
Maximum likelihood (ML)
Simulation Output Analysis
880.P20 Winter 2006 Richard Kass 1 Maximum Likelihood Method (MLM) Does this procedure make sense? The MLM answers this question and provides a method.
G. Cowan Lectures on Statistical Data Analysis Lecture 3 page 1 Lecture 3 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
A statistical model Μ is a set of distributions (or regression functions), e.g., all uni-modal, smooth distributions. Μ is called a parametric model if.
Maximum Likelihood Estimator of Proportion Let {s 1,s 2,…,s n } be a set of independent outcomes from a Bernoulli experiment with unknown probability.
ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: Deterministic vs. Random Maximum A Posteriori Maximum Likelihood Minimum.
SUPA Advanced Data Analysis Course, Jan 6th – 7th 2009 Advanced Data Analysis for the Physical Sciences Dr Martin Hendry Dept of Physics and Astronomy.
Likelihood Methods in Ecology November 16 th – 20 th, 2009 Millbrook, NY Instructors: Charles Canham and María Uriarte Teaching Assistant Liza Comita.
1 Lecture 16: Point Estimation Concepts and Methods Devore, Ch
ELEC 303 – Random Signals Lecture 18 – Classical Statistical Inference, Dr. Farinaz Koushanfar ECE Dept., Rice University Nov 4, 2010.
PROBABILITY AND STATISTICS FOR ENGINEERING Hossein Sameti Department of Computer Engineering Sharif University of Technology Principles of Parameter Estimation.
1 Statistics and Image Quality Evaluation III Oleh Tretiak Medical Imaging Systems Fall, 2002.
HYPOTHESIS TESTING Distributions(continued); Maximum Likelihood; Parametric hypothesis tests (chi-squared goodness of fit, t-test, F-test) LECTURE 2 Supplementary.
Brief Review Probability and Statistics. Probability distributions Continuous distributions.
UC SF VA Medical Imaging Informatics 2009, Nschuff Course # Slide 1/31 Department of Radiology & Biomedical Imaging MEDICAL IMAGING INFORMATICS:
Statistical Estimation Vasileios Hatzivassiloglou University of Texas at Dallas.
1 Introduction to Statistics − Day 4 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Lecture 2 Brief catalogue of probability.
Machine Learning 5. Parametric Methods.
Friday 23 rd February 2007 Alex 1/70 Alexandre Renaux Washington University in St. Louis Minimal bounds on the Mean Square Error: A Tutorial.
Stats Term Test 4 Solutions. c) d) An alternative solution is to use the probability mass function and.
Week 21 Order Statistics The order statistics of a set of random variables X 1, X 2,…, X n are the same random variables arranged in increasing order.
G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 1 Statistical Data Analysis: Lecture 9 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
Learning Theory Reza Shadmehr Distribution of the ML estimates of model parameters Signal dependent noise models.
Outline Random variables –Histogram, Mean, Variances, Moments, Correlation, types, multiple random variables Random functions –Correlation, stationarity,
Lecture 22 Image Restoration. Image restoration Image restoration is the process of recovering the original scene from the observed scene which is degraded.
Week 21 Statistical Model A statistical model for some data is a set of distributions, one of which corresponds to the true unknown distribution that produced.
Presentation : “ Maximum Likelihood Estimation” Presented By : Jesu Kiran Spurgen Date :
ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.
Labwork 3.
Stat 223 Introduction to the Theory of Statistics
Detection theory 1. Definition of the problematic
HST 583 fMRI DATA ANALYSIS AND ACQUISITION
STATISTICS POINT ESTIMATION
STATISTICAL INFERENCE
Visual Recognition Tutorial
12. Principles of Parameter Estimation
Probability Theory and Parameter Estimation I
Ch3: Model Building through Regression
Image Analysis Image Restoration.
Chapter 2 Minimum Variance Unbiased estimation
Summarizing Data by Statistics
Solving an estimation problem
2.3 Estimating PDFs and PDF Parameters
Stat 223 Introduction to the Theory of Statistics
Computing and Statistical Data Analysis / Stat 7
12. Principles of Parameter Estimation
Applied Statistics and Probability for Engineers
Maximum Likelihood Estimation (MLE)
Presentation transcript:

Fundamentals of estimation and detection in signals and images Teaching team : Matthieu Boffety François Goudail

Layout and objective of the course 1. Basics of estimation theory How extract information from a noisy signal in an optimal way ? 2. Basics of detection theory How choose between two hypotheses in an optimal way ? A few new mathematics But mainly a new way of seeing phyiscs …

Organization of the course Lectures : 5 slots (15h) Printed document « Introduction to estimation and detection theory » Books : Ph. Réfrégier, « Noise theory and application to physics » S. Kay, « Fundamentals of statistical signal processing », vol. I and II Lectures : blackboard + slides + exercises Labworks (Matlab) : 5 slots (15h) Signal processing is an experimental science ! Project (Matlab) : write a scientific paper !

FED : Fundamentals of estimation and detection IM3D : Images, mouvement, 3D PAI: Programmation pour les activités de l’ingénieur

Evaluation Project : write a scientific paper (to be handed on 21/10) 1/3 of the final mark Project : write a scientific paper (to be handed on 21/10) Written examination (19/10, 2h, no documents) 1/3 of the final mark Labworks : on-site evaluation for all labworks reports (6 pages max) on labworks 2, 4 and 5 to be handed in and evaluated. 1/3 of the final mark

I. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

I. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

1. The problematic of estimation One observes a series of N measurements assumed to be produced by a random process. They are gathered in a set called a sample. Observed sample : deterministic vector Realization of Random sample : random vector Statistic : Likelihood :

The likelihood i.i.d. sample :

Estimation problem : example Let us consider a i.i.d. sample with exponential PDF: Our objective is to estimate the parameter q. Propose a method ? The mean and the variance of an exponential random variable are : Solution 1: Solution 2: What is the better solution ? Need for a criterion to evaluate the quality of an estimator

II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

Quality of an estimator : True value of the parameter : estimator (statistic) Quality criterion : Mean Square Error MSE can also be written as: where:

Pdf of Standard deviation of the estimator : sT Bias of the estimator : bT

Examples Example 1 : arithmetic mean (as an estimator of the mean) unbiased Example 2 : empirical variance (as an estimator of the variance) biased Gaussian sample:

Estimation problem : example Let us consider a i.i.d. sample with exponential PDF: Our objective is to estimate the parameter q. Propose a method ? The mean and the variance of an exponential random variable are : Solution 1: Solution 2: What is the better estimator ?

Pdf of two estimators Exponential sample, N=100, estimated parameter : mean True parameter : q0=1 Histograms estimated on 105 trials

Examples Example 3 : histogram (as an estimate of the PDF) Estimate : from the sample : Estimator : unbiased Bad estimation if Pk small

II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

Moment estimator Assume that the sample has three parameters : Consider the three first moments The solution of the following system is the moment estimator : Statistical moments Empirical moments Example 1 : Gaussian sample – q=(m,s)

Moment estimator Assume that the sample has three parameters : Consider the three first moments The solution of the following system is the moment estimator : Statistical moments Empirical moments Example 2 : Gamma sample - q=(m, L)

Reduction of speckle noise Spatial averaging of an image: I(i,j) = intensity backscattered at pixel (i,j) of the image. In the presence of speckle, I(i,j) is an exponential RV with mean <I>. Averaging on a region: Mean : Variance :

Density of is the sum of NR exponential RV What is its probability density ? -> Gamma density of order NR (L) L =1 L =10 As L increases, the probability density of the noise becomes « thinner » (the variance decreases), tends to a Gaussian (CLT) L =2

Reduction of speckle noise Gamma noise, L=9 P(I) I Raw image After spatial averaging on a 3x3 neighborhood P(I) I Exponential speckle

Moment estimator Assume that the sample has three parameters : Consider the three first moments Solution of the following system is the moment estimator : Statistical moments Empirical moments Example 2 : Gamma sample - q=(m, L) L =1 L =2 L =10

II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

The likelihood i.i.d. sample :

Example : i.i.d. Gaussian sample Marginal PDF: Loglikelihood ?

Maximum likelihood (ML) estimator

Examples Example 1 : i.i.d. white Gaussian sample For this estimation problem, the ML estimator is identical to the moment estimator

Examples Example 2 : Gamma sample The ML estimator of m is identical to the moment estimator The ML estimator of L is solution of the following equation: with: The ML estimator of L is different from the moment estimator

Examples Example 3 : i.i.d. correlated Gaussian sample Objective : estimate m and G

II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

Cramèr-Rao lower bound Example : i.i.d. exponential sample

Small CRLB : good precision Large CRLB : weak precision

Asymptotic efficiency of the ML estimator

Conditions of strict efficiency of the ML estimator

II. Estimation theory 1. The problematic of estimation 2. Mesuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

Gaussian Cauchy a=0 5 realizations

Gaussian Cauchy a=0.01 5 realizations

Robust estimation Empirical mean Variance Médian