Fundamentals of estimation and detection in signals and images Teaching team : Matthieu Boffety François Goudail
Layout and objective of the course 1. Basics of estimation theory How extract information from a noisy signal in an optimal way ? 2. Basics of detection theory How choose between two hypotheses in an optimal way ? A few new mathematics But mainly a new way of seeing phyiscs …
Organization of the course Lectures : 5 slots (15h) Printed document « Introduction to estimation and detection theory » Books : Ph. Réfrégier, « Noise theory and application to physics » S. Kay, « Fundamentals of statistical signal processing », vol. I and II Lectures : blackboard + slides + exercises Labworks (Matlab) : 5 slots (15h) Signal processing is an experimental science ! Project (Matlab) : write a scientific paper !
FED : Fundamentals of estimation and detection IM3D : Images, mouvement, 3D PAI: Programmation pour les activités de l’ingénieur
Evaluation Project : write a scientific paper (to be handed on 21/10) 1/3 of the final mark Project : write a scientific paper (to be handed on 21/10) Written examination (19/10, 2h, no documents) 1/3 of the final mark Labworks : on-site evaluation for all labworks reports (6 pages max) on labworks 2, 4 and 5 to be handed in and evaluated. 1/3 of the final mark
I. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation
I. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation
1. The problematic of estimation One observes a series of N measurements assumed to be produced by a random process. They are gathered in a set called a sample. Observed sample : deterministic vector Realization of Random sample : random vector Statistic : Likelihood :
The likelihood i.i.d. sample :
Estimation problem : example Let us consider a i.i.d. sample with exponential PDF: Our objective is to estimate the parameter q. Propose a method ? The mean and the variance of an exponential random variable are : Solution 1: Solution 2: What is the better solution ? Need for a criterion to evaluate the quality of an estimator
II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation
Quality of an estimator : True value of the parameter : estimator (statistic) Quality criterion : Mean Square Error MSE can also be written as: where:
Pdf of Standard deviation of the estimator : sT Bias of the estimator : bT
Examples Example 1 : arithmetic mean (as an estimator of the mean) unbiased Example 2 : empirical variance (as an estimator of the variance) biased Gaussian sample:
Estimation problem : example Let us consider a i.i.d. sample with exponential PDF: Our objective is to estimate the parameter q. Propose a method ? The mean and the variance of an exponential random variable are : Solution 1: Solution 2: What is the better estimator ?
Pdf of two estimators Exponential sample, N=100, estimated parameter : mean True parameter : q0=1 Histograms estimated on 105 trials
Examples Example 3 : histogram (as an estimate of the PDF) Estimate : from the sample : Estimator : unbiased Bad estimation if Pk small
II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation
Moment estimator Assume that the sample has three parameters : Consider the three first moments The solution of the following system is the moment estimator : Statistical moments Empirical moments Example 1 : Gaussian sample – q=(m,s)
Moment estimator Assume that the sample has three parameters : Consider the three first moments The solution of the following system is the moment estimator : Statistical moments Empirical moments Example 2 : Gamma sample - q=(m, L)
Reduction of speckle noise Spatial averaging of an image: I(i,j) = intensity backscattered at pixel (i,j) of the image. In the presence of speckle, I(i,j) is an exponential RV with mean <I>. Averaging on a region: Mean : Variance :
Density of is the sum of NR exponential RV What is its probability density ? -> Gamma density of order NR (L) L =1 L =10 As L increases, the probability density of the noise becomes « thinner » (the variance decreases), tends to a Gaussian (CLT) L =2
Reduction of speckle noise Gamma noise, L=9 P(I) I Raw image After spatial averaging on a 3x3 neighborhood P(I) I Exponential speckle
Moment estimator Assume that the sample has three parameters : Consider the three first moments Solution of the following system is the moment estimator : Statistical moments Empirical moments Example 2 : Gamma sample - q=(m, L) L =1 L =2 L =10
II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation
The likelihood i.i.d. sample :
Example : i.i.d. Gaussian sample Marginal PDF: Loglikelihood ?
Maximum likelihood (ML) estimator
Examples Example 1 : i.i.d. white Gaussian sample For this estimation problem, the ML estimator is identical to the moment estimator
Examples Example 2 : Gamma sample The ML estimator of m is identical to the moment estimator The ML estimator of L is solution of the following equation: with: The ML estimator of L is different from the moment estimator
Examples Example 3 : i.i.d. correlated Gaussian sample Objective : estimate m and G
II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation
Cramèr-Rao lower bound Example : i.i.d. exponential sample
Small CRLB : good precision Large CRLB : weak precision
Asymptotic efficiency of the ML estimator
Conditions of strict efficiency of the ML estimator
II. Estimation theory 1. The problematic of estimation 2. Mesuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation
Gaussian Cauchy a=0 5 realizations
Gaussian Cauchy a=0.01 5 realizations
Robust estimation Empirical mean Variance Médian