Fundamentals of estimation and detection in signals and images Teaching team : Matthieu Boffety François Goudail

Layout and objective of the course 1. Basics of estimation theory How extract information from a noisy signal in an optimal way ? 2. Basics of detection theory How choose between two hypotheses in an optimal way ? A few new mathematics But mainly a new way of seeing phyiscs …

Organization of the course Lectures : 5 slots (15h) Printed document « Introduction to estimation and detection theory » Books : Ph. Réfrégier, « Noise theory and application to physics » S. Kay, « Fundamentals of statistical signal processing », vol. I and II Lectures : blackboard + slides + exercises Labworks (Matlab) : 5 slots (15h) Signal processing is an experimental science ! Project (Matlab) : write a scientific paper !

FED : Fundamentals of estimation and detection IM3D : Images, mouvement, 3D PAI: Programmation pour les activités de l’ingénieur

Evaluation Project : write a scientific paper (to be handed on 21/10) 1/3 of the final mark Project : write a scientific paper (to be handed on 21/10) Written examination (19/10, 2h, no documents) 1/3 of the final mark Labworks : on-site evaluation for all labworks reports (6 pages max) on labworks 2, 4 and 5 to be handed in and evaluated. 1/3 of the final mark

I. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

I. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

1. The problematic of estimation One observes a series of N measurements assumed to be produced by a random process. They are gathered in a set called a sample. Observed sample : deterministic vector Realization of Random sample : random vector Statistic : Likelihood :

The likelihood i.i.d. sample :

Estimation problem : example Let us consider a i.i.d. sample with exponential PDF: Our objective is to estimate the parameter q. Propose a method ? The mean and the variance of an exponential random variable are : Solution 1: Solution 2: What is the better solution ? Need for a criterion to evaluate the quality of an estimator

II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

Quality of an estimator : True value of the parameter : estimator (statistic) Quality criterion : Mean Square Error MSE can also be written as: where:

Pdf of Standard deviation of the estimator : sT Bias of the estimator : bT

Examples Example 1 : arithmetic mean (as an estimator of the mean) unbiased Example 2 : empirical variance (as an estimator of the variance) biased Gaussian sample:

Estimation problem : example Let us consider a i.i.d. sample with exponential PDF: Our objective is to estimate the parameter q. Propose a method ? The mean and the variance of an exponential random variable are : Solution 1: Solution 2: What is the better estimator ?

Pdf of two estimators Exponential sample, N=100, estimated parameter : mean True parameter : q0=1 Histograms estimated on 105 trials

Examples Example 3 : histogram (as an estimate of the PDF) Estimate : from the sample : Estimator : unbiased Bad estimation if Pk small

II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

Moment estimator Assume that the sample has three parameters : Consider the three first moments The solution of the following system is the moment estimator : Statistical moments Empirical moments Example 1 : Gaussian sample – q=(m,s)

Moment estimator Assume that the sample has three parameters : Consider the three first moments The solution of the following system is the moment estimator : Statistical moments Empirical moments Example 2 : Gamma sample - q=(m, L)

Reduction of speckle noise Spatial averaging of an image: I(i,j) = intensity backscattered at pixel (i,j) of the image. In the presence of speckle, I(i,j) is an exponential RV with mean <I>. Averaging on a region: Mean : Variance :

Density of is the sum of NR exponential RV What is its probability density ? -> Gamma density of order NR (L) L =1 L =10 As L increases, the probability density of the noise becomes « thinner » (the variance decreases), tends to a Gaussian (CLT) L =2

Reduction of speckle noise Gamma noise, L=9 P(I) I Raw image After spatial averaging on a 3x3 neighborhood P(I) I Exponential speckle

Moment estimator Assume that the sample has three parameters : Consider the three first moments Solution of the following system is the moment estimator : Statistical moments Empirical moments Example 2 : Gamma sample - q=(m, L) L =1 L =2 L =10

II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

The likelihood i.i.d. sample :

Example : i.i.d. Gaussian sample Marginal PDF: Loglikelihood ?

Maximum likelihood (ML) estimator

Examples Example 1 : i.i.d. white Gaussian sample For this estimation problem, the ML estimator is identical to the moment estimator

Examples Example 2 : Gamma sample The ML estimator of m is identical to the moment estimator The ML estimator of L is solution of the following equation: with: The ML estimator of L is different from the moment estimator

Examples Example 3 : i.i.d. correlated Gaussian sample Objective : estimate m and G

II. Estimation theory 1. The problematic of estimation 2. Measuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

Cramèr-Rao lower bound Example : i.i.d. exponential sample

Small CRLB : good precision Large CRLB : weak precision

Asymptotic efficiency of the ML estimator

Conditions of strict efficiency of the ML estimator

II. Estimation theory 1. The problematic of estimation 2. Mesuring the quality of an estimator 3. Estimation methods Moment method Maximum likelihood estimator 4. Cramer-Rao Lower Bound – Efficiency of an estimator 5. Robust estimation

Gaussian Cauchy a=0 5 realizations

Gaussian Cauchy a=0.01 5 realizations

Robust estimation Empirical mean Variance Médian