Friday 23 rd February 2007 Alex 1/70 Alexandre Renaux Washington University in St. Louis Minimal bounds on the Mean Square Error: A Tutorial.

Slides:

Advertisements

Similar presentations

Modeling of Data. Basic Bayes theorem Bayes theorem relates the conditional probabilities of two events A, and B: A might be a hypothesis and B might.

Advertisements

Ordinary Least-Squares

CHAPTER 8 More About Estimation. 8.1 Bayesian Estimation In this chapter we introduce the concepts related to estimation and begin this by considering.

CHAPTER 13 M ODELING C ONSIDERATIONS AND S TATISTICAL I NFORMATION “All models are wrong; some are useful.”  George E. P. Box Organization of chapter.

Pattern Recognition and Machine Learning

General Classes of Lower Bounds on Outage Error Probability and MSE in Bayesian Parameter Estimation Tirza Routtenberg Dept. of ECE, Ben-Gurion University.

1 SAMPLE COVARIANCE BASED PARAMETER ESTIMATION FOR DIGITAL COMMUNICATIONS Javier Villares Piera Advisor: Gregori Vázquez Grau Signal Processing for Communications.

1 12. Principles of Parameter Estimation The purpose of this lecture is to illustrate the usefulness of the various concepts introduced and studied in.

Estimation  Samples are collected to estimate characteristics of the population of particular interest. Parameter – numerical characteristic of the population.

Fundamental Performance Limits in Image Registration By Dirk Robinson and Peyman Milanfar IEEE Transactions on Image Processing Vol. 13, No. 9, 9/2004.

NEWCOM – SWP2 MEETING 1 Impact of the CSI on the Design of a Multi-Antenna Transmitter with ML Detection Antonio Pascual Iserte Dpt.

Visual Recognition Tutorial

Maximum likelihood (ML) and likelihood ratio (LR) test

The Mean Square Error (MSE):. Now, Examples: 1) 2)

Point estimation, interval estimation

Maximum likelihood (ML)

AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory We seek to determine from a set of data, a set of parameters such that their values would.

Maximum likelihood (ML) and likelihood ratio (LR) test

Presenting: Assaf Tzabari

Parametric Inference.

2. Point and interval estimation Introduction Properties of estimators Finite sample size Asymptotic properties Construction methods Method of moments.

Introduction to Signal Estimation. 94/10/142 Outline 

G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.

Visual Recognition Tutorial

APPROXIMATE EXPRESSIONS FOR THE MEAN AND COVARIANCE OF THE ML ESTIMATIOR FOR ACOUSTIC SOURCE LOCALIZATION Vikas C. Raykar | Ramani Duraiswami Perceptual.

Maximum likelihood (ML)

G. Hendeby Performance Issues in Non-Gaussian Filtering Problems NSSPW ‘06 Corpus Christi College, Cambridge Performance Issues in Non-Gaussian Filtering.

EE513 Audio Signals and Systems Statistical Pattern Classification Kevin D. Donohue Electrical and Computer Engineering University of Kentucky.

Binary Variables (1) Coin flipping: heads=1, tails=0 Bernoulli Distribution.

Prof. Dr. S. K. Bhattacharjee Department of Statistics University of Rajshahi.

ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: Deterministic vs. Random Maximum A Posteriori Maximum Likelihood Minimum.

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 3: LINEAR MODELS FOR REGRESSION.

FIM Regularity for Gaussian Semi-Blind(SB) MIMO FIR Channel Estimation

Parametric Methods 指導教授：黃文傑 W.J. Huang 學生：蔡漢成 H.C. Tsai.

Semi-Blind (SB) Multiple-Input Multiple-Output (MIMO) Channel Estimation Aditya K. Jagannatham DSP MIMO Group, UCSD ArrayComm Presentation.

School of Electrical & Computer Engineering Image Denoising Using Steerable Pyramids Alex Cunningham Ben Clarke Dy narath Eang ECE November 2008.

ELEC 303 – Random Signals Lecture 18 – Classical Statistical Inference, Dr. Farinaz Koushanfar ECE Dept., Rice University Nov 4, 2010.

PROBABILITY AND STATISTICS FOR ENGINEERING Hossein Sameti Department of Computer Engineering Sharif University of Technology Principles of Parameter Estimation.

0 IEEE SECON 2004 Estimation Bounds for Localization October 7 th, 2004 Cheng Chang EECS Dept,UC Berkeley Joint work with Prof.

BCS547 Neural Decoding. Population Code Tuning CurvesPattern of activity (r) Direction (deg) Activity

Chapter 7 Point Estimation of Parameters. Learning Objectives Explain the general concepts of estimating Explain important properties of point estimators.

BCS547 Neural Decoding.

A Semi-Blind Technique for MIMO Channel Matrix Estimation Aditya Jagannatham and Bhaskar D. Rao The proposed algorithm performs well compared to its training.

EE 3220: Digital Communication

1 Introduction to Statistics − Day 4 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Lecture 2 Brief catalogue of probability.

Baseband Receiver Receiver Design: Demodulation Matched Filter Correlator Receiver Detection Max. Likelihood Detector Probability of Error.

G. Giorgetti, ACM MELT 2008 – San Francisco – September 19, 2008 Localization using Signal Strength: TO RANGE OR NOT TO RANGE? Gianni Giorgetti Sandeep.

Introduction to Estimation Theory: A Tutorial

Yi Jiang MS Thesis 1 Yi Jiang Dept. Of Electrical and Computer Engineering University of Florida, Gainesville, FL 32611, USA Array Signal Processing in.

Week 21 Order Statistics The order statistics of a set of random variables X 1, X 2,…, X n are the same random variables arranged in increasing order.

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 1 Statistical Data Analysis: Lecture 9 1Probability, Bayes’ theorem 2Random variables and.

Parameter Estimation. Statistics Probability specified inferred Steam engine pump “prediction” “estimation”

Performance of Digital Communications System

Digital Communications I: Modulation and Coding Course Spring Jeffrey N. Denenberg Lecture 3c: Signal Detection in AWGN.

Week 21 Statistical Model A statistical model for some data is a set of distributions, one of which corresponds to the true unknown distribution that produced.

Presentation : “ Maximum Likelihood Estimation” Presented By : Jesu Kiran Spurgen Date :

Part 3: Estimation of Parameters. Estimation of Parameters Most of the time, we have random samples but not the densities given. If the parametric form.

Estimator Properties and Linear Least Squares

Visual Recognition Tutorial

12. Principles of Parameter Estimation

Probability Theory and Parameter Estimation I

Fundamentals of estimation and detection in signals and images

Tirza Routtenberg Dept. of ECE, Ben-Gurion University of the Negev

Ch3: Model Building through Regression

Recent Advances in Iterative Parameter Estimation

Special Topics In Scientific Computing

Summarizing Data by Statistics

Solving an estimation problem

12. Principles of Parameter Estimation

Presentation transcript:

Friday 23 rd February 2007 Alex 1/70 Alexandre Renaux Washington University in St. Louis Minimal bounds on the Mean Square Error: A Tutorial

Friday 23 rd February 2007 Alex 2/70 Outline Framework and motivations Minimal bounds on the MSE: unification Minimal bounds on the MSE: application Conclusion and perspectives Minimal bounds on the Mean Square Error

Friday 23 rd February 2007 Alex 3/70 Outline Framework and motivations Minimal bounds on the MSE: unification Minimal bounds on the MSE: application Conclusion and perspectives

Friday 23 rd February 2007 Alex 4/70 Statistical signal processing Extract informations (estimation) Applications: Radar/Sonar Digital communications Medical imaging Astrophysic … Framework and motivations 1

Friday 23 rd February 2007 Alex 5/70 Observations space Parameters space Physical process Observations model Estimation rule performances Statistical framework Framework and motivations 1

Friday 23 rd February 2007 Alex 6/70 Performances r.v. ’’Distance’’ between and Mean Square Error Variance Bias Estimates distibution Framework and motivations 1

Friday 23 rd February 2007 Alex 7/70 Performances: Cramér-Rao inequality For unbiased with Fisher Information Matrix Estimates distribution Cramér-Rao If equality, then efficient estimator Framework and motivations 1

Friday 23 rd February 2007 Alex 8/70 Maximum Likelihood estimators The parameters support is finite Direction of Arrivals estimation Frequency estimation Context Framework and motivations 1

Friday 23 rd February 2007 Alex 9/70 Asymptotic Non-Information SNR T Signal to Noise Ratio (dB) Mean Square Error (dB) MSE behavior of ML estimator: 3 areas Rife and Boorstyn 1974 Van Trees 1968 Framework and motivations 1

Friday 23 rd February 2007 Alex 10/70 Single frequency estimation (100 observations) Normalized frequency Run 1 SNR = 10 dB SNR MSE Framework and motivations 1

Friday 23 rd February 2007 Alex 11/70 Single frequency estimation (100 observations) Run 2 SNR = 10 dB Framework and motivations 1 SNR MSE Normalized frequency

Friday 23 rd February 2007 Alex 12/70 SNR MSE Run 20 SNR = 10 dB Single frequency estimation (100 observations) Framework and motivations 1 Normalized frequency

Friday 23 rd February 2007 Alex 13/70 Run 1 SNR = -6 dB SNR MSE Single frequency estimation (100 observations) Framework and motivations 1 Normalized frequency

Friday 23 rd February 2007 Alex 14/70 Normalized frequency SNR MSE Single frequency estimation (100 observations) Framework and motivations 1 Run 2 SNR = -6 dB

Friday 23 rd February 2007 Alex 15/70 Outlier Normalized frequency SNR MSE Run 7 SNR = -6 dB Single frequency estimation (100 observations) Framework and motivations 1

Friday 23 rd February 2007 Alex 16/70 Outlier Run 1 SNR = -20 dB Normalized frequency SNR MSE Single frequency estimation (100 observations) Framework and motivations 1

Friday 23 rd February 2007 Alex 17/70 Outlier SNR MSE Normalized frequency Run 2 SNR = -20 dB Single frequency estimation (100 observations) Framework and motivations 1

Friday 23 rd February 2007 Alex 18/70 Outlier Normalized frequency SNR MSE Run 20 SNR = -20 dB Single frequency estimation (100 observations) Framework and motivations 1

Friday 23 rd February 2007 Alex 19/70 Asymptotic Non-Information SNR T Signal to Noise Ratio (dB) - Asymptotic MSE - Asymptotic efficiency - Threshold prediction - Global MSE - Ultimate performances Framework and motivations 1 Mean Square Error (dB)

Friday 23 rd February 2007 Alex 20/70 Outline Framework and motivations Minimal bounds on the MSE: unification Minimal bounds on the MSE: application Conclusion and prospect

Friday 23 rd February 2007 Alex 21/70 Minimal bounds on the MSE: unification Asymptotic Non-Information SNR T Signal to Noise Ratio (dB) Mean Square Error (dB) Cramér-Rao bound Insuffisancy of the Cramér- Rao bound - Optimistic - Bias - Threshold Other minimal Bounds (tightest) 2

Friday 23 rd February 2007 Alex 22/70 Two categories Determininstic BoundsBayesian Bounds Deterministic parameters Random parameters Bound the local MSEBound the global MSE Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 23/70 Two categories Determininstic BoundsBayesian Bounds Deterministic parameters Random parameters Bound the local MSEBound the global MSE Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 24/70 Deterministic bounds unification In a class of unbiased estimator, we want to find the particular estimator for which the variance is minimal at the true value of the parameter Class of unbiased estimator ???? Constrained optimization problem Minimal bounds on the MSE: unification 2 Barankin Approach Glave IEEE IT 1973

Friday 23 rd February 2007 Alex 25/70 Bias Barankin (1949) Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 26/70 Deterministic bounds unification Minimal bounds on the MSE: unification 2 Needs the resolution of an integral equation Sometimes, doesn’t exist Barankin

Friday 23 rd February 2007 Alex 27/70 Bias Cramér-Rao Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 28/70 Deterministic bounds unification Minimal bounds on the MSE: unification 2 CramerRaoFisherFrechetDarmois

Friday 23 rd February 2007 Alex 29/70 Bias Bhattacharyya (1946) Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 30/70 Bias BhattacharyyaBarankin Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 31/70 Deterministic bounds unification Minimal bounds on the MSE: unification 2 ? BhattacharyyaGuttmanFraser

Friday 23 rd February 2007 Alex 32/70 Bias McAulay-Seidman (1969) Test points (Barankin) Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 33/70 Bias McAulay-SeidmanBarankin Test points Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 34/70 Deterministic bounds unification Minimal bounds on the MSE: unification How to choose test points ? 2

Friday 23 rd February 2007 Alex 35/70 Bias Chapman-Robbins (1951) 1 test point Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 36/70 Deterministic bounds unification Minimal bounds on the MSE: unification 2 Chapman Robbins HammersleyKiefer

Friday 23 rd February 2007 Alex 37/70 Bias Abel (1993) Deterministic bounds unification Minimal bounds on the MSE: unification Test points 2

Friday 23 rd February 2007 Alex 38/70 Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 39/70 Bias Quinlan-Chaumette-Larzabal (2006) Test points Deterministic bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 40/70 Deterministic bounds f 0 =0, K=32 observtions Don’t take into accout the support of the parameter Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 41/70 Already used in Signal Processing Deterministic bounds Minimal bounds on the MSE: unification 2 CRB for wide range of topics ChRB and Barankin (McAulay-Seidman version) Time delay estimation DOA estimation Digital communications (synchronization parameters) Abel bound Digital communications (synchronization parameters)

Friday 23 rd February 2007 Alex 42/70 Two categories Determininstic BoundsBayesian Bounds Deterministic parameters Random parameters Bound the local MSEBound the global MSE Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 43/70 Best Bayesian bound: MSE of the conditional mean estimator (MMSEE) is the solution of Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 44/70 For your information Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 45/70 Best Bayesian bound Minimal bound Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 46/70 Degres of freedom Constrained optimization problem Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 47/70 s h 1 Best Bayesian bound Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 48/70 s h 1 Bayesian Cramér-Rao bound (Van Trees 1968) Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 49/70 Bayesian bounds unification Minimal bounds on the MSE: unification 2 Van Trees

Friday 23 rd February 2007 Alex 50/70 s h 1 Reuven-Messer bound (1997) (Bayesian Barankin bound) Bobrovsky-Zakaï bound (1976) (1 test point) Test points … Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 51/70 Bayesian bounds unification Minimal bounds on the MSE: unification 2 ReuvenMesserBobrovskyZakai

Friday 23 rd February 2007 Alex 52/70 h 1 Bayesian Bhattacharyya bound (Van Trees 1968) Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 53/70 Bayesian bounds unification Minimal bounds on the MSE: unification 2 Van Trees

Friday 23 rd February 2007 Alex 54/70 s h 1 Weiss-Weinstein bound (1985) Bayesian bounds unification Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 55/70 Bayesian bounds unification Minimal bounds on the MSE: unification 2 WeissWeinstein

Friday 23 rd February 2007 Alex 56/70 Relationship between deterministic and Bayesian bounds Deterministic boundsBayesian bounds Constrained optimization problem Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 57/70 Deterministic bounds Cramér-Rao Bhattacharyya Chapman-Robbins McAulay-Seidman Abel ??? Bayesian bounds Bayesian Cramér-Rao Bayesian Bhattacharyya Bobrovsky-Zakaï Reuven-Messer ??? Weiss-Weinstein Relationship between deterministic and Bayesian bounds Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 58/70 Deterministic bounds Cramér-Rao Bhattacharyya Chapman-Robbins McAulay-Seidman Abel ??? Bayesian bounds Bayesian Cramér-Rao Bayesian Bhattacharyya Bobrovsky-Zakaï Reuven-Messer Bayesian Abel Weiss-Weinstein Relationship between deterministic and Bayesian bounds Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 59/70 Deterministic bounds Cramér-Rao Bhattacharyya Chapman-Robbins McAulay-Seidman Abel Deterministic Weiss-Weinstein Bayesian bounds Bayesian Cramér-Rao Bayesian Bhattacharyya Bobrovsky-Zakaï Reuven-Messer Bayesian Abel Weiss-Weinstein Relationship between deterministic and Bayesian bounds Minimal bounds on the MSE: unification 2

Friday 23 rd February 2007 Alex 60/70 Already used in Signal Processing Minimal bounds on the MSE: unification 2 Bayesian bounds BCRB and Bayesian Barankin bound ??? Bobrovsky-Zakai and Bayesian Abel bounds Digital communications (synchronization parameters) Weiss-Weinstein bounds Spectral Analysis Underwater acoustic Array Processing Digital communications Physics (gravitational waves)

Friday 23 rd February 2007 Alex 61/70 Outline Framework and motivations Minimal bounds on the MSE: unification Minimal bounds on the MSE: application Conclusion and perspectives

Friday 23 rd February 2007 Alex 62/70 Synchronization problem (spectral analysis) Observations (complex) Pilot symbols (known) Additive noise, Gaussian, circular, iid Parameter of interest deterministic or random 3 Minimal bounds on the MSE: application

Friday 23 rd February 2007 Alex 63/70 QPSK Modulation 20 observations Maximum Likelihood Local MSE Synchronization problem (spectral analysis) 3 Minimal bounds on the MSE: application

Friday 23 rd February 2007 Alex 64/70 Bayesian bounds for deterministic parameters estimators The support of the parameter is taken into account Synchronization problem (spectral analysis) 3 Minimal bounds on the MSE: application

Friday 23 rd February 2007 Alex 65/70 QPSK Modulation 20 observations Synchronization problem (spectral analysis) 3 Minimal bounds on the MSE: application Maximum Likelihood Global MSE

Friday 23 rd February 2007 Alex 66/70 Outline Framework and motivations Minimal bounds on the MSE: unification Minimal bounds on the MSE : application Conclusion and perspectives

Friday 23 rd February 2007 Alex 67/70 Conclusion and perspectives 4 Contributions Deterministic bounds unification Bayesian bounds unification Bayesian Abel bound and deterministic Weiss-Weinstein bound Closed-form expressions of the minimal bounds in a synchronization framework (+ Gaussian observation model with parameterized mean)

Friday 23 rd February 2007 Alex 68/70 BTW -Understanding the CRB - CRB for Singular FIM - Ziv-Zakai Family (Bayesian) - Minimal bounds for discret time filtering (Bayesian) - Some global class of Cramér-Rao bound (Bayesian) - Hybrid bounds (mixing non-random and random parameters) Conclusion and perspectives 4

Friday 23 rd February 2007 Alex 69/70 Perspectives Estimators set for Bayesian bounds Interpretation in terms of bias of the Weiss-Weinstein bound Closed form expressions for the multiple parameters case Conclusion and perspectives 4

Friday 23 rd February 2007 Alex 70/70 Thank you Minimal bounds on the Mean Square Error