Lecture 6 Resolution and Generalized Inverses. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture.

Slides:



Advertisements
Similar presentations
Environmental Data Analysis with MatLab
Advertisements

Lecture 1 Describing Inverse Problems. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture 03Probability.
Lecture 20 Continuous Problems Linear Operators and Their Adjoints.
Environmental Data Analysis with MatLab Lecture 21: Interpolation.
Lecture 10 Nonuniqueness and Localized Averages. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture.
Lecture 14 Nonlinear Problems Grid Search and Monte Carlo Methods.
Environmental Data Analysis with MatLab Lecture 8: Solving Generalized Least Squares Problems.
Lecture 13 L1 , L∞ Norm Problems and Linear Programming
Lecture 23 Exemplary Inverse Problems including Earthquake Location.
Lecture 22 Exemplary Inverse Problems including Filter Design.
Data Modeling and Parameter Estimation Nov 9, 2005 PSCI 702.
Lecture 18 Varimax Factors and Empircal Orthogonal Functions.
Environmental Data Analysis with MatLab Lecture 16: Orthogonal Functions.
Lecture 7 Backus-Gilbert Generalized Inverse and the Trade Off of Resolution and Variance.
Lecture 3 Probability and Measurement Error, Part 2.
Inverse Theory CIDER seismology lecture IV July 14, 2014
Lecture 2 Probability and Measurement Error, Part 1.
Lecture 5 A Priori Information and Weighted Least Squared.
Lecture 19 Continuous Problems: Backus-Gilbert Theory and Radon’s Problem.
Lecture 15 Nonlinear Problems Newton’s Method. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture.
Lecture 4 The L 2 Norm and Simple Least Squares. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture.
Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals.
Lecture 9 Inexact Theories. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture 03Probability and.
Lecture 17 Factor Analysis. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture 03Probability and.
Lecture 21 Continuous Problems Fréchet Derivatives.
Lecture 24 Exemplary Inverse Problems including Vibrational Problems.
Environmental Data Analysis with MatLab Lecture 5: Linear Models.
Lecture 3 Review of Linear Algebra Simple least-squares.
Curve-Fitting Regression
Lecture 12 Equality and Inequality Constraints. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture.
Lecture 7 Advanced Topics in Least Squares. the multivariate normal distribution for data, d p(d) = (2  ) -N/2 |C d | -1/2 exp{ -1/2 (d-d) T C d -1 (d-d)
Lecture 4: Practical Examples. Remember this? m est = m A + M [ d obs – Gm A ] where M = [G T C d -1 G + C m -1 ] -1 G T C d -1.
Lecture 8 The Principle of Maximum Likelihood. Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture.
3D Geometry for Computer Graphics
Lecture 11 Vector Spaces and Singular Value Decomposition.
Linear and generalised linear models
Basic Mathematics for Portfolio Management. Statistics Variables x, y, z Constants a, b Observations {x n, y n |n=1,…N} Mean.
Linear and generalised linear models
Basics of regression analysis
Lecture 3: Inferences using Least-Squares. Abstraction Vector of N random variables, x with joint probability density p(x) expectation x and covariance.
Environmental Data Analysis with MatLab Lecture 7: Prior Information.
Linear and generalised linear models Purpose of linear models Least-squares solution for linear models Analysis of diagnostics Exponential family and generalised.
Today Wrap up of probability Vectors, Matrices. Calculus
Principles of Pattern Recognition
Some matrix stuff.
Linear Regression Andy Jacobson July 2006 Statistical Anecdotes: Do hospitals make you sick? Student’s story Etymology of “regression”
G(m)=d mathematical model d data m model G operator d=G(m true )+  = d true +  Forward problem: find d given m Inverse problem (discrete parameter estimation):
Modern Navigation Thomas Herring
Unit 3: Matrices.
Curve-Fitting Regression
Multivariate Statistics Matrix Algebra I W. M. van der Veld University of Amsterdam.
SUPA Advanced Data Analysis Course, Jan 6th – 7th 2009 Advanced Data Analysis for the Physical Sciences Dr Martin Hendry Dept of Physics and Astronomy.
Introduction to Matrices and Matrix Approach to Simple Linear Regression.
Colorado Center for Astrodynamics Research The University of Colorado 1 STATISTICAL ORBIT DETERMINATION The Minimum Variance Estimate ASEN 5070 LECTURE.
ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: Normal Equations The Orthogonality Principle Solution of the Normal Equations.
Unit 3: Matrices. Matrix: A rectangular arrangement of data into rows and columns, identified by capital letters. Matrix Dimensions: Number of rows, m,
Geology 5670/6670 Inverse Theory 4 Feb 2015 © A.R. Lowry 2015 Read for Fri 6 Feb: Menke Ch 4 (69-88) Last time: The Generalized Inverse The Generalized.
Environmental Data Analysis with MatLab 2 nd Edition Lecture 22: Linear Approximations and Non Linear Least Squares.
Colorado Center for Astrodynamics Research The University of Colorado 1 STATISTICAL ORBIT DETERMINATION Statistical Interpretation of Least Squares ASEN.
Introduction to Vectors and Matrices
CH 5: Multivariate Methods
Matrices Definition: A matrix is a rectangular array of numbers or symbolic elements In many applications, the rows of a matrix will represent individuals.
Unfolding Problem: A Machine Learning Approach
Linear regression Fitting a straight line to observations.
Unit 3: Matrices
Singular Value Decomposition SVD
5.2 Least-Squares Fit to a Straight Line
Introduction to Vectors and Matrices
Unfolding with system identification
Presentation transcript:

Lecture 6 Resolution and Generalized Inverses

Syllabus Lecture 01Describing Inverse Problems Lecture 02Probability and Measurement Error, Part 1 Lecture 03Probability and Measurement Error, Part 2 Lecture 04The L 2 Norm and Simple Least Squares Lecture 05A Priori Information and Weighted Least Squared Lecture 06Resolution and Generalized Inverses Lecture 07Backus-Gilbert Inverse and the Trade Off of Resolution and Variance Lecture 08The Principle of Maximum Likelihood Lecture 09Inexact Theories Lecture 10Nonuniqueness and Localized Averages Lecture 11Vector Spaces and Singular Value Decomposition Lecture 12Equality and Inequality Constraints Lecture 13L 1, L ∞ Norm Problems and Linear Programming Lecture 14Nonlinear Problems: Grid and Monte Carlo Searches Lecture 15Nonlinear Problems: Newton’s Method Lecture 16Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals Lecture 17Factor Analysis Lecture 18Varimax Factors, Empircal Orthogonal Functions Lecture 19Backus-Gilbert Theory for Continuous Problems; Radon’s Problem Lecture 20Linear Operators and Their Adjoints Lecture 21Fréchet Derivatives Lecture 22 Exemplary Inverse Problems, incl. Filter Design Lecture 23 Exemplary Inverse Problems, incl. Earthquake Location Lecture 24 Exemplary Inverse Problems, incl. Vibrational Problems

Purpose of the Lecture Introduce the idea of a Generalized Inverse, the Data and Model Resolution Matrices and the Unit Covariance Matrix Quantify the spread of resolution and the size of the covariance Use the maximization of resolution and/or covariance as the guiding principle for solving inverse problems

Part 1 The Generalized Inverse, the Data and Model Resolution Matrices and the Unit Covariance Matrix

all of the solutions of the form m est = Md + v

let’s focus on this matrix

m est = G -g d + v rename it the “generalized inverse” and use the symbol G -g

(let’s ignore the vector v for a moment) Generalized Inverse G -g operates on the data to give an estimate of the model parameters if d pre = Gm est then m est = G -g d obs

Generalized Inverse G -g if d pre = Gm est then m est = G -g d obs sort of looks like a matrix inverse except M⨉N, not square and GG -g ≠I and G -g G≠I

so actually the generalized inverse is not a matrix inverse at all

d pre = Gm est and m est = G -g d obs d pre = Nd obs with N = GG -g “data resolution matrix” plug one equation into the other

Data Resolution Matrix, N How much does d i obs contribute to its own prediction? d pre = Nd obs

if N=I d i pre = d i obs d pre = d obs d i obs completely controls its own prediction

d pre d obs = (A) The closer N is to I, the more d i obs controls its own prediction

straight line problem

= ii j j d pre = N d obs only the data at the ends control their own prediction

d obs = Gm true and m est = G -g d obs m est = Rm true with R = G -g G “model resolution matrix” plug one equation into the other

Model Resolution Matrix, R How much does m i true contribute to its own estimated value? m est = Rm true

if R=I m i est = m i true m est = m true m i est reflects m i true only

else if R≠I m i est = … + R i,i-1 m i-1 true + R i,i m i true + R i,i+1 m i+1 true + … m i est is a weighted average of all the elements of m true

m est m true = The closer R is to I, the more m i est reflects only m i true

Discrete version of Laplace Transform large c: d is “shallow” average of m(z) small c: d is “deep” average of m(z)

e -c hi z z m(z) z z ⨉ ⨉ e -c lo z d lo d hi integrate

= ii j j m est = R m true the shallowest model parameters are “best resolved”

Covariance associated with the Generalized Inverse “unit covariance matrix” divide by σ 2 to remove effect of the overall magnitude of the measurement error

unit covariance for straight line problem model parameters uncorrelated when this term zero happens when data are centered about the origin

Part 2 The spread of resolution and the size of the covariance

a resolution matrix has small spread if only its main diagonal has large elements it is close to the identity matrix

“Dirichlet” Spread Functions

a unit covariance matrix has small size if its diagonal elements are small error in the data corresponds to only small error in the model parameters (ignore correlations)

Part 3 minimization of spread of resolution and/or size of covariance as the guiding principle for creating a generalized inverse

over-determined case note that for simple least squares G -g = [G T G] -1 G T model resolution R=G -g G = [G T G] -1 G T G=I always the identify matrix

suggests that we try to minimize the spread of the data resolution matrix, N find G -g that minimizes spread(N)

spread of the k -th row of N now compute

first term

second term third term is zero

putting it all together which is just simple least squares G -g = [G T G] -1 G T

the simple least squares solution minimizes the spread of data resolution and has zero spread of the model resolution

under-determined case note that for minimum length solution G -g = G T [GG T ] -1 data resolution N=GG -g = G G T [GG T ] -1 =I always the identify matrix

suggests that we try to minimize the spread of the model resolution matrix, R find G -g that minimizes spread(R)

minimization leads to [GG T ]G -g = G T which is just minimum length solution G -g = G T [GG T ] -1

the minimum length solution minimizes the spread of model resolution and has zero spread of the data resolution

general case leads to

a Sylvester Equation, so explicit solution in terms of matrices leads to general case

1 special case #1 0 ε2ε2 [G T G+ε 2 I]G -g =G T G -g =[G T G+ε 2 I] -1 G T I damped least squares

0 special case #2 1 ε2ε2 G -g [GG T +ε 2 I] =G T G -g =G T [GG T +ε 2 I] -1 I damped minimum length

so no new solutions have arisen … … just a reinterpretation of previously- derived solutions

reinterpretation instead of solving for estimates of the model parameters We are solving for estimates of weighted averages of the model parameters, where the weights are given by the model resolution matrix

criticism of Direchlet spread() functions when m represents m(x) is that they don’t capture the sense of “being localized” very well

These two rows of the model resolution matrix have the same spread … R ij index, j ii … but the left case is better “localized”

we will take up this issue in the next lecture