1. Introduction to Subspace Tracking IEEE AES Dallas Chapter Meeting Richardson, TX Feb 28, 2006 Scott Baker L-3 Communications Greenville, TX scott.baker@L-3com.com

2. Outline Introduction Time Series Data Model Subspaces & the Subspace Tracking Idea Subspace Tracking Categories A Secular Equation Subspace Tracker A Plane Rotation Subspace Tracker Simulations

3. Data Model The time-series model for a sum of complex sinusoids in noise may be written This time series may be placed in the vector form And in a matrix form Type? Singular? Type? Size? Looks like?

4. Data Model Two Typical Goals in Time Series Estimation 1)Parameter estimation Number of signals Q The frequencies themselves The signal amplitudes 2)Noise reduction Find a cleaner version of x(n) Typical assumptions Signal model is stationary over a block Noise is pre-whitened as needed N >= M (more snapshots than sensors)

5. Data Model The data matrix is often place in form of a correlation matrix Data matrix forms lead to an SVD Correlation matrix forms lead to an EVD where What is U? What is V?

6. Data Model The singular value decomposition can be seen in more detail as N x N N x M M x M u i are unitary left singular vectors v i are unitary right singular vectors  i are singular values

7. Data Model The eigenvalue decomposition can be seen in more detail as M x M v i are unitary right eigenvectors i are eigenvalues How is the SVD and EVD related? How are the singular values and eigenvalues related? How are the singular vectors and eigenvectors related? How can we interpret an eigenvalue and an eigenvector?

8. Data Model M = 40; x = sin(2*pi*1 * [1:100]/22); % 1 Hz sinewave, figure; plot(x) X = datamtrx(x, M); R = X'*X; [V, L] = svd(R); figure; plot(V(:,1));hold on; plot(V(:,2)) >> disp(L(1:3,1:3)) 693.4222 0 0 0 531.3492 0 0 0 0.0000 Figure 1 - xFigure 2 – V (1 st two) Box 1 Box 2 N=61 Period = 22 samples Figure 3 – V (last 38) Why not one?

9. Subspaces & Tracking From this example, consider a partition of the EVD Signal Eigenvectors Noise Eigenvectors Signal Eigenvalues “large” Noise Eigenvalues “small” Span? V S and A ?

10. Subspaces & Tracking Extend our previous example, by adding noise M = 40; x = sin(2*pi*1 * [1:100]/22); % 1 Hz sinewave, x = x + randn(1,100); figure; plot(x) X = datamtrx(x, M); R = X'*X; [V, L] = svd(R); figure; plot(V(:,1));hold on; plot(V(:,2)) Box 1 Only one change Figure 1 - x Figure 2 – V (1 st two)Figure 3 – Eigenvalues Effect? Meaning?

11. Knowledge of the signal eigenvectors V S gives us knowledge of the frequency matrix A Knowing the span of V S is equivalent to knowing the span of V N One can show that V S V S H + V N V N H = I MUSIC The MUSIC algorithm uses V N to estimate the frequencies It is possible to show V N H A = 0 Which the same as V N H [a 1,a 2,…,a Q ] = 0 Subspaces & Tracking

12. Recall that the frequency matrix had the form Subspaces & Tracking So, we know the structure of a column and if we had the q th frequency, we could construct the q th column We do know that V N H a q = 0 and we know V N from the decomposition of the data To find  q, what can we do?

13. Yup, good guess! We only need to build a sweep vector a(  ) for  from DC to ½ the sample frequency We can plot the inverse instead so that peaks are at the Q frequencies S(  ) = 1 / || V N H a(  )|| S(  ) is then the frequency spectrum with peaks at the Q frequencies Way cool! Subspaces & Tracking

14. So how do we do this again? Step 1: Build a data matrix Step 2: Build a correlation matrix and find its EVD Step 3: Sort the eigenvalues largest to smallest and partition the EVD into signal portion & noise portion S(  ) = 1 / || V N H a(  )|| Step 4: Compute the MUSIC spectrum & locate Q peaks

15. Subspaces & Tracking randn('state',1); n=[0:99]; x=exp(i*pi/2*n)+2*exp(i*pi/4*n)+exp(i*pi/3*n)+randn(1,100); R=corrmtx(x,12,'mod'); pmusic(R,3,'whole') S(  )  MUSIC Spectrum (type ‘help pmusic’)

16. So are we done yet? In stationary cases, yes In non-stationary cases, no The Q frequencies are often time-varying Hopping sigals Chirping signals Any kind of frequency modulated signal It is too expensive to compute an EVD at every instant of time Instead, use a subspace tracker to track the changes in V S (or equivalently V N ) as new data comes in Subspaces & Tracking

17. Let’s slightly modify our data model so that it is recursive Subspaces & Tracking The factor  is called a fading factor and is slightly less than unity The effective window size is B = 1 / (1-  ) The update assumes that we know the old EVD What type of update is this called? And we wish to know the new EVD Do we need the whole EVD?

18. Subspace Tracking Categories There are about 250 published papers on subspace tracking It sky-rocketed after the invention of the MUSIC algorithm There are several broad categories Secular equation (e.g., Bunch) Plane rotation (e.g., Businger) Optimization Lanczos Perturbation Power Method & Subspace Iteration Eigenvector computation with a priori eigenvalues Analog or Continuous Time Hybrid Techniques

19. The Secular Equation Subspace Tracker Golub discovered the secular equation in 1971 Can be considered the first eigen- based subspace tracker (Businger wrote the first overall ST in 1970) Reveals a nice relation between original and updated eigenvalues Seven years later, Bunch, Nielsen and Sorensen introduced the classic secular equation subspace tracker A keen observation is that the eigenvalues of the new correlation are larger assuming the data has non-zero energy Gene Golub (Stanford) updated eigenvalue, largest original eigenvalue, largest

20. The Secular Equation Subspace Tracker This observation is important because it was noted that the rank-1 modification of the correlation can be written as the rank-1 modification of a diagonal matrix Stick in the EVD Because V is? And z is? What is left in the brackets then? Recall the monotonicity relation

21. The Secular Equation Subspace Tracker where Is D a diagonal matrix? Note that we can form the EVD And so the resultant EVD of the updated correlation is Note we have the form Remember the EVD…

22. The Secular Equation Subspace Tracker Answer: They are equal Hence, to find the updated eigenvalues, we can find the eigenvalue of D. Recall from basic linear algebra, we can write These two expressions can be used to find a formula for the updated eigenvalues Combining these three equations leads to the secular equation

23. The Secular Equation Subspace Tracker Original eigenvalue (p) Updated eigenvalue (z) The secular equation has zeros & poles at the eigenvalue locations Efficient to curve fit here between poles Eigenvector Computation (i.i. not needed)

24. A Plane Rot. Subspace Tracker The first subspace tracker developed in 1970 P. Businger of Bell Labs Used plane rotations Impetus was updating a least squares problem SVD based solution Businger acknowledged the problem was posed by G. Dahlquist and related by G. Golub

25. How might we proceed to update an SVD? A Plane Rot. Subspace Tracker What is Q? Z? Is this in SVD form? What structure does this factor have? Adding a row Similar trick to what?

26. The Q and Z matrices are called plane rotation matrices The specific type of plane rotation used here is called a Given’s rotation A Plane Rot. Subspace Tracker Real Given’s Rotation

27. So how do we use Given’s rotations to diagonalize the matrix? We have to be careful Eliminating one element in a column / row may perturb another column / row Need to apply Given’s rotations to both rows and columns A Plane Rot. Subspace Tracker ** * b a 0 x 0 xr x x Q = Given’s Matrix Original Matrix Transformed Matrix

28. * *** * * * * Q1Q1 = * ** 0 * * * * x * * * * x 0000 x x = * * * * * Q3Q3 00 * A Plane Rot. Subspace Tracker Left (Row) Given’s Rotations First What is the resultant form?

29. A Plane Rot. Subspace Tracker The bidiagonal form is reduced to a diagonal form The process is iterative Start with column rotations (right) Then a set of row rotations (left) Repeat until the energy is minimal off the main diagonal

30. So, going back to our SVD update Q and Z are series of Given’s rotation matrices Q 1 Q 2...Q L Z 1 Z 2 ….Z P And so our final result then is in the proper SVD form A Plane Rot. Subspace Tracker

31. Example Simulation Single signal SNR is 15 dB Sample Rate = 640MHz Linear FM signal 140 to 160 MHz 20MHz / 1000 samples 12.8 GHz / second  = 0.975 (80 samples) TQR + root-MUSIC Spline Interpolation Polynomial = feature This is a spline fit to the first ramp.

32. Other Examples