July 30th, 2004comp.dsp conference1 Frequency Estimation Techniques Peter J. Kootsookos

July 30th, 2004comp.dsp conference2 Frequency Estimation Techniques Talk Summary Some acknowledgements What is frequency estimation? o What other problems are there? Some algorithms o Maximum likelihood o Subspace techniques o Quinn-Fernandes Associated problems o Analytic signal generation  Kay / Lank-Reed-Pollon estimators o Performance bounds: Cramér-Rao Lower Bound

July 30th, 2004comp.dsp conference3 Frequency Estimation Techniques Some Acknowledgements Eric Jacobson – for his presence on comp.dsp and for his work on the topic. Andrew Reilly – for his presence on comp.dsp and for analytic signal advice. Steven M. Kay – for his books on estimation and detection generally, and published research work on the topic. Barry G. Quinn – as a colleague and for his work the topic. I. Vaughan L. Clarkson – as a colleague and for his work on the topic. CRASys – Now defunct Cooperative Research Centre for Robust & Adaptive Systems.

July 30th, 2004comp.dsp conference4 Frequency Estimation Techniques Talk Summary Some acknowledgements What is frequency estimation? o What other problems are there? Some algorithms o Maximum likelihood o Subspace techniques o Quinn-Fernandes Associated problems o Analytic signal generation  Kay / Lank-Reed-Pollon estimators o Performance bounds: Cramér-Rao Lower Bound

July 30th, 2004comp.dsp conference5 Frequency Estimation Techniques What is frequency estimation? Find the parameters A, , , and  2 in y(t) = A cos [  t- ) +  )] +  (t) where t = 0..T-1,  T-1/2 and  (t) is a noise with zero mean and variance  2.  is used to denote the vector [A    2 ] T.

July 30th, 2004comp.dsp conference6 Frequency Estimation Techniques What other problems are there? y(t) = A cos [  t- ) +  )] +  (t) What about A(t) ? o Estimating A(t) is envelope estimation (AM demodulation). o If the variation of A(t) is slow enough, the problem of estimating  and estimating A(t) decouples. What about  (t)? o This is the frequency tracking problem. What’s  (t) ? o Usually assumed additive, white, & Gaussian. o Maximum likelihood technique depends on Gaussian assumption.

July 30th, 2004comp.dsp conference7 Frequency Estimation Techniques What other problems are there? [continued] Amplitude-varying example: condition monitoring in rotating machinery.

July 30th, 2004comp.dsp conference8 Frequency Estimation Techniques What other problems are there? [continued] Frequency tracking example: SONAR Thanks to Barry Quinn & Ted Hannan for the plot from their book “The Estimation & Tracking of Frequency”.

July 30th, 2004comp.dsp conference9 Frequency Estimation Techniques What other problems are there? [continued] Multi-harmonic frequency estimation y(t) =  A m cos [m  t- ) +  m )] +  (t) For periodic, but not sinusoidal, signals. Each component is harmonically related to the fundamental frequency. p m=1

July 30th, 2004comp.dsp conference10 Frequency Estimation Techniques What other problems are there? [continued] Multi-tone frequency estimation y(t) =  A m cos [  m  t- ) +  m )] +  (t) Here, there are multiple frequency components with no relationship between the frequencies. p m=1

July 30th, 2004comp.dsp conference11 Frequency Estimation Techniques Talk Summary Some acknowledgements What is frequency estimation? o What other problems are there? Some algorithms o Maximum likelihood o Subspace techniques o Quinn-Fernandes Associated problems o Analytic signal generation  Kay / Lank-Reed-Pollon estimators o Performance bounds: Cramér-Rao Lower Bound

July 30th, 2004comp.dsp conference12 Frequency Estimation Techniques The Maximum Likelihood Approach The likelihood function for this problem, assuming that  (t) is Gaussian is L(  ) = 1/((2  ) T/2  |R  |) exp(–(Y –Ŷ(  )) T R -1  (Y –Ŷ(  ))/ 2) where R  = The covariance matrix of the noise  Y = [y(0) y(1) … y(T-1)] T Ŷ = [A cos(  ) A cos(  +  ) … A cos(  (T-1) +  )] T Y is a vector of the date samples, and Ŷ is a vector of the modeled samples.

July 30th, 2004comp.dsp conference13 Frequency Estimation Techniques The Maximum Likelihood Approach [continued] Two points to note: The functional form of the equation L(  ) = 1/((2  ) T/2  |R  |) exp(–(Y –Ŷ(  )) T R -1  (Y –Ŷ(  ))/ 2) is determined by the Gaussian distribution of the noise. If the noise is white, then the covariance matrix R is just  2 I – a scaled identity matrix.

July 30th, 2004comp.dsp conference14 Frequency Estimation Techniques The Maximum Likelihood Approach [continued] Often, it is easier to deal with the log-likelihood function: ℓ (  ) = –(Y –Ŷ (  ) ) T R -1  (Y –Ŷ (  ) ) where the additive constant, and multiplying constant have been ignored as they do not affect the position of the peak (unless  is zero or infinite). If the noise is also assumed to be white, the maximum likelihood problem looks like a least squares problem as maximizing the expression above is the same as minimizing (Y –Ŷ (  ) ) T (Y –Ŷ (  ) )

July 30th, 2004comp.dsp conference15 Frequency Estimation Techniques The Maximum Likelihood Approach [continued] If the complex-valued signal model is used, then estimating  is equivalent to maximizing the periodogram: P(  ) = |  y(t) exp(-i  t) | 2 For the real-valued signal used here, this equivalence is only true as T tends to infinity. t=0 T-1

July 30th, 2004comp.dsp conference16 Frequency Estimation Techniques Talk Summary Some acknowledgements What is frequency estimation? o What other problems are there? Some algorithms o Maximum likelihood o Subspace techniques o Quinn-Fernandes Associated problems o Analytic signal generation  Kay / Lank-Reed-Pollon estimators o Performance bounds: Cramér-Rao Lower Bound

July 30th, 2004comp.dsp conference17 Frequency Estimation Techniques Subspace Techniques The peak of the spectrum produced by spectral estimators other than the periodogram can be used for frequency estimation. Signal subspace estimators use either P Bar (  ) = v*(  ) R Bar v(  ) or P MV (  ) = 1/( v*(  ) R MV -1 v(  ) ) where v(  ) = [ 1 exp(i  exp(i2  exp(I(T-1)  and an estimate of the covariance matrix is used. ^ ^ Note: If R yy is full rank, the P Bar is the same as the periodogram.

July 30th, 2004comp.dsp conference18 Frequency Estimation Techniques Subspace Techniques - Signal Bartlett: R Bar =   k e k e* k Minimum Variance: R MV -1 =  1/  k e k e* k Assuming there are p frequency components. ^ ^ k=1 p p

July 30th, 2004comp.dsp conference19 Frequency Estimation Techniques Subspace Techniques - Noise Pisarenko: R Pis -1 = e p+1 e* p+1 Multiple Signal Classification (MUSIC): R MUSIC -1 =  e k e* k Assuming there are p frequency components. Key Idea: The noise subspace is orthogonal to the signal subspace, so zeros of the noise subspace will indicate signal frequencies. ^ ^ M k=p+1 While Pisarenko is not statistically efficient, it is very fast to calculate.

July 30th, 2004comp.dsp conference20 Frequency Estimation Techniques Quinn-Fernandes The technique of Quinn & Fernandes assumes that the data fits the ARMA(2,2) model: y(t) –  y(t-1) + y(t-2) =  (t) –  (t-1) +  (t-2) 1. Set  1 = 2cos(  ). 2. Filter the data to form z j (t) = y(t) +  j z j (t-1) – z j (t-2) 3. Form  j by regressing ( z j (t) + z j (t-2) ) on z j (t-1)  j =  t  ( z j (t) + z j (t-2) ) z j (t-1) /  t z j 2 (t-1) 4. If |  j -  j | is small enough, set  = cos -1 (  j / 2), otherwise set  j+1 =  j and iterate from 2.

July 30th, 2004comp.dsp conference21 Frequency Estimation Techniques Quinn-Fernandes [continued] The algorithm can be interpreted as finding the maximum of a smoothed periodogram.

July 30th, 2004comp.dsp conference22 Frequency Estimation Techniques Talk Summary Some acknowledgements What is frequency estimation? o What other problems are there? Some algorithms o Maximum likelihood o Subspace techniques o Quinn-Fernandes Associated problems o Analytic signal generation  Kay / Lank-Reed-Pollon estimators o Performance bounds: Cramér-Rao Lower Bound

July 30th, 2004comp.dsp conference23 Frequency Estimation Techniques Associated Problems Other questions that need answering are: What happens when the signal is real-valued, and my frequency estimation technique requires a complex-valued signal? o Analytic Signal generation How well can I estimate frequency? o Cramer-Rao Lower Bound o Threshold performance

July 30th, 2004comp.dsp conference24 Frequency Estimation Techniques Associated Problems: Analytic Signal Generation Many signal processing problems already use “analytic” signals: communications systems with “in-phase” and “quadrature” components, for example. An analytic signal, exp(i-blah), can be generated from a real-valued signal, cos(blah), by use of the Hilbert transform: z(t) = y(t) + i H[ y(t) ] where H[.] is the Hilbert transform operation. Problems occur if the implementation of the Hilbert transform is poor. This can occur if, for example, too short an FIR filter is used.

July 30th, 2004comp.dsp conference25 Frequency Estimation Techniques Associated Problems: Analytic Signal Generation [continued] Another approach is to FFT y(t) to obtain Y(k). From Y(k), form Z(k) = 2Y(k) for k = 1 to T/2 - 1 Y(k) for k = 0 0 for k = T/2 to T and then inverse FFT Z(k) to find z(t). Unless Y(k) is interpolated, this can cause problems. Makes sure the DC term is correct.

July 30th, 2004comp.dsp conference26 Frequency Estimation Techniques Associated Problems: Analytic Signal Generation [continued] If you know something about the signal (e.g. frequency range of interest), then use of a band-pass Hilbert transforming filter is a good option. See the paper by Andrew Reilly, Gordon Fraser & Boualem Boashash, “Analytic Signal Generation : Tips & Traps” IEEE Trans. on ASSP, vol 42(11), pp They suggest designing a real-coefficient low-pass filter with appropriate bandwidth using a good FIR filter algorithm (e.g. Remez). The designed filter is then modulated with a complex exponential of frequency f s /4.

July 30th, 2004comp.dsp conference27 Frequency Estimation Techniques Kay’s Estimator and Related Estimators If an analytic signal, z(t), is obtained, then the simple relation: arg( z(t+1)z*(t) ) can be used to find an estimate of the frequency at time t. See this by writing: z(t+1)z*(t) = exp(i (  (t+1) +  ) ) exp(-i (  t +  ) ) = exp(i  )

July 30th, 2004comp.dsp conference28 Frequency Estimation Techniques Kay’s Estimator and Related Estimators [continued] What Kay did was to form an estimator  = arg( w(t) z(t+1)z*(t) ) where the weights, w(t), are chosen to minimize the mean square error. Kay found that, for very small noise w(t) = 6t(T-t) / (T(T 2 -1)) which is a parabolic window.  T-2 t=0 ^

July 30th, 2004comp.dsp conference29 Frequency Estimation Techniques Kay’s Estimator and Related Estimators [continued] If the SNR is known, then it’s possible to choose an optimal set of weights. For “infinite” noise, the rectangular window is best – this is the Lank- Reed-Pollon estimator. The figure shows how the weights vary with SNR.

July 30th, 2004comp.dsp conference30 Frequency Estimation Techniques Associated Problems: Cramer-Rao Lower Bound The lower bound on the variance of unbiased estimators of the frequency a single tone in noise is var(  ) >= 12  2 / (T(T 2 -1)A 2 ) ^

July 30th, 2004comp.dsp conference31 Frequency Estimation Techniques Associated Problems: Cramer-Rao Lower Bound [continued] The CRLB for the multi-harmonic case is: var(  ) >= 12  2 / (T(T 2 -1) m 2 A m 2 ) So the effective signal energy in this case is influenced by the square of the harmonic order.  p m=1 ^

July 30th, 2004comp.dsp conference32 Frequency Estimation Techniques Associated Problems: Threshold Performance Key idea: The performance degrades when peaks in the noise spectrum exceed the peak of the frequency component. Dotted lines in the figure show the probability of this occurring.

July 30th, 2004comp.dsp conference33 Frequency Estimation Techniques Associated Problems: Threshold Performance [continued] For the multi-harmonic case, two threshold mechanisms occur: the noise outlier case and rational harmonic locking. This means that, sometimes, ½, 1/3, 2/3, 2 or 3 times the true frequency is estimated.

July 30th, 2004comp.dsp conference34 Frequency Estimation Techniques Talk Summary Some acknowledgements What is frequency estimation? o What other problems are there? Some algorithms o Maximum likelihood o Subspace techniques o Quinn-Fernandes Associated problems o Analytic signal generation  Kay / Lank-Reed-Pollon estimators o Performance bounds: Cramér-Rao Lower Bound

July 30th, 2004comp.dsp conference35 Frequency Estimation Techniques Thanks! Thanks to Lori Ann, Al and Rick for hosting and/or organizing this get- together.

July 30th, 2004comp.dsp conference36 Frequency Estimation Techniques Good-bye!