1. The Profound Impact of Negative Power Law Noise on the Estimation of Causal Behavior Victor S. Reinhardt Raytheon Space and Airborne Systems El Segundo, CA, USA 2009 Joint Meeting of the European Frequency and Time Forum and the IEEE International Frequency Control Symposium, Besançon, France

2. Page 2 EFTF-IFCS 2009 -- V. Reinhardt Introduction Well-known that correlated or systematic noiseWell-known that correlated or systematic noise  Cannot be properly separated from true causal behavior imbedded in noisy data  By any fitting or estimation technique  i.e., Least Squares Fit (LSQF) or Kalman filter But the profound implications of thisBut the profound implications of this  In dealing with highly correlated negative power law (neg-p) noise  Have not been fully appreciated  Neg-p  PSD L p (f)  |f| p for p < 0 This paper will investigate these implicationsThis paper will investigate these implications

3. Page 3 EFTF-IFCS 2009 -- V. Reinhardt Introduction Paper will 1 st show that such neg-p correlationsPaper will 1 st show that such neg-p correlations  Lead such fitting techniques  To generate anomalous fit solutions  And faulty estimates of the true fit errors Then it will explore profound consequences of this in a variety of areasThen it will explore profound consequences of this in a variety of areas

4. Page 4 EFTF-IFCS 2009 -- V. Reinhardt x(t) t x(t n ) Over Data Observation Interval T = Nt s (t n =nt s ) Model We Will Use for Estimating Causal Behavior in Noisy Data x(t) = Any data variable representing a measurable output from an instrument N Samples of Data x(t n )   Can also be fn of  (t) but x(t,  (t))  x(t) for most of paper

5. Page 5 EFTF-IFCS 2009 -- V. Reinhardt Model We Will Use for Estimating Causal Behavior in Noisy Data For example x(t) = x osc (t) – x osc (t-  ) is observable output of 2-way ranging RxFor example x(t) = x osc (t) – x osc (t-  ) is observable output of 2-way ranging Rx Where x osc (t) is the unobservable true time error of the reference oscillatorWhere x osc (t) is the unobservable true time error of the reference oscillator

6. Page 6 EFTF-IFCS 2009 -- V. Reinhardt x(t) t Over Data Observation Interval T = Nt s (t n =nt s ) Model We Will Use for Estimating Causal Behavior in Noisy Data x(t) is sum of (true) causal behavior x c (t) plus (true) noise x r (t) (True) Causal Behavior x c (t) (True) Noise x r (t) = x c (t)+ x r (t)

7. Page 7 EFTF-IFCS 2009 -- V. Reinhardt x(t) t Over Data Observation Interval T = Nt s (t n =nt s ) Model We Will Use for Estimating Causal Behavior in Noisy Data Fitting process generates x a,M (t) an M- Parameter estimate of x c (t) (True) Causal Behavior x c (t) x a,M (t) = M-Parameter Estimate of x c (t) (True) Noise x r (t) = x c (t)+ x r (t)

8. Page 8 EFTF-IFCS 2009 -- V. Reinhardt x(t n ) x c (t)x a,M (t n ) Data interval T x(t) t x w,M (t n ) = x a,M (t n ) – x c (t n ) x w,M (t)  only detailed measure of the true accuracy The Accuracy of the Causal Estimate is x w,M (t n ) ) Point Variance of x w,M (t n ) (Kalman) ( E x r = 0)  w,M (t n ) 2 = E x w,M (t n ) 2 E = Ensemble average Weighted Average Variance over T (LSQF)  w,M 2 =  n  n  w,M (t n ) 2  n = data weighting

9. Page 9 EFTF-IFCS 2009 -- V. Reinhardt A Major Thesis of This Paper is Cannot substitute other error measures for  w,M (t n ) 2 or  w,M 2Cannot substitute other error measures for  w,M (t n ) 2 or  w,M 2  Just because they diverge due to neg-p noise And that such divergences are in fact indicators of a real inaccuracy problem with the fit solutionAnd that such divergences are in fact indicators of a real inaccuracy problem with the fit solution

10. Page 10 EFTF-IFCS 2009 -- V. Reinhardt x(t) t x(t n ) x c (t)x a,M (t n ) Data interval T x w,M But the Accuracy x w,M is Not an Observable Error Measure  j,M (t n ) 2 = E x j,M (t n ) 2  j,M 2 =  n  n  j,M (t n ) 2 Only true observable is x j,M (t n ) = x(t n ) – x a,M (t n ) the Data Precision

11. Page 11 EFTF-IFCS 2009 -- V. Reinhardt x(t) t x(t n ) x c (t)x a,M (t n ) Data interval T x w,M But the Accuracy x w,M is Not an Observable Error Measure x j,M (t n ) 2  wj,M (t n ) Estimates accuracy based on theoretical  d (t n ) &  d From Data Precision can form Fit Precision  wj,M (t n ) 2 =  d (t n )  j,M (t n ) 2  wj,M 2 =  d  j,M 2

12. Page 12 EFTF-IFCS 2009 -- V. Reinhardt But the Accuracy x w,M is Not an Observable Error Measure Can showuniform LSQF & uncorrelated x r (t n )Can show for uniform LSQF & uncorrelated x r (t n )    do = M/(N – M) Will show  do & equiv  do (t n ) can generate very misleading accuracy estimates when neg-p noise is presentWill show  do & equiv  do (t n ) can generate very misleading accuracy estimates when neg-p noise is present

13. Page 13 EFTF-IFCS 2009 -- V. Reinhardt x(t) t x(t n ) x c (t)x a,M (t n ) Data interval T M th order  -Measures over Interval    ,2 (t n ) 2  Allan variance of time error   ,3 (t n ) 2  Hadamard variance of time error Cannot use  -measures for accuracy just because accuracy diverges  (  )x(t n )  (  )x(t n +  )  2 (  )x(t n ) M th Order  -Measures:  (  ) M x(t n )  ,M (t n ) 2 = E [  (  ) M (t n )] 2  ,M 2 =  n  n  ,M (t n ) 2 Can be shown to be measures of stability and precision under specific fitting conditions

14. Page 14 EFTF-IFCS 2009 -- V. Reinhardt Neg-p Noise x p (t) x p (t) generally represented as wide-sense stationary (WSS) random processx p (t) generally represented as wide-sense stationary (WSS) random process But x p (t) is inherently non-stationary (NS)But x p (t) is inherently non-stationary (NS) NS picture: x p (t) starts at finite tNS picture: x p (t) starts at finite t WSS picture: x p (t) must start at t = -  to maintain time invarianceWSS picture: x p (t) must start at t = -  to maintain time invariance So WSS x p (t)   for all t since neg-p noise grows without bound as time from start  So WSS x p (t)   for all t since neg-p noise grows without bound as time from start   t x p (t) NS Picture of Neg-p Noise x p (t) = 0 [t<0] 0

15. Page 15 EFTF-IFCS 2009 -- V. Reinhardt Neg-p Noise x p (t) Only NSOnly NS covariance or correlation function ( E x p = 0) is finite    Difference time between covariant arguments  t g  Time from start of process R p (t g,  ) finite for finite t g &  R p (t g,  )   as t g   for all  Because of this the WSS covariance R p (  ) is infinite for all Because of this the WSS covariance R p (  ) is infinite for all  t x p (t) tgtg  Neg-p Noise x p (t) = 0 [t<0]

16. Page 16 EFTF-IFCS 2009 -- V. Reinhardt Neg-p Noise x p (t) In NS f-domain are 3 major spectral functionsIn NS f-domain are 3 major spectral functions Loève Spectrum Wigner-Ville Function Loève Spectrum Ambiguity function Can define L p (f) without using R p (  ) as limit of W p (t g,f)Can define L p (f) without using R p (  ) as limit of W p (t g,f) Fourier Transf

17. Page 17 EFTF-IFCS 2009 -- V. Reinhardt x p (t) Not the Same as x r (t) Because of System Filtering H s (f) x r (t) = h s (t)  x p (t)  X r (f) = H s (f) X p (f)x r (t) = h s (t)  x p (t)  X r (f) = H s (f) X p (f) Important later because H s (f) can have HP filtering as well as LP filtering propertiesImportant later because H s (f) can have HP filtering as well as LP filtering properties Typical HP filtering H s (f) for time or phase errorTypical HP filtering H s (f) for time or phase error Tx Rx ~ x Tx (t) xx vv  d =  x -  v Delay Mismatch H s (f)  f 2 |f|<<1/  d (2 nd Order) PLL H s (f)  f 4 |f|<<1/Loop BW Tx Rx ~ x Tx (t) ~ x Rx (t) PLL vv xx In Asynchronous Timing Systems In Both Synchronous & Asynchronous Timing Systems

18. Page 18 EFTF-IFCS 2009 -- V. Reinhardt What is happening here is that the neg-p noise ensemble member is mimicking the signature of the causal behavior over TWhat is happening here is that the neg-p noise ensemble member is mimicking the signature of the causal behavior over T So part of the noise cannot be separated from the causal behaviorSo part of the noise cannot be separated from the causal behavior f -2 Noise f 0 Noise Kalman Filter f 0 Noise LSQF The Effect of Neg-p Noise on LSQF & Kalman Estimation long term error-1.xls Correlated Noise Model f -2 Noise White Noise Model f -3 Noise x a,M ±  wj,M –x -- x c

19. Page 19 EFTF-IFCS 2009 -- V. Reinhardt Variables that are linearly dependent over T cannot be separated by any solution processVariables that are linearly dependent over T cannot be separated by any solution process  Solution matrix has a zero determinant Adding correlated x r (t) models when x r (t) and x c (t) are correlated with each other will just generate ill-formed equationsAdding correlated x r (t) models when x r (t) and x c (t) are correlated with each other will just generate ill-formed equations f -2 Noise f 0 Noise Kalman Filter f 0 Noise LSQF The Effect of Neg-p Noise on LSQF & Kalman Estimation long term error-1.xls Correlated Noise Model f -2 Noise White Noise Model f -3 Noise x a,M ±  wj,M –x -- x c

20. Page 20 EFTF-IFCS 2009 -- V. Reinhardt E x(t) x (1) (t) x (3) (t) x (2) (t) Ergodicity & Proper Behavior of a Practical Realization of a Fit Fitting theory based on ensemble means EFitting theory based on ensemble means E But practical realizations must use T finite time mean over single ensemble memberBut practical realizations must use T finite time mean over single ensemble member Noise must be ergodic-like over T for practical realization to work as expected  T  ENoise must be ergodic-like over T for practical realization to work as expected  T  E  Strict ergodicity T   = E T T Ergodic- like T  E..

21. Page 21 EFTF-IFCS 2009 -- V. Reinhardt Ergodicity & Proper Behavior of a Practical Realization of a Fit Strong connection between correlation time  c of noise & ergodic-like behaviorStrong connection between correlation time  c of noise & ergodic-like behavior  A white process is locally ergodic T  0 = E  But a correlated process is only intermediate ergodic  E So must have T >>  c for a practical realization to work as expected when a noise process is correlatedSo must have T >>  c for a practical realization to work as expected when a noise process is correlated E x(t)  T T x (1) (t) x (3) (t) x (2) (t) x (n) (t)  c  T E T >>  c

22. Page 22 EFTF-IFCS 2009 -- V. Reinhardt Connection Between T,  c & Anomalous Fitting Behavior But for neg-p noise can show that  c = But for neg-p noise can show that  c =   Unless HP filter H s (f) suppresses neg-p pole Neg-p noise will generate anomalous fit behavior for all TNeg-p noise will generate anomalous fit behavior for all T Unless H s (f) makes  c finite for x r (t)Unless H s (f) makes  c finite for x r (t) T T/  c =2000T/  c =200T/  c =20T/  c =2 Correlated Noise with Finite  c x a,M ±  wj,M –x -- x c

23. Page 23 EFTF-IFCS 2009 -- V. Reinhardt Calculating  d for Neg-p Noise  wj,M 2 =  d  j,M 2  Estimate of  w,M 2 TheoreticallyTheoretically  d =  w,M 2 /  j,M 2 Have shown previously thatHave shown previously that  K ,M (f) = spectral kernel representing fit  Model error adds extra term to  ,M 2  Model error occurs when x a,M (t) is not complex enough to track x c (t) over T Can use above to calculate  d when p-order of noise is knownCan use above to calculate  d when p-order of noise is known  = w or j

24. Page 24 EFTF-IFCS 2009 -- V. Reinhardt Calculating  d for Neg-p Noise  wj,M 2 =  d  j,M 2  Estimate of  w,M 2 For x a,M (t) = (M-1) th order polynomialFor x a,M (t) = (M-1) th order polynomial  K j,M (f)  2M th order highpass filter  K w,M (f)  2M th order lowpass filter Can use above K ,M (f) properties to graphically explain the behavior of  d for neg-p noiseCan use above K ,M (f) properties to graphically explain the behavior of  d for neg-p noise 1 Log 10 (fT) M=5  f 10 M=4  f 8 M=3  f 6 M=2  f 4 M=1  f 2 N=1000 dB K j,M (f) (Uniform LSQF) f T = M/(2T)

25. Page 25 EFTF-IFCS 2009 -- V. Reinhardt Graphing  w,M 2 &  j,M 2 Can show for white noise (uniform LSQF) unless an ideal Nyquist LP H s (f) is usedCan show for white noise (uniform LSQF) unless an ideal Nyquist LP H s (f) is used  Well-known effect of over-sampling for BW  j,M 2 fTfT fhfh 0  w,M 2 L 0 (f) K w,M (f) K j,M (f) White Noise K ,M (f)  Sharp cut-offs at f T H s (f)  LP cutoff f h H s (f)  LP cutoff f h  d = f T /(f h -f T )  M/(N-M)

26. Page 26 EFTF-IFCS 2009 -- V. Reinhardt Graphing  w,M 2 &  j,M 2 For neg-p noiseFor neg-p noise  w,M 2 &  d    While  While  w,M 2 is finite  For all neg-p (when H s (f) has no HP properties) Will later show this is an indication of a real problem with the fit accuracy by using the NS picture of neg-p noiseWill later show this is an indication of a real problem with the fit accuracy by using the NS picture of neg-p noise fTfT fhfh 0  Neg-p Noise L p (f)

27. Page 27 EFTF-IFCS 2009 -- V. Reinhardt Effect of HP filtering H s (f) for Neg-p Noise When H s (f) suppresses neg-p pole at f = 0   d finiteWhen H s (f) suppresses neg-p pole at f = 0   d finite When this is true x r (t) has finite  c is WSS & is intermediate ergodicWhen this is true x r (t) has finite  c is WSS & is intermediate ergodic Practical fit behavior OK for f T >  c  1/(2f l )Practical fit behavior OK for f T >  c  1/(2f l )  d <  do for f T << f l  d >  do for f T >> f l  2 j,M fTfT fhfh f dB  flfl f T >> f l  2 w,M <  L p (f) dB H s (f) suppresses L p (f) pole flfl fhfh fTfT f T << f l f dB  L p (f) dB H s (f)  HP knee f l & LP knee f h

28. Page 28 EFTF-IFCS 2009 -- V. Reinhardt Explaining Physical Reality of Infinite WSS  w,M 2 with NS Picture x r (t 0 ) excluded from x j,M (t n ) because x r (t 0 ) is treated as part of causal behavior by fitx r (t 0 ) excluded from x j,M (t n ) because x r (t 0 ) is treated as part of causal behavior by fit  w,M 2   as t 0   with no indication from  j,M 2  w,M 2   as t 0   with no indication from  j,M 2 Thus the WSS  w,M 2 infinity indicates that the physical  w,M 2 will truly become very large when t 0 >> T (the usual situation)Thus the WSS  w,M 2 infinity indicates that the physical  w,M 2 will truly become very large when t 0 >> T (the usual situation) For example a PLL will cycle slip when  w,M 2   for the calculated loop phase errorFor example a PLL will cycle slip when  w,M 2   for the calculated loop phase error x = x r (x c = 0) x j,M LSQF with x a,1 = a 0 Start of Noise Ensemble Members t0t0 Start of Data T x r (t 0 )

29. Page 29 EFTF-IFCS 2009 -- V. Reinhardt x = x r (x c = 0) x j,M LSQF with x a,1 = a 0 Start of Noise Ensemble Members t0t0 Start of Data T x r (t 0 ) A Simulation Misrepresents Physical Situation when t 0 not >>  c Because steady state not reachedBecause steady state not reached  Note in above that  w,M   j,M when t 0 = 0 while  w,M >>  j,M in physical situation when t 0 >> T Thus all neg-p noise simulations are inherently misleading unless H s (f) makes  c finiteThus all neg-p noise simulations are inherently misleading unless H s (f) makes  c finite T Start of Noise & Data Start of Noise & Data t 0 = 0

30. Page 30 EFTF-IFCS 2009 -- V. Reinhardt  ,M (t) as Stability and Precision Measures When x a,M (t) = (M-1) th Order Polynomial Removing “causal” behavior from data biases  ,M (t) as a true random instability measureRemoving “causal” behavior from data biases  ,M (t) as a true random instability measure   ,M (t) generated from x r (t) with no fit not the same as  ,M (t) generated from x(t) with removal of x a,M (t)  Fit removes noise correlated with causal behavior  Reduces true noise contribution for   T  j,M =  ,M (t 0 ) for “Unbiased”  n x j,M (t m ) x a,M (t,a) Unweighted LSQF over all t 0 …t M       x(t m )   = T/M Stability over  = T/M Precision x(t M ) x j,M (t M ) x a,M (t) Passes thru t 0 …t M-1       x(t 0 ) º ExtrapolationInterpolation  j,M (t 0 )   ,M (t 0 ) x j,M (t m ) x k,M (t,a) Passes thru t 0 …t M but not t m      x(t m ) º

31. Page 31 EFTF-IFCS 2009 -- V. Reinhardt Secondary Accuracy after Cal- ibration Can Sidestep Divergence x(t)  x(t,  (t))x(t)  x(t,  (t))   (t) = other independent variables Cal x(t) periodically at t n ’ against primary standard when  (t n ’) =  oCal x(t) periodically at t n ’ against primary standard when  (t n ’) =  o  To generate cal function x a,M’ (k) (t) Now use secondary (calibrated) data x (k) (t) = x(t) – x a,M’ (k) (t)Now use secondary (calibrated) data x (k) (t) = x(t) – x a,M’ (k) (t)  To determine  sensitivity coefficients  & sidestep x r (t) divergence for  -coefficient determination But divergence still present for determining true causal behavior free of noiseBut divergence still present for determining true causal behavior free of noise Cal x (k) (t) after Cal Start of Noise x (k) (t) = x(t) – x a,M’ (k) (t)  (t n’ ) =  o

32. Page 32 EFTF-IFCS 2009 -- V. Reinhardt Conclusions & Further Implications Cannot separate neg-p noise from true causal behavior for any T  Exception for HP H s (f)Cannot separate neg-p noise from true causal behavior for any T  Exception for HP H s (f)  Fits generate anomalous results for any T unless sufficiently HP filtered by H s (f) Unbiased or pure neg-p behavior unobservable in data containing causal behaviorUnbiased or pure neg-p behavior unobservable in data containing causal behavior  x(t) with x a,M (t) removed not equivalent to x r (t)  Can explain downward biases in Allan & Hadamard variances when   T

33. Page 33 EFTF-IFCS 2009 -- V. Reinhardt Conclusions & Further Implications Noise whitening is an improper procedure to use when neg-p noise presentNoise whitening is an improper procedure to use when neg-p noise present  Fits to correlated noise as well as x c (t) Must have T >>  c for fitting technique to behave as theoretically expectedMust have T >>  c for fitting technique to behave as theoretically expected  Problematic for neg-p noise  unless HP H s (f)

34. Page 34 EFTF-IFCS 2009 -- V. Reinhardt Conclusions & Further Implications Also effects M-corner hat & cross-correlation techniquesAlso effects M-corner hat & cross-correlation techniques  Assume statistical independence implies T cross-correlations  0 as N -1/2 When there is no HP filtering H s (f) expect such anomalies for all T & all neg-p ordersWhen there is no HP filtering H s (f) expect such anomalies for all T & all neg-p orders For cross-correlation with 1 st order PLL H s (f)For cross-correlation with 1 st order PLL H s (f)  Expect anomalies for f -3 noise that grow as slowly as log(T) The above needs further studyThe above needs further study http://www.ttcla.org/vsreinhardt/