Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 1 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Statistical Processes for Time and Frequency A Tutorial Victor S. Reinhardt 10/17/01

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 2 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Statistical Processes for Time and Frequency--Agenda Review of random variables Random processes Linear systems Random walk and flicker noise Oscillator noise

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 3 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Review of Random Variables

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 4 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Continuous Random Variable Random Variable x –Repeat N identical experiments = Ensemble of experiments –Unpredictable (Variable) Result x n N x = Number of of times value x n between x and x+dx Probability density function (PDF) or distribution p(x) Ensemble of N Identical Experiments Unpredictable Result x1x1 x2x2 x3x3 xNxN x Number of Occurrences NxNx xx+dx N x+dx N x-dx

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 5 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. PDF and Expectation Values Range of random variable x from a to b Mean value =  [x] Standard variance =  d 2 [x] Standard deviation =  d [x] x ab   The expectation value of f(x) is the average of f(x) over the ensemble defined by p(x)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 6 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Probability Distributions Gaussian (Normal) PDF –Range = (- , +  )– Mean =  –Standard deviation =  d Uniform –Range = (-D/2, +D/2)– Mean = 0 –Standard deviation = D/ –Examples: Quantization error, totally random phase error P gauss (x) x x -D/2 D/20 P uniform (x)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 7 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Statistics A statistic is an estimate of a parameter like  or  Repeat experiment N times to get x 1, x 2, …… x N Statistic for mean  [x] is arithmetic mean Statistics for standard variance  d [x] –Standard Variance (  known a priori) –Standard Variance (with estimate of  ) Good Statistics –Converge to the parameter as N   with zero error –Expectation value = parameter value for any N (Unbaised)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 8 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Multiple Random Variables x 1 and x 2 two random variables (1 and 2 not ensemble indices but indicate different random variables) –Joint PDF = p (2) (x 1,x 2 ) (2) means 2-variable probability –Expectation value –Single Variable PDF –Conditional PDF = p(x 1 |x 2 ) is PDF of x 2 occuring given that x 1 occurred Mean & Covariance matrix Statistical Independence –p (2) (x 1,x 2 ) = p (1) (x 1 )p’ (1) (x 2 ) –Then (k & k’ = 1,2)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 9 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Ensembles Revisited The ensemble for x is a set of statistically independent random variables x 1, x 2, ….. x N with all PDFs the same = p (1) (x) Thus Ensemble of N Identical Experiments x1x1 x2x2 x3x3 xNxN Each with same PDF p(x) Each statistically independent (F=2 for normal distribution)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 10 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Random Processes

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 11 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Random Processes A random function in time u(t) –Is a random ensemble of functions –That is defined by a hierarchy probability density functions (PDF) –p (1) (u,t) = 1st order PDF –p (2) (u 1,t 1 ; u 2,t 2 ) = 2nd order joint PDF –etc One can ensemble average at fixed times Or time average n th member Ensemble Average E[...] u 1 (t) t u 2 (t) u N (t) Time Average

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 12 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Time mean Autocorrelation function Wide sense stationarity Strict Stationarity –All PDFs invariant under t n  t n - t’ Ergodic process –Time and ensemble averages equivalent Time Averages and Stationarity (= 0 for random processes we will consider) Ensemble Average u 1 (t) t u 2 (t) u N (t) Time Average A Stationary Non-ergodic Process Ergodic_Theorem: Stationary processes are ergodic only if there are no stationary subsets of the ensemble with nonzero probability Op Amp Offset Voltage

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 13 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Types of Random Processes Strict Stationarity: All PDFs invariant under time translation (no absolute time reference) –Invariant under t n  t n - t’ (all n and any t’) –Implies p (1) (x,t) = p (1) (x) = independent of time p (2) (x 1,t 1 ; x 2,t 2 ) = p (2) (x 1,0; x 2,t 2 - t 1 ) = function of t 2 - t 1 Purely random process: Statistical independence –p (n) (x 1,t 1 ; x 2,t 2 ; ….x n,t n ) = p (1) (x 1,t 1 ) p (1) (x 2,t 2 ) ….. p (1) (x n,t n ) Markoff Process: Highest structure is 2nd order PDF –p(x 1,t 1 ;...x n-1,t n-1 | x n,t n ) = p(x n-1,t n-1 | x n,t n ) –p(x 1,t 1 ;...x n-1,t n-1 | x n,t n ) is conditional PDF for x n (t n ) given that x 1 (t 1 ) ;...x n-1,t n-1 have occurred –p (n) (x 1,t 1 ; x 2,t 2 ; ….x n,t n ) = p (1) (x 1,t 1 ) p(x k-1,t k-1 | x k,t k )

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 14 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Linear Systems

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 15 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Linear Systems In time domain given by convolution with response function h(t) Fourier transform to frequency domain The fourier transform of the output is u(t) Linear system h(t), H(f) U(f) v(t) V(f) Frequency Domain Time Domain

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 16 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. U(f) V(f) R1R1 R2R2 C - +   = R 2 C G=-1 Single-Pole Low Pass Filter   = R 1 C (3-dB bandwidth) log(f/B 3 ) (Causal filter)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 17 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Spectral Density of a Random Process Requires wide sense stationary process The spectral density is the fourier transform of the autocorrelation function For linearly related variables given by The spectral densities have a simple relationship S v (f) =  2 S u (f) S u (f) = S v (f) 1 22 v(t) = du dt U(f) System H(f) V(f) = H(f)V(f) S v (f) = |H(f)| 2 S u (f) Important Property of S(f) V(f) = j  U(f)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 18 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Average Power and Variance Autocorrelation Function back from Spectral Density Average power (intensity) Average power in terms of input For ergodic processes Where  d 2 the standard variance is (Mean is assumed zero)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 19 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. White Noise Uncorrelated (zero mean) process Generates white spectrum At output B n is noise bandwidth of system H(f) for Ideal Bandpass Filter 0 f 1 BB fofo -f o For Single-Pole LP Filter B n  B 3-dB as number of poles increases For Thermal (Nyquist) Noise N o = kT

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 20 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. White noise filtered by single pole filter –  1 =  2 =  o –Called Gauss-Markoff Process for gaussian noise Frequency Domain Time Domain Correlation Time =  o –Correlation width =  t = 2  o White Noise Filter Band Limited Noise Band-Limited White Noise & Correlation Time B3B3 N o /2 0 tt

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 21 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Model of Spectrum Analyzer –Downconverts signal to baseband –Resolution Filter: BW = B r –Detector –Video Filter averages for T = 1/(2B v ) Spectrum Analyzer Measures Periodogram (B r  0) –u T (t) = Truncated data from t to t+T –Fourier Trans Wiener-Khinchine Theorem –When T   –Periodogram  Spectral Density Spectrum Analyzers and Spectral Density tt tt tt tt tt Averaging Time T tt T/  t Independent Samples tt tt  t = Correlation Width = 1/(2B r ) Radiometer Formula (finite B r ) Same as Model of Specrum Analyzer Res Filter B r X In Det Video Filter B v Out

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 22 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Response Function and Standard Variance for Time Averaged Signals Finite time average over  Response Fn for average Variance of with H 1 For So   2 diverges when |H 1 (f)| 2 S y (f)  0 1/  h 1 (t’) t+  t Response Function ( for non-stationary noise) y = (f-f o )/f o v(t) = t, 

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 23 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Response Function for Zero Dead- Time Sample (Allan) Variance Response for difference of time averaged signals Variance with H 2 (Allan variance) For So   2 doesn’t diverge for 1/  h 2 (t’) -1/  t+  t t+2  |H 2 (f)| 2  f 2 0 S y (f)  1/f 2 Response Function v(t) = t+T,  - t,  ( for noise up to random run) y = (f-f o )/f o

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 24 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Graphing to Understand System Errors Can represent system error as h(t) includes –Response for measurement –Plus rest of system Graphing h(t) or H(f) helps understanding Example: Frequency error for satellite ranging –Ranging:  d 2 ( ,T) =  2 2 (T,  ) = Allan variance with dead time  and averaging time T reversed –Radar:  d 2 ( ,T) =  2 2 ( ,T) = no resversal of T and  ~ Satellite y(t) X Round Trip Time T Average  y for Time  y(t-T) Meas Error  Satellite Ranging y =  f/f yy 1/  -1/  t   t+T t+  t+T+  v(t) = t+T,  - t,  T (  > T) h(t)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 25 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Random Walk and Flicker Noise

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 26 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Integrated White Noise--Random Walk (Wiener Process) Let u(t) be white noise And Then –where t < = the smaller of t or t’ Note R v is not stationary (not function of t-t’) –This is a classic random walk with a start at t=0 –The standard deviation is a function of t Random Walk Increases as t ½ ±  1 (t) v(t) t

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 27 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. A filter described by h(t-t’) is called a Wiener filter –Must know properties of filter for all past times To generate (stationary) colored noise can Wiener filter white noise –Can turn convolution into differential (difference) equation (Kalman filter) for simulations White Noise u(t) Wiener Filter h(t-t’) Colored Noise v(t) Generating Colored Noise from White Noise |H(f)| 2 S v (f) Colored Noise Wiener Filter White Noise

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 28 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Wiener Filter for Random Walk U(f) R1R1 R2R2 C - +   = R 1 C  = R 1 /R 2 HH hh V(f) G=-1

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 29 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Wiener Filter for Flicker Noise Impedance of diffusive line White current noise generates flicker voltage noise –N i = Current noise density R C R C Z Heavyside Model of Diffusive Line R C ZZ Impedance Analysis White Current Noise Flicker Voltage Noise R C R C v(t) i(t)

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 30 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Multiplicative Flicker of Phase Noise Nonlinearities in RF amplifier produce AM/PM Low frequency amplitude flicker processes modulates phase around carrier through AM/PM Modulation noise or multiplicative noise is what appears around every carrier SvSv f 0 SS f fofo AM/PM converts low frequency amplitude fluctuations into phase fluctuations about carrier

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 31 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Single-Pole Filters T. C. =  Independent current sources Integrate outputs over  An Alternative Wiener Filter for Flicker Noise S I (f)=I 2 V(f) R C - + G=-1  = RC =  -1 N Independent White Current Noise Sources Filter  1  0 Filter  2 Filter  N  Sum (Integrate) Over Outputs Flicker Noise White Current Source I(f) R = Constant

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 32 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. A Practical Wiener Filter for Flicker Noise Single-pole every decade With independent white noise inputs Spectrum For time domain simulation turn convolutions into difference equations for each filter and sum Error in dB from 1/f Results for m = 0 to 8 S f (f) S (f) vmvm

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 33 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Oscillator Noise

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 34 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Properties of a Resonator High frequency approximation (single pole)  f = 3-dB full width Phase shift near f o d  /dy = 2Q ff |Y R | 2

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 35 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Simple Model of an Oscillator Amplifier and resonator in positive feedback loop Amplifier –Amp phase noise S  amp (f) = FkT/P in (1+ f f /f) –Thermal noise + flicker noise Resonator (Near Resonance)  R = -2Q L y [ y = (f - f o )/f o ] Oscillation Conditions –Loop Gain = |G a  L |  1 –Phase shift around loop = 0  R +  amp = 0 Gain = G a Phase Shift =  amp Noise Figure = F Flicker Knee = f f White Noise Density = FkT Resonator Amp Oscillation Conditions |G a  R | = Loop Gain  1   Around Loop = 0 P in Near Resonance  R = -2Q L y Loss =  R = Y R Loaded Q = Q L Noise G a,  a Thermal Flicker of Phase

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 36 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Phase Shift Around Loop = 0  amp = 2Q L y = -  R –Thus the oscillator fractional frequency y must change in response to amplifier phase disturbances  amp Amp Phase Noise is Converted to Oscillator Frequency Noise S y-osc (f) = 1/(2Q L ) 2 S  -amp (f) But y =  o -1 d  /t so S  -osc (f) = (f o 2 /f 2 ) S y-osc (f) And thus we obtain Leeson’s Equation S  -osc (f) = ((f o /(2Q L f)) 2 +1)(FkT/P in )(1+ f f /f) Leeson’s Equation The Oscillator  f/f must shift to compensate for the amp phase disturbances Converted NoiseOriginal Amp Noise Resonator Phase vs  f/f Response y  R = -  a

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 37 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. Oscillator Noise Spectrum Oscillator noise Spectrum S  (f) = K 3 /f 3 + K 2 /f 2 + K 1 /f + K 0 –Some components may mask others Converted noise –K 2 = FkT/P in (f o /(2Q L f)) 2 –K 3 = FkT/P in (f f /f) (f o /(2Q L f)) 2 –Varies with (f o /(2Q L ) 2 and FkT/P in Original amp noise –K o = FkT/P in –K 1 = FkT/P in (f f /f) –Only function of FkT/P in –and flicker knee S  -osc (f) = (f o /(2Q L f)) 2 +1)(FkT/P in )(1+ f f /f) Oscillator Noise Spectrum Leeson’s Equation S  (f) f K 3 /f 3 K 2 /f 2 K 1 /f K0K0 QLQL Converted Noise Amp Noise

Statisitcs Tutorial V. S. Reinhardt 10/17/01 Page 38 Space Systems Copyright 2005 Victor S. Reinhardt--Rights to copy material is granted so long as a source reference is listed on each page, section, or graphic utilized. References R. G. Brown, Introduction to Random Signal Analysis and Kalman Filtering, Wiley, D. Middleton, An Introduction to Statistical Communication Theory, McGraw-Hill, W. B, Davenport, Jr. and W. L. Root, An Introduction to the Theory of Random Signals and Noise, Mc-Graw-Hill, A. Van der Ziel, Noise Sources, Characterization, Measurement, Prentice-Hall, D. B. Sullivan, D. W. Allan, D. A. Howe, F. L. Walls, Eds, Characterization of Clocks and Oscillators, NIST Technical Note 1337, U. S. Govt. Printing office, 1990 (CODEN:NTNOEF). B. E. Blair, Ed, Time and Frequency Fundamentals, NBS Monograph 140, U. S. Govt. Printing office, 1974 (CODEN:NBSMA6). D. B. Leeson, “A Simple Model of Feedback Oscillator Noise Spectrum,” Proc, IEEE, v54, Feb., 1966, p