Lecture 7 Linear time invariant systems Stochastic processes Lecture 7 Linear time invariant systems

Random process

1st order Distribution & density function First-order distribution First-order density function

2end order Distribution & density function 2end order distribution 2end order density function

EXPECTATIONS Expected value The autocorrelation

Some random processes Single pulse Multiple pulses Periodic Random Processes The Gaussian Process The Poisson Process Bernoulli and Binomial Processes The Random Walk Wiener Processes The Markov Process

Recap: Power spectrum density

Power spectrum density Since the integral of the squared absolute Fourier transform contains the full power of the signal it is a density function. So the power spectral density of a random process is: Due to absolute factor the PSD is always real 𝑆𝑥𝑥 𝑓 = 𝑙𝑖𝑚 𝑇→∞ ⁡𝐸 −𝑇 𝑇 𝑠 𝑡 𝑒 −𝑗2𝜋𝑓𝑡 𝑑𝑡 2 2𝑇

Power spectrum density The PSD is a density function. In the case of the random process the PSD is the density function of the random process and not necessarily the frequency spectrum of a single realization. Example A random process is defined as Where ωr is a unifom distributed random variable wiht a range from 0-π What is the PSD for the process and The power sepctrum for a single realization X 𝑡 =sin⁡( 𝜔 𝑟 𝑡)

Properties of the PSD Sxx(f) is real and nonnegative The average power in X(t) is given by: 𝐸 𝑋 2 (𝑡) =𝑅𝑥𝑥 0 = −∞ ∞ 𝑆𝑥𝑥 𝑓 𝑑𝑓 If X(t) is real Rxx(τ) and Sxx(f) are also even 𝑆𝑥𝑥 −𝑓 =𝑆𝑥𝑥 𝑓 If X(t) has periodic components Sxx(f)has impulses Independent on phase

Wiener-Khinchin 1 If the X(t) is stationary in the wide-sense the PSD is the Fourier transform of the Autocorrelation

Wiener-Khinchin Two method for estimation of the PSD Fourier Transform |X(f)|2 X(t) Sxx(f) Fourier Transform Autocorrelation

The inverse Fourier Transform of the PSD Since the PSD is the Fourier transformed autocorrelation The inverse Fourier transform of the PSD is the autocorrelation

Cross spectral densities If X(t) and Y(t) are two jointly wide-sense stationary processes, is the Cross spectral densities Or

Properties of Cross spectral densities Since is Syx(f) is not necessary real If X(t) and Y(t) are orthogonal Sxy(f)=0 If X(t) and Y(t) are independent Sxy(f)=E[X(t)] E[Y(t)] δ(f)

Cross spectral densities example 1 Hz Sinus curves in white noise Where w(t) is Gaussian noise 𝑋 𝑡 = sin 2𝜋 𝑡 +3 𝑤(𝑡) 𝑌 𝑡 = sin 2𝜋 𝑡+ 𝜋 2 +3 𝑤(𝑡)

The periodogram The estimate of the PSD The PSD can be estimate from the autocorrelation Or directly from the signal 𝑆𝑥𝑥 ω = 𝑚=−𝑁+1 𝑁−1 𝑅𝑥𝑥 [𝑚] 𝑒 −𝑗ω𝑚 𝑆𝑥𝑥 ω = 1 𝑁 𝑛=0 𝑁−1 𝑥 [𝑛] 𝑒 −𝑗ω𝑛 2

Bias in the estimates of the autocorrelation 𝑅𝑥𝑥 𝑚 = 𝑛=0 𝑁− 𝑚 −1 𝑥 𝑛 𝑥[𝑛+𝑚]

Variance in the PSD The variance of the periodogram is estimated to the power of two of PSD 𝑉𝑎𝑟 𝑆𝑥𝑥 𝜔 = 𝑆𝑥𝑥(𝜔) 2

Averaging Divide the signal into K segments of M length 𝑥𝑖=𝑥 𝑖−1 𝑀+1:𝑖 𝑀 1≤𝑖≤𝐾 Calculate the periodogram of each segment 𝑆𝑖𝑥𝑥 ω = 1 𝑀 𝑛=0 𝑀−1 𝑥 𝑖[𝑛] 𝑒 −𝑗ω𝑛 2 Calculate the average periodogram 𝑆 𝑥𝑥[ω]= 1 𝐾 𝑖=0 𝐾 𝑆𝑖𝑥𝑥[ω]

Illustrations of Averaging

PSD units Typical units: Electrical measurements: V2/Hz or dB V/Hz Sound: Pa2/Hz or dB/Hz How to calculate dB I a power spectrum: PSDdB(f) = 10 log10 { PSD(f)  } .

Agenda (Lec. 7) Recap: Linear time invariant systems Stochastic signals and LTI systems Mean Value function Mean square value Cross correlation function between input and output Autocorrelation function and spectrum output Filter examples Intro to system identification

Focus continuous signals and system Continuous signal: Discrete signal:

Systems

Recap: Linear time invariant systems (LTI) What is a Linear system: The system applies to superposition 1 2 3 4 5 6 8 10 12 14 16 18 20 Linear system x(t) y(t) 1 2 3 4 5 -20 -15 -10 -5 10 15 20 25 Nonlinear systems x(t) y(t) x[n] Ö 20 log(x[n])

Recap: Linear time invariant systems (LTI) A time invariant systems is independent on explicit time (The coefficient are independent on time) That means If: y2(t)=f[x1(t)] Then: y2(t+t0)=f[x1(t+t0)] The same to Day tomorrow and in 1000 years 70 years 45 years 20 years A non Time invariant

Examples A linear system A nonlinear system A time invariant system y(t)=3 x(t) A nonlinear system y(t)=3 x(t)2 A time invariant system A time variant system y(t)=3t x(t)

The impulse response The output of a system if Dirac delta is input

Convolution The output of LTI system can be determined by the convoluting the input with the impulse response

Fourier transform of the impulse response The Transfer function (System function) is the Fourier transformed impulse response The impulse response can be determined from the Transfer function with the invers Fourier transform

Fourier transform of LTI systems Convolution corresponds to multiplication in the frequency domain Time domain * = Frequency domain x =

Causal systems Independent on the future signal

Stochastic signals and LTI systems Estimation of the output from a LTI system when the input is a stochastic process Α is a delay factor like τ

Statistical estimates of output The specific distribution function fX(x,t) is difficult to estimate. Therefor we stick to Mean Autocorrelation PSD Mean square value.

Expected Value of Y(t) (1/2) How do we estimate the mean of the output? 𝐸 𝑌 𝑡 =𝐸 −∞ ∞ 𝑋 𝑡−𝛼 ℎ 𝛼 𝑑𝛼 𝑌(𝑡)= −∞ ∞ 𝑋 𝑡−𝛼 ℎ 𝛼 𝑑𝛼 𝐸 𝑌 𝑡 = −∞ ∞ 𝐸 𝑋 𝑡−𝛼 ℎ 𝛼 𝑑𝛼 If mean of x(t) is defined as mx(t) 𝐸 𝑌 𝑡 = −∞ ∞ 𝑚𝑥(𝑡−𝛼)ℎ 𝛼 𝑑𝛼

Expected Value of Y(t) (2/2) If x(t) is wide sense stationary 𝑚𝑥 𝑡−𝛼 =𝑚𝑥 𝑡 =𝑚𝑥 (𝑚𝑥 𝑖𝑠 𝑎 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡) 𝑚𝑦=𝐸 𝑌 𝑡 = −∞ ∞ 𝑚𝑥(𝑡−𝛼)ℎ 𝛼 𝑑𝛼 −∞ ∞ 𝑚𝑥ℎ 𝛼 𝑑𝛼 𝑚𝑦=𝐸 𝑌 𝑡 =𝑚𝑥 −∞ ∞ (𝑡−𝛼)ℎ 𝛼 𝑑𝛼 Alternative estimate: At 0 Hz the transfer function is equal to the DC gain −∞ ∞ ℎ 𝛼 𝑑𝛼=𝐻(0) Therefor: 𝑚𝑦=𝐸 𝑌 𝑡 =𝑚𝑥 𝐻(0)

Expected Mean square value (1/2) 𝑌(𝑡)= −∞ ∞ 𝑋 𝑡−𝛼 ℎ 𝛼 𝑑𝛼 𝐸 𝑌 𝑡 2 =𝐸 𝑌 𝑡 𝑌 𝑡 𝐸 𝑌 𝑡 2 =𝐸 −∞ ∞ 𝑋 𝑡−𝛼1 ℎ 𝛼1 𝑑𝛼1 −∞ ∞ 𝑋 𝑡−𝛼2 ℎ 𝛼2 𝑑𝛼2 𝐸 𝑌 𝑡 2 =𝐸 −∞ ∞ −∞ ∞ 𝑋 𝑡−𝛼1 𝑋 𝑡−𝛼2 ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2 𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝐸 𝑋 𝑡−𝛼1 𝑋 𝑡−𝛼2 ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2 𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝑡−𝛼1,𝑡−𝛼2) ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2 𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝛼1,𝛼2) ℎ 𝑡−𝛼1 ℎ 𝑡−𝛼2 𝑑𝛼1𝑑𝛼2

Expected Mean square value (2/2) 𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝛼1,𝛼2) ℎ 𝑡−𝛼1 ℎ 𝑡−𝛼2 𝑑𝛼1𝑑𝛼2 𝛼=𝑡−𝛼1 𝛽=𝑡−𝛼2 By substitution: 𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝑡−𝛼,𝑡−𝛽)ℎ 𝛼 ℎ 𝛽 𝑑𝛼1𝑑𝛼2 If X(t)is WSS 𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝛼−𝛽) ℎ 𝛼 ℎ 𝛽 𝑑𝛼1𝑑𝛼2 Thereby the Expected Mean square value is independent on time

Cross correlation function between input and output Can we estimate the Cross correlation between input and out if X(t) is wide sense stationary 𝑅𝑦𝑥 𝑡+𝜏,𝑡 =𝐸 𝑌 𝑡+𝜏 𝑋∗(𝑡) 𝑅𝑦𝑥 𝑡+𝜏,𝑡 =𝐸 −∞ ∞ 𝑋 𝑡−𝛼+𝜏 ℎ 𝛼 𝑑𝛼 𝑋 ∗ (𝑡) 𝑅𝑦𝑥 𝑡+𝜏,𝑡 =𝐸 −∞ ∞ 𝑋 𝑡−𝛼+𝜏 𝑋 ∗ (𝑡)ℎ 𝛼 𝑑𝛼 𝑅𝑥𝑥 𝜏 =𝐸 𝑋 𝑡+𝜏 𝑋 (𝑡) 𝑅𝑦𝑥 𝜏 = −∞ ∞ 𝑅𝑥𝑥 𝜏−𝛼 ℎ 𝛼 𝑑𝛼=𝑅𝑥𝑥 𝜏 ∗ℎ(𝜏) Thereby the cross-correlation is the convolution between the auto-correlation of x(t) and the impulse response

Autocorrelation of the output (1/2) 𝑅𝑦𝑦 𝜏 =𝑅𝑦𝑦 𝑡+𝜏,𝑡 =𝐸 𝑌 𝑡+𝜏 𝑌(𝑡) Y(t) and Y(t+τ) is : 𝑌(𝑡+𝜏)= −∞ ∞ 𝑋 𝑡+𝜏−𝛼 ℎ 𝛼 𝑑𝛼 𝑌(𝑡)= −∞ ∞ 𝑋 𝑡−𝛽 ℎ 𝛽 𝑑𝛽 𝑅𝑦𝑦 𝜏 =𝐸 −∞ ∞ 𝑋 𝑡+𝜏−𝛼 ℎ 𝛼 𝑑𝛼 −∞ ∞ 𝑋 𝑡−𝛽 ℎ 𝛽 𝑑𝛽 𝑅𝑦𝑦 𝜏 =𝐸 −∞ ∞ −∞ ∞ 𝑋 𝑡+𝜏−𝛼 𝑋 𝑡−𝛽 ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽 𝑅𝑦𝑦 𝜏 = −∞ ∞ −∞ ∞ 𝐸[𝑋 𝑡+𝜏−𝛼 𝑋 𝑡−𝛽 ]ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽 𝑅𝑦𝑦 𝜏 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝜏−𝛼+𝛽)ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽

Autocorrelation of the output (2/2) 𝑅𝑦𝑦 𝜏 = −∞ ∞ −∞ ∞ 𝐸[𝑋 𝑡+𝜏−𝛼 𝑋 𝑡−𝛽 ]ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽 By substitution: α=-β 𝑅𝑦𝑦 𝜏 = −∞ ∞ −∞ ∞ 𝐸[𝑋 𝑡+𝜏−𝛼 𝑋 𝑡+𝛼 ]ℎ 𝛼 ℎ −𝑎 𝑑𝛼𝑑𝛼 Remember: 𝑅𝑦𝑥 𝜏 =𝑅𝑥𝑥 𝜏 ∗ℎ 𝜏 = −∞ ∞ 𝑅𝑥𝑥 𝜏−𝛼 ℎ 𝛼 𝑑𝛼 𝑅𝑦𝑦 𝜏 =𝑅𝑦𝑥 𝜏 ∗ℎ(−𝜏) 𝑅𝑦𝑦 𝜏 =𝑅𝑥𝑥 𝜏 ∗ℎ(𝜏)∗ℎ(−𝜏)

Spectrum of output Given: The power spectrum is 𝑅𝑦𝑦 𝜏 =𝑅𝑥𝑥 𝜏 ∗ℎ 𝜏 ∗ℎ(−𝜏) |𝐻 𝑓 | 2 =𝐻 𝑓 𝐻 ∗ (𝑓) 𝑆𝑦𝑦 𝑓 =𝑆𝑥𝑥 𝑓 𝐻 𝑓 𝐻 ∗ (𝑓) 𝑆𝑦𝑦 𝑓 =𝑆𝑥𝑥 𝑓 |𝐻 𝑓 | 2 x =

Filter examples

Typical LIT filters FIR filters (Finite impulse response) IIR filters (Infinite impulse response) Butterworth Chebyshev Elliptic

Ideal filters Highpass filter Band stop filter Bandpassfilter

Filter types and rippels

Analog lowpass Butterworth filter Is ”all pole” filter Squared frequency transfer function N:filter order fc: 3dB cut off frequency Estimate PSD from filter

Chebyshev filter type I Transfer function Where ε is relateret to ripples in the pass band Where TN is a N order polynomium

Transformation of a low pass filter to other types (the s-domain) Filter type Transformation New Cutoff frequency Lowpas>Lowpas Lowpas>Highpas Lowpas>Stopband Old Cutoff frequency Lowest Cutoff frequency New Cutoff frequency Highest Cutoff frequency

Discrete time implantation of filters A discrete filter its Transfer function in the z-domain or Fourier domain Where bk and ak is the filter coefficients In the time domain:

Filtering of a Gaussian process X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t and n values Filtering of a Gaussian process Where w[n] are independent zero mean Gaussian random variables.

The Gaussian Process X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t and n values Example: randn() in Matlab

The Gaussian Process and a linear time invariant systems Out put = convolution between input and impulse response Gaussian input Gaussian output

Example x(t): h(t): Low pass filter y(t):

Filtering of a Gaussian process example 2 Band pass filter

Intro to system identification Modeling of signals using linear Gaussian models: Example: AR models The output is modeled by a linear combination of previous samples plus Gaussian noise.

Modeling example Estimated 3th order model

Agenda (Lec. 7) Recap: Linear time invariant systems Stochastic signals and LTI systems Mean Value function Mean square value Cross correlation function between input and output Autocorrelation function and spectrum output Filter examples Intro to system identification