Introduction to Fourier Processing A spectrum of possibilities…

9/4/2015Copyright, G. A. Tagliarini, PhD 2 Organization Signals: analog and discrete The Fourier Transform – Definition – Discretization – Nyquist limits – Computation via the FFT Applications – Correlation – Convolution – Filtering Two-dimensional FFTs

9/4/2015Copyright, G. A. Tagliarini, PhD 3 A Signal

9/4/2015Copyright, G. A. Tagliarini, PhD 4 Given: An analog signal x a (t), t ≥ 0 The signal is sampled at uniform intervals of length T to produce a discrete time signal x(k) ≡ x a (kT), k = 0, 1, 2,… The signal x(k) is said to have a sampling frequency f s = 1/T The sampled signal was periodic

9/4/2015Copyright, G. A. Tagliarini, PhD 5 The Fourier Transform (FT)

9/4/2015Copyright, G. A. Tagliarini, PhD 6 The Inverse Fourier Transform (FT -1 )

9/4/2015Copyright, G. A. Tagliarini, PhD 7 Observations The Fourier transform maps real-valued functions of t (time) into complex-valued functions of f (frequency) One can reconstruct a time signal by using FT -1 Since we will work with discrete samples from signals, our digital signal processing (DSP) will typically not require analytical solutions of integral equations!!!

9/4/2015Copyright, G. A. Tagliarini, PhD 8 The Energy Density Spectrum of x(t) The energy density spectrum, a.k.a. power spectrum, of a signal x(t) is given by |X(f)| 2 If x(t) is real, X(-f) = X * (f), the complex conjugate of X(f) If 0≤f 0 ≤f 1, the energy in the band from f 0 through f 1 is given by:

9/4/2015Copyright, G. A. Tagliarini, PhD 9 Example 1: Energy Density Spectrum See example for FFT and PSD

9/4/2015Copyright, G. A. Tagliarini, PhD 10 Bandwidth Limited Signals If  a value B>0, such that X(f) = 0 for all values of f>B, then x(t) is said to be bandwidth limited Bandwidth limited signals occur often – Bio-mechanical properties limit the highest frequency signals that animals can generate naturally: Vocalizations in humans (40 Hz – 20 KHz) Vocalizations in dolphins and whales (up to 100 KHz) – Electrical and physical characteristics of components also constrain frequency ranges in electrical circuits Common analog telephone circuits ( ,000 Hz) FM radio band width ( KHz)

9/4/2015Copyright, G. A. Tagliarini, PhD 11 Nyquist Sampling Recall that the sampling frequency, f s, is the reciprocal of the sampling interval T, i.e., f s = 1/T Suppose a signal x(t) is bandwidth limited to B Hz By the sampling theorem, if f s ≥ 2 B, the signal can be approximated (interpolated) between samples. The Nyquist critical frequency is f cN = 2 B The critical sampling interval T c = 1/(2B)

9/4/2015Copyright, G. A. Tagliarini, PhD 12 Nyquist Sampling: Interpretation Given a signal x(t) with bandwidth limit B: – The Nyquist critical frequency is f cN = 2 B – The critical sampling interval T c = 1/(2B) Meaning – To reconstruct x(t) without distortion x(t), the sampling rate must be at least twice the highest frequency present – The maximum sampling interval is one half the wavelength of the highest frequency component The sampling process must obtain at least two samples per cycle of the highest frequency present!

9/4/2015Copyright, G. A. Tagliarini, PhD 13 Interpolating Between Samples Assume – x(t) is bandwidth limited by B – The sampling frequency f s ≥ 2B, where T = 1/ f s Then, by the sampling theorem we can interpolate between samples using:

9/4/2015Copyright, G. A. Tagliarini, PhD 14 One More Useful Notion Recall a definition of the definite integral

9/4/2015Copyright, G. A. Tagliarini, PhD 15 The Discrete Fourier Transform Given the definition of the Fourier Transform The discrete Fourier transform (DFT) is given by:

9/4/2015Copyright, G. A. Tagliarini, PhD 16 The Discrete Fourier Transform The discrete Fourier transform (DFT): approximates the Fourier transform at the N frequency values determined by m, 0≤m<N

9/4/2015Copyright, G. A. Tagliarini, PhD 17 The Inverse of the Discrete Fourier Transform Recall the inverse Fourier transform is: The inverse of the DFT is given by:

9/4/2015Copyright, G. A. Tagliarini, PhD 18 Example 2: A Two-sample DFT Suppose – N = 2 – Samples x 0, x 1 are given corresponding to x a (kT), for T = 0, 1 Find X(m)=?

9/4/2015Copyright, G. A. Tagliarini, PhD 19 Example 2: A Two-sample DFT (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 20 Example 2: A Two-sample DFT (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 21 Example 3: Four Sample DFT Suppose – N = 4 – Samples x 0, x 1, … x 3 are given corresponding to x a (kT), for k = 0,…,3 Find X(m)=?

9/4/2015Copyright, G. A. Tagliarini, PhD 22 Example 3: A Four Sample DFT (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 23 Example 3: A Four Sample DFT (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 24 Example 3: A Four Sample DFT (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 25 Example 3: A Four Sample DFT (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 26 Example 3: A Four Sample DFT (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 27 Example 4: An Eight Sample DFT Suppose – N = 8 – Samples x 0, x 1, … x 7 are given corresponding to x a (kT), for T = 0,…,7 Find X(m)=?

9/4/2015Copyright, G. A. Tagliarini, PhD 28 Example 4: An Eight Sample DFT (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 29 Example 4: Fourier Transform Computations with m=0

9/4/2015Copyright, G. A. Tagliarini, PhD 30 Example 4: Fourier Transform Computations with m=0 (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 31 Example 4: Fourier Transform Computations (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 32 Example 4: Fourier Transform Computations with m=1 (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 33 Example 4: Fourier Transform Computations with m=1 (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 34 Example 4: Fourier Transform Computations (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 35 Example 4: Fourier Transform Computations with m=2 (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 36 Reducing the Computations The definition of the DFT naturally gives rise to an O(n 2 ) algorithm for implementation There is considerable computational symmetry that arises from taking sines and cosines at regularly spaced binary subdivisions of the complex unit circle |z|=1

9/4/2015Copyright, G. A. Tagliarini, PhD 37 Useful Notation

9/4/2015Copyright, G. A. Tagliarini, PhD 38 Writing the DFT With the W N Notation

9/4/2015Copyright, G. A. Tagliarini, PhD 39 Writing the DFT With the W N Notation (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 40 Writing the DFT With the W N Notation (continued)

9/4/2015Copyright, G. A. Tagliarini, PhD 41 So What? Some Improvement This sum has cut the number multiplications in half; i.e., a Fourier transform of length N=2 p, which normally requires sums of N terms each including a product can be computed from a sum having only N/2 products The number of sums has stayed the same Can we do better? What happens if we consider the even and odd terms separately?

9/4/2015Copyright, G. A. Tagliarini, PhD 42 Considering Even and Odd Terms

9/4/2015Copyright, G. A. Tagliarini, PhD 43 Considering Even and Odd Terms

9/4/2015Copyright, G. A. Tagliarini, PhD 44 Considering Even and Odd Terms

9/4/2015Copyright, G. A. Tagliarini, PhD 45 Observations An N-point DFT can be computed by calculating two N/2-point DFTs, each requiring half the effort – DFT is O(N 2 ) – The subdivided effort is 2 O((N/2) 2 ) = O(N 2 /2) This process can be iterated over log 2 (N) levels Thus, the FFT is O(1/2 N log 2 (N))

9/4/2015Copyright, G. A. Tagliarini, PhD 46 An FFT Algorithm (Cooley and Tukey) 1. Set θ = -2πd/N and r = N/2 2. For i = 1 to N-1 do //Calculate FT 1. Set w = cos(iθ)+j sin(iθ) 2. For k = 0 to N-1 do 1. Set u = 1 2. For m = 0 to r-1 do 1. t = z(k+m) - z(k+m+r) 2. z(k+m) = z(k+m) + z(k+m+r) 3. z(k+m+r) = tu 4. u = w u 3. k = k + 2r 3. i = 2i and r = r/2

9/4/2015Copyright, G. A. Tagliarini, PhD 47 An FFT Algorithm (continued) 3. For i = 0 to N-1 do //Sort results 1. Set r = i and k = 0 2. For m = 1 to N-1 do //Bit reversal 1. k = 2k + (r % 2) 2. r = r/2 3. m = 2m 3. If k > i do //swap z(i) and z(k); note t is complex 1. t = z(i) 2. z(i) = z(k) 3. z(k) = t

9/4/2015Copyright, G. A. Tagliarini, PhD 48 An FFT Algorithm (continued) 4. If d < 0 //Find FFT -1, Scale results For i = 0 to N-1 do z(i) = z(i)/N

9/4/2015Copyright, G. A. Tagliarini, PhD 49 Interpreting the Results What are the frequencies f k ? f k = (k f s ) / N, 0 ≤ k < N/2 Recall X(-f) = X*(f) if x(t) is real Spectra – Amplitude: A(m) ≡ |X(m)| – Phase: φ(m) ≡ arctan( Im(X(m)) / Re(X(m)) ) – Power (energy): PSD(m) ≡ |X(m)| 2

9/4/2015Copyright, G. A. Tagliarini, PhD 50 Correlation: Background x(k) =  u(k-d) + v(k) – x(k) is the received signal – u(k) is the transmitted signal attenuated by  – v(k) is additive random noise

9/4/2015Copyright, G. A. Tagliarini, PhD 51 Correlation: Definition Given a signal x(k) of length N and y(k) of length M≤N Assume y is padded with zeros to length N The cross correlation r xy of x with y is given by:

9/4/2015Copyright, G. A. Tagliarini, PhD 52 Correlation: x(N-1)…x(0)…x(i) y(0) y(N-1)…y(0)…… y(N-1)… y(0) …y(N-1-i) i=0 i=N-1 …y(N-1) i

9/4/2015Copyright, G. A. Tagliarini, PhD 53 Correlation: Fast Computation The cross correlation can be computed quickly using the FFT and FFT -1 r xy (k) = FFT -1 (X(m)Y*(m)), 0≤m<N Here – X(m) = FFT(x(k)) and – Y*(m) = FFT(y(k))

9/4/2015Copyright, G. A. Tagliarini, PhD 54 Convolution: Introduction A dynamic system is one which exhibits memory analogously to a sequential circuit Let – u(k) represent the system input at time k – y(k) represent the system output at time k We write y(k)=f(y(k-1), y(k-2),…, y(k-n), u(k), u(k-1), u(k-2),…,u(k-m)), i.e., the system output depends on some history as well as the current input

9/4/2015Copyright, G. A. Tagliarini, PhD 55 Convolution: Linear Dynamic System

9/4/2015Copyright, G. A. Tagliarini, PhD 56 Convolution: Linear Dynamic System If a i = 0 for i = 1,…, n, the system forms a moving average (MA) model If b j = 0 for j = 1,…, m, the system forms an auto-regressive (AR) model If there are both i and j for which a i ≠ 0 and b j ≠ 0, the system forms an auto-regressive, moving average (ARMA) model

9/4/2015Copyright, G. A. Tagliarini, PhD 57 Convolution: Pulse Response 1 A causal signal is one for which u(k)=0 for k<0 Suppose also that y(k)=0 for k<0 (initial conditions are zero) Let  (k)≡1 if k=0 and 0 if k≠0 The output of a linear system H, generated by the unit pulse input  (k) with initial conditions of zero is called the pulse response of the system, denoted by h(k) ≡ H(  (k)) (Equation *)

9/4/2015Copyright, G. A. Tagliarini, PhD 58 Convolution: Pulse Response 2 Compute the pulse response h(k) of a system by letting u(k) =  (k) Also note that where u(i) scales shifted pulses  (k-i)

9/4/2015Copyright, G. A. Tagliarini, PhD 59 Convolution: By the linearity in Equation *, h(k) = H(  (k)), we have – H(u(i)  (k-i)) = u(i) h(k-i), so that This is the convolution of h(k) with u(k)

9/4/2015Copyright, G. A. Tagliarini, PhD 60 Fast Convolution Given input u(k) and pulse response h(k) – U(m) = FFT(u(k)) – H(m) = FFT(h(k)) The Fast Convolution y(k) is given by y(k) = FFT -1 (H(m) U(m))

9/4/2015Copyright, G. A. Tagliarini, PhD 61 Example: Averaging Filter Let – h(k) = 1/p, for 0 ≤ k < p and – h(k) = 0 elsewhere The convolution of h(k) with an input u(k) implements a MA filter of length p This filter calculates an average of p samples of the input signal u

9/4/2015Copyright, G. A. Tagliarini, PhD 62 Digital Filtering: Notation (Again!!) Let – Input signal u(k) with FT(u(k)) = U(m) – Pulse response h(k) with FT(h(k)) = H(m) – Output signal y(k) with FT(y(k)) = Y(m) – 0 ≤ m < N Y(m) = H(m) U(m)

9/4/2015Copyright, G. A. Tagliarini, PhD 63 Digital Filtering: Polar Notation 1 The polar notation for the Fourier transforms U(m) = A u (m) exp(j  u (m)) Y(m) = A y (m) exp(j  y (m)) H(m) = A(m) exp(j  (m)) Where A(m) and  (m) are the magnitude and phase response of the filter respectively

9/4/2015Copyright, G. A. Tagliarini, PhD 64 Digital Filtering: Polar Notation 2 Since A y (m) exp(j  y (m)) = Y(m) = H(m) U(m) = A(m) exp(j  (m)) A u (m) exp(j  u (m)) = A(m) A u (m) exp(j  (m)) exp(j  u (m)) = A(m) A u (m) exp(j (  (m)+  u (m))) Output Amplitude is the product of filter and signal amplitude Phase is sum of the filter and signal phases

9/4/2015Copyright, G. A. Tagliarini, PhD 65 Digital Filtering: Polar Notation 3 Again Output amplitude is the product of filter and signal amplitude spectra, A y (m) = A(m) A u (m) Output phase is sum of the filter and signal phase spectra,  y (m)(  (m)+  u (m)))

9/4/2015Copyright, G. A. Tagliarini, PhD 66 Common Filter Types f s /2fHfH A(f) 0 1 f s /2fHfH A(f) 0 1 f s /2fLfL A(f) 0 1 f s /2fHfH A(f) 0 1 Low Pass Band Pass High Pass Notch fLfL fLfL

9/4/2015Copyright, G. A. Tagliarini, PhD 67 Filter Design

9/4/2015Copyright, G. A. Tagliarini, PhD 68 Filter Design Approach:

9/4/2015Copyright, G. A. Tagliarini, PhD 69 Fourier Filter Coefficients

9/4/2015Copyright, G. A. Tagliarini, PhD 70 Gibbs Phenomena: Low Pass FilterLow Pass Filter

9/4/2015Copyright, G. A. Tagliarini, PhD 71 Windowing Coefficients

9/4/2015Copyright, G. A. Tagliarini, PhD 72 Windowing Functions

9/4/2015Copyright, G. A. Tagliarini, PhD 73 Suppressing the Gibbs Phenomenon: Windowed Low Pass FilterLow Pass Filter

9/4/2015Copyright, G. A. Tagliarini, PhD 74 FIR Filter Algorithm

9/4/2015Copyright, G. A. Tagliarini, PhD 75 Two-Dimensional Fourier Transform Previous signals were all one-dimensional There are common two-dimensional signals—IMAGES! There are some differences – Sampling interval reflects change of position – Spatial frequency reflects rate of change in pixel values

9/4/2015Copyright, G. A. Tagliarini, PhD 76 Notation for 2-D Fourier Transforms

9/4/2015Copyright, G. A. Tagliarini, PhD 77 Definition: 2-D DFT

9/4/2015Copyright, G. A. Tagliarini, PhD 78 Definition: 2-D Inverse DFT

9/4/2015Copyright, G. A. Tagliarini, PhD 79 Algorithm for 2-D FFT

9/4/2015Copyright, G. A. Tagliarini, PhD 80 2-D Spectral Definitions

9/4/2015Copyright, G. A. Tagliarini, PhD 81 Interpreting the Results

9/4/2015Copyright, G. A. Tagliarini, PhD 82 Example: A Structured Image Construct a linear combination of 2-D sine functions Render a 2-D power spectral density estimate Observe the effects of ideal filtering on the power spectrum

9/4/2015Copyright, G. A. Tagliarini, PhD 83 Examples

9/4/2015Copyright, G. A. Tagliarini, PhD 84 Example: Bandpass Filter

9/4/2015Copyright, G. A. Tagliarini, PhD 85 System Identification Given a linear system Model the input/output relationship Black box system with unknown transfer function Input u(k)Output y(k)

9/4/2015Copyright, G. A. Tagliarini, PhD 86 Modeling Basis

9/4/2015Copyright, G. A. Tagliarini, PhD 87 Modeling Basis Re-Written

9/4/2015Copyright, G. A. Tagliarini, PhD 88 Another Simplifying Notation

9/4/2015Copyright, G. A. Tagliarini, PhD 89 The System Identification Question Can we find a parameter vector θ that minimizes the error e(θ) given by e(θ) ≡ y – Xθ? Yes (!) but it takes some calculus. Consider the real valued, quadratic, error expression J(θ) given by J(θ) ≡ e T (θ)e(θ) (note J(θ)=|e(θ)| 2 ), differentiate, and find the critical value

9/4/2015Copyright, G. A. Tagliarini, PhD 90 Some Algebra J(θ) = e T (θ)e(θ) = (y – Xθ) T (y – Xθ) = (y T – (Xθ) T ) (y – Xθ) = (y T – θ T X T ) (y – Xθ) = y T y – y T Xθ – θ T X T y + θ T X T Xθ = y T y – 2y T Xθ + θ T X T Xθ (Why?) dJ(θ)/dθ = –2y T X+2θ T X T X Hence dJ(θ)/dθ = 0 ↔ y T X = θ T X T X

9/4/2015Copyright, G. A. Tagliarini, PhD 91 Some Algebra (continued) dJ(θ)/dθ = 0 ↔ y T X = θ T X T X ↔ y T X (X T X) -1 = θ T ↔ θ T = y T X (X T X) -1 ↔ (θ T ) T = (y T X (X T X) -1 ) T ↔ θ = ((X T X) -1 ) T X T (y T ) T ↔ θ = (X T X) -1 X T y Thus, if X T X is invertible and y is given we can compute θ!

9/4/2015Copyright, G. A. Tagliarini, PhD 92 Conditions for X T X to be Invertible The number of samples N must be at least as great as the number of unknown parameters n + (m + 1) The magnitude spectrum of u(k) must be nonzero for at least n + (m+1) frequencies