1. Introduction to Fourier Processing A spectrum of possibilities…

2. 9/4/2015Copyright, G. A. Tagliarini, PhD 2 Organization Signals: analog and discrete The Fourier Transform – Definition – Discretization – Nyquist limits – Computation via the FFT Applications – Correlation – Convolution – Filtering Two-dimensional FFTs

3. 9/4/2015Copyright, G. A. Tagliarini, PhD 3 A Signal

4. 9/4/2015Copyright, G. A. Tagliarini, PhD 4 Given: An analog signal x a (t), t ≥ 0 The signal is sampled at uniform intervals of length T to produce a discrete time signal x(k) ≡ x a (kT), k = 0, 1, 2,… The signal x(k) is said to have a sampling frequency f s = 1/T The sampled signal was periodic

5. 9/4/2015Copyright, G. A. Tagliarini, PhD 5 The Fourier Transform (FT)

6. 9/4/2015Copyright, G. A. Tagliarini, PhD 6 The Inverse Fourier Transform (FT -1 )

7. 9/4/2015Copyright, G. A. Tagliarini, PhD 7 Observations The Fourier transform maps real-valued functions of t (time) into complex-valued functions of f (frequency) One can reconstruct a time signal by using FT -1 Since we will work with discrete samples from signals, our digital signal processing (DSP) will typically not require analytical solutions of integral equations!!!

8. 9/4/2015Copyright, G. A. Tagliarini, PhD 8 The Energy Density Spectrum of x(t) The energy density spectrum, a.k.a. power spectrum, of a signal x(t) is given by |X(f)| 2 If x(t) is real, X(-f) = X * (f), the complex conjugate of X(f) If 0≤f 0 ≤f 1, the energy in the band from f 0 through f 1 is given by:

9. 9/4/2015Copyright, G. A. Tagliarini, PhD 9 Example 1: Energy Density Spectrum See example for FFT and PSD

10. 9/4/2015Copyright, G. A. Tagliarini, PhD 10 Bandwidth Limited Signals If  a value B>0, such that X(f) = 0 for all values of f>B, then x(t) is said to be bandwidth limited Bandwidth limited signals occur often – Bio-mechanical properties limit the highest frequency signals that animals can generate naturally: Vocalizations in humans (40 Hz – 20 KHz) Vocalizations in dolphins and whales (up to 100 KHz) – Electrical and physical characteristics of components also constrain frequency ranges in electrical circuits Common analog telephone circuits (100 - 3,000 Hz) FM radio band width (15 - 30 KHz)

11. 9/4/2015Copyright, G. A. Tagliarini, PhD 11 Nyquist Sampling Recall that the sampling frequency, f s, is the reciprocal of the sampling interval T, i.e., f s = 1/T Suppose a signal x(t) is bandwidth limited to B Hz By the sampling theorem, if f s ≥ 2 B, the signal can be approximated (interpolated) between samples. The Nyquist critical frequency is f cN = 2 B The critical sampling interval T c = 1/(2B)

12. 9/4/2015Copyright, G. A. Tagliarini, PhD 12 Nyquist Sampling: Interpretation Given a signal x(t) with bandwidth limit B: – The Nyquist critical frequency is f cN = 2 B – The critical sampling interval T c = 1/(2B) Meaning – To reconstruct x(t) without distortion x(t), the sampling rate must be at least twice the highest frequency present – The maximum sampling interval is one half the wavelength of the highest frequency component The sampling process must obtain at least two samples per cycle of the highest frequency present!

13. 9/4/2015Copyright, G. A. Tagliarini, PhD 13 Interpolating Between Samples Assume – x(t) is bandwidth limited by B – The sampling frequency f s ≥ 2B, where T = 1/ f s Then, by the sampling theorem we can interpolate between samples using:

14. 9/4/2015Copyright, G. A. Tagliarini, PhD 14 One More Useful Notion Recall a definition of the definite integral

15. 9/4/2015Copyright, G. A. Tagliarini, PhD 15 The Discrete Fourier Transform Given the definition of the Fourier Transform The discrete Fourier transform (DFT) is given by:

16. 9/4/2015Copyright, G. A. Tagliarini, PhD 16 The Discrete Fourier Transform The discrete Fourier transform (DFT): approximates the Fourier transform at the N frequency values determined by m, 0≤m<N

17. 9/4/2015Copyright, G. A. Tagliarini, PhD 17 The Inverse of the Discrete Fourier Transform Recall the inverse Fourier transform is: The inverse of the DFT is given by:

18. 9/4/2015Copyright, G. A. Tagliarini, PhD 18 Example 2: A Two-sample DFT Suppose – N = 2 – Samples x 0, x 1 are given corresponding to x a (kT), for T = 0, 1 Find X(m)=?

19. 9/4/2015Copyright, G. A. Tagliarini, PhD 19 Example 2: A Two-sample DFT (continued)

20. 9/4/2015Copyright, G. A. Tagliarini, PhD 20 Example 2: A Two-sample DFT (continued)

21. 9/4/2015Copyright, G. A. Tagliarini, PhD 21 Example 3: Four Sample DFT Suppose – N = 4 – Samples x 0, x 1, … x 3 are given corresponding to x a (kT), for k = 0,…,3 Find X(m)=?

22. 9/4/2015Copyright, G. A. Tagliarini, PhD 22 Example 3: A Four Sample DFT (continued)

23. 9/4/2015Copyright, G. A. Tagliarini, PhD 23 Example 3: A Four Sample DFT (continued)

24. 9/4/2015Copyright, G. A. Tagliarini, PhD 24 Example 3: A Four Sample DFT (continued)

25. 9/4/2015Copyright, G. A. Tagliarini, PhD 25 Example 3: A Four Sample DFT (continued)

26. 9/4/2015Copyright, G. A. Tagliarini, PhD 26 Example 3: A Four Sample DFT (continued)

27. 9/4/2015Copyright, G. A. Tagliarini, PhD 27 Example 4: An Eight Sample DFT Suppose – N = 8 – Samples x 0, x 1, … x 7 are given corresponding to x a (kT), for T = 0,…,7 Find X(m)=?

28. 9/4/2015Copyright, G. A. Tagliarini, PhD 28 Example 4: An Eight Sample DFT (continued)

29. 9/4/2015Copyright, G. A. Tagliarini, PhD 29 Example 4: Fourier Transform Computations with m=0

30. 9/4/2015Copyright, G. A. Tagliarini, PhD 30 Example 4: Fourier Transform Computations with m=0 (continued)

31. 9/4/2015Copyright, G. A. Tagliarini, PhD 31 Example 4: Fourier Transform Computations (continued)

32. 9/4/2015Copyright, G. A. Tagliarini, PhD 32 Example 4: Fourier Transform Computations with m=1 (continued)

33. 9/4/2015Copyright, G. A. Tagliarini, PhD 33 Example 4: Fourier Transform Computations with m=1 (continued)

34. 9/4/2015Copyright, G. A. Tagliarini, PhD 34 Example 4: Fourier Transform Computations (continued)

35. 9/4/2015Copyright, G. A. Tagliarini, PhD 35 Example 4: Fourier Transform Computations with m=2 (continued)

36. 9/4/2015Copyright, G. A. Tagliarini, PhD 36 Reducing the Computations The definition of the DFT naturally gives rise to an O(n 2 ) algorithm for implementation There is considerable computational symmetry that arises from taking sines and cosines at regularly spaced binary subdivisions of the complex unit circle |z|=1

37. 9/4/2015Copyright, G. A. Tagliarini, PhD 37 Useful Notation

38. 9/4/2015Copyright, G. A. Tagliarini, PhD 38 Writing the DFT With the W N Notation

39. 9/4/2015Copyright, G. A. Tagliarini, PhD 39 Writing the DFT With the W N Notation (continued)

40. 9/4/2015Copyright, G. A. Tagliarini, PhD 40 Writing the DFT With the W N Notation (continued)

41. 9/4/2015Copyright, G. A. Tagliarini, PhD 41 So What? Some Improvement This sum has cut the number multiplications in half; i.e., a Fourier transform of length N=2 p, which normally requires sums of N terms each including a product can be computed from a sum having only N/2 products The number of sums has stayed the same Can we do better? What happens if we consider the even and odd terms separately?

42. 9/4/2015Copyright, G. A. Tagliarini, PhD 42 Considering Even and Odd Terms

43. 9/4/2015Copyright, G. A. Tagliarini, PhD 43 Considering Even and Odd Terms

44. 9/4/2015Copyright, G. A. Tagliarini, PhD 44 Considering Even and Odd Terms

45. 9/4/2015Copyright, G. A. Tagliarini, PhD 45 Observations An N-point DFT can be computed by calculating two N/2-point DFTs, each requiring half the effort – DFT is O(N 2 ) – The subdivided effort is 2 O((N/2) 2 ) = O(N 2 /2) This process can be iterated over log 2 (N) levels Thus, the FFT is O(1/2 N log 2 (N))

46. 9/4/2015Copyright, G. A. Tagliarini, PhD 46 An FFT Algorithm (Cooley and Tukey) 1. Set θ = -2πd/N and r = N/2 2. For i = 1 to N-1 do //Calculate FT 1. Set w = cos(iθ)+j sin(iθ) 2. For k = 0 to N-1 do 1. Set u = 1 2. For m = 0 to r-1 do 1. t = z(k+m) - z(k+m+r) 2. z(k+m) = z(k+m) + z(k+m+r) 3. z(k+m+r) = tu 4. u = w u 3. k = k + 2r 3. i = 2i and r = r/2

47. 9/4/2015Copyright, G. A. Tagliarini, PhD 47 An FFT Algorithm (continued) 3. For i = 0 to N-1 do //Sort results 1. Set r = i and k = 0 2. For m = 1 to N-1 do //Bit reversal 1. k = 2k + (r % 2) 2. r = r/2 3. m = 2m 3. If k > i do //swap z(i) and z(k); note t is complex 1. t = z(i) 2. z(i) = z(k) 3. z(k) = t

48. 9/4/2015Copyright, G. A. Tagliarini, PhD 48 An FFT Algorithm (continued) 4. If d < 0 //Find FFT -1, Scale results For i = 0 to N-1 do z(i) = z(i)/N

49. 9/4/2015Copyright, G. A. Tagliarini, PhD 49 Interpreting the Results What are the frequencies f k ? f k = (k f s ) / N, 0 ≤ k < N/2 Recall X(-f) = X*(f) if x(t) is real Spectra – Amplitude: A(m) ≡ |X(m)| – Phase: φ(m) ≡ arctan( Im(X(m)) / Re(X(m)) ) – Power (energy): PSD(m) ≡ |X(m)| 2

50. 9/4/2015Copyright, G. A. Tagliarini, PhD 50 Correlation: Background x(k) =  u(k-d) + v(k) – x(k) is the received signal – u(k) is the transmitted signal attenuated by  – v(k) is additive random noise

51. 9/4/2015Copyright, G. A. Tagliarini, PhD 51 Correlation: Definition Given a signal x(k) of length N and y(k) of length M≤N Assume y is padded with zeros to length N The cross correlation r xy of x with y is given by:

52. 9/4/2015Copyright, G. A. Tagliarini, PhD 52 Correlation: x(N-1)…x(0)…x(i) y(0) y(N-1)…y(0)…… y(N-1)… y(0) …y(N-1-i) i=0 i=N-1 …y(N-1) i

53. 9/4/2015Copyright, G. A. Tagliarini, PhD 53 Correlation: Fast Computation The cross correlation can be computed quickly using the FFT and FFT -1 r xy (k) = FFT -1 (X(m)Y*(m)), 0≤m<N Here – X(m) = FFT(x(k)) and – Y*(m) = FFT(y(k))

54. 9/4/2015Copyright, G. A. Tagliarini, PhD 54 Convolution: Introduction A dynamic system is one which exhibits memory analogously to a sequential circuit Let – u(k) represent the system input at time k – y(k) represent the system output at time k We write y(k)=f(y(k-1), y(k-2),…, y(k-n), u(k), u(k-1), u(k-2),…,u(k-m)), i.e., the system output depends on some history as well as the current input

55. 9/4/2015Copyright, G. A. Tagliarini, PhD 55 Convolution: Linear Dynamic System

56. 9/4/2015Copyright, G. A. Tagliarini, PhD 56 Convolution: Linear Dynamic System If a i = 0 for i = 1,…, n, the system forms a moving average (MA) model If b j = 0 for j = 1,…, m, the system forms an auto-regressive (AR) model If there are both i and j for which a i ≠ 0 and b j ≠ 0, the system forms an auto-regressive, moving average (ARMA) model

57. 9/4/2015Copyright, G. A. Tagliarini, PhD 57 Convolution: Pulse Response 1 A causal signal is one for which u(k)=0 for k<0 Suppose also that y(k)=0 for k<0 (initial conditions are zero) Let  (k)≡1 if k=0 and 0 if k≠0 The output of a linear system H, generated by the unit pulse input  (k) with initial conditions of zero is called the pulse response of the system, denoted by h(k) ≡ H(  (k)) (Equation *)

58. 9/4/2015Copyright, G. A. Tagliarini, PhD 58 Convolution: Pulse Response 2 Compute the pulse response h(k) of a system by letting u(k) =  (k) Also note that where u(i) scales shifted pulses  (k-i)

59. 9/4/2015Copyright, G. A. Tagliarini, PhD 59 Convolution: By the linearity in Equation *, h(k) = H(  (k)), we have – H(u(i)  (k-i)) = u(i) h(k-i), so that This is the convolution of h(k) with u(k)

60. 9/4/2015Copyright, G. A. Tagliarini, PhD 60 Fast Convolution Given input u(k) and pulse response h(k) – U(m) = FFT(u(k)) – H(m) = FFT(h(k)) The Fast Convolution y(k) is given by y(k) = FFT -1 (H(m) U(m))

61. 9/4/2015Copyright, G. A. Tagliarini, PhD 61 Example: Averaging Filter Let – h(k) = 1/p, for 0 ≤ k < p and – h(k) = 0 elsewhere The convolution of h(k) with an input u(k) implements a MA filter of length p This filter calculates an average of p samples of the input signal u

62. 9/4/2015Copyright, G. A. Tagliarini, PhD 62 Digital Filtering: Notation (Again!!) Let – Input signal u(k) with FT(u(k)) = U(m) – Pulse response h(k) with FT(h(k)) = H(m) – Output signal y(k) with FT(y(k)) = Y(m) – 0 ≤ m < N Y(m) = H(m) U(m)

63. 9/4/2015Copyright, G. A. Tagliarini, PhD 63 Digital Filtering: Polar Notation 1 The polar notation for the Fourier transforms U(m) = A u (m) exp(j  u (m)) Y(m) = A y (m) exp(j  y (m)) H(m) = A(m) exp(j  (m)) Where A(m) and  (m) are the magnitude and phase response of the filter respectively

64. 9/4/2015Copyright, G. A. Tagliarini, PhD 64 Digital Filtering: Polar Notation 2 Since A y (m) exp(j  y (m)) = Y(m) = H(m) U(m) = A(m) exp(j  (m)) A u (m) exp(j  u (m)) = A(m) A u (m) exp(j  (m)) exp(j  u (m)) = A(m) A u (m) exp(j (  (m)+  u (m))) Output Amplitude is the product of filter and signal amplitude Phase is sum of the filter and signal phases

65. 9/4/2015Copyright, G. A. Tagliarini, PhD 65 Digital Filtering: Polar Notation 3 Again Output amplitude is the product of filter and signal amplitude spectra, A y (m) = A(m) A u (m) Output phase is sum of the filter and signal phase spectra,  y (m)(  (m)+  u (m)))

66. 9/4/2015Copyright, G. A. Tagliarini, PhD 66 Common Filter Types f s /2fHfH A(f) 0 1 f s /2fHfH A(f) 0 1 f s /2fLfL A(f) 0 1 f s /2fHfH A(f) 0 1 Low Pass Band Pass High Pass Notch fLfL fLfL

67. 9/4/2015Copyright, G. A. Tagliarini, PhD 67 Filter Design

68. 9/4/2015Copyright, G. A. Tagliarini, PhD 68 Filter Design Approach:

69. 9/4/2015Copyright, G. A. Tagliarini, PhD 69 Fourier Filter Coefficients

70. 9/4/2015Copyright, G. A. Tagliarini, PhD 70 Gibbs Phenomena: Low Pass FilterLow Pass Filter

71. 9/4/2015Copyright, G. A. Tagliarini, PhD 71 Windowing Coefficients

72. 9/4/2015Copyright, G. A. Tagliarini, PhD 72 Windowing Functions

73. 9/4/2015Copyright, G. A. Tagliarini, PhD 73 Suppressing the Gibbs Phenomenon: Windowed Low Pass FilterLow Pass Filter

74. 9/4/2015Copyright, G. A. Tagliarini, PhD 74 FIR Filter Algorithm

75. 9/4/2015Copyright, G. A. Tagliarini, PhD 75 Two-Dimensional Fourier Transform Previous signals were all one-dimensional There are common two-dimensional signals—IMAGES! There are some differences – Sampling interval reflects change of position – Spatial frequency reflects rate of change in pixel values

76. 9/4/2015Copyright, G. A. Tagliarini, PhD 76 Notation for 2-D Fourier Transforms

77. 9/4/2015Copyright, G. A. Tagliarini, PhD 77 Definition: 2-D DFT

78. 9/4/2015Copyright, G. A. Tagliarini, PhD 78 Definition: 2-D Inverse DFT

79. 9/4/2015Copyright, G. A. Tagliarini, PhD 79 Algorithm for 2-D FFT

80. 9/4/2015Copyright, G. A. Tagliarini, PhD 80 2-D Spectral Definitions

81. 9/4/2015Copyright, G. A. Tagliarini, PhD 81 Interpreting the Results

82. 9/4/2015Copyright, G. A. Tagliarini, PhD 82 Example: A Structured Image Construct a linear combination of 2-D sine functions Render a 2-D power spectral density estimate Observe the effects of ideal filtering on the power spectrum

83. 9/4/2015Copyright, G. A. Tagliarini, PhD 83 Examples

84. 9/4/2015Copyright, G. A. Tagliarini, PhD 84 Example: Bandpass Filter

85. 9/4/2015Copyright, G. A. Tagliarini, PhD 85 System Identification Given a linear system Model the input/output relationship Black box system with unknown transfer function Input u(k)Output y(k)

86. 9/4/2015Copyright, G. A. Tagliarini, PhD 86 Modeling Basis

87. 9/4/2015Copyright, G. A. Tagliarini, PhD 87 Modeling Basis Re-Written

88. 9/4/2015Copyright, G. A. Tagliarini, PhD 88 Another Simplifying Notation

89. 9/4/2015Copyright, G. A. Tagliarini, PhD 89 The System Identification Question Can we find a parameter vector θ that minimizes the error e(θ) given by e(θ) ≡ y – Xθ? Yes (!) but it takes some calculus. Consider the real valued, quadratic, error expression J(θ) given by J(θ) ≡ e T (θ)e(θ) (note J(θ)=|e(θ)| 2 ), differentiate, and find the critical value

90. 9/4/2015Copyright, G. A. Tagliarini, PhD 90 Some Algebra J(θ) = e T (θ)e(θ) = (y – Xθ) T (y – Xθ) = (y T – (Xθ) T ) (y – Xθ) = (y T – θ T X T ) (y – Xθ) = y T y – y T Xθ – θ T X T y + θ T X T Xθ = y T y – 2y T Xθ + θ T X T Xθ (Why?) dJ(θ)/dθ = –2y T X+2θ T X T X Hence dJ(θ)/dθ = 0 ↔ y T X = θ T X T X

91. 9/4/2015Copyright, G. A. Tagliarini, PhD 91 Some Algebra (continued) dJ(θ)/dθ = 0 ↔ y T X = θ T X T X ↔ y T X (X T X) -1 = θ T ↔ θ T = y T X (X T X) -1 ↔ (θ T ) T = (y T X (X T X) -1 ) T ↔ θ = ((X T X) -1 ) T X T (y T ) T ↔ θ = (X T X) -1 X T y Thus, if X T X is invertible and y is given we can compute θ!

92. 9/4/2015Copyright, G. A. Tagliarini, PhD 92 Conditions for X T X to be Invertible The number of samples N must be at least as great as the number of unknown parameters n + (m + 1) The magnitude spectrum of u(k) must be nonzero for at least n + (m+1) frequencies