Fourier Transformations Grad Algorithms Fourier Transformations Sum of Sine/Cosine Functions A Pulse, Linearity, & Periodic Duality Change of Basis Orthogonal Basis Rectangular vs Polar vs Complex Continuous Functions Radio Signal Convolution Linear Filter * FFT in nlogn Time FFT Code FFT Butterfly JPG Polynomial Basis Evaluating & Interpolating FFT in nlogn Time Roots of Unity Same FFT Code & Butterfly Inverse FFT Polynomial Multiplication Integer Multiplication Jeff Edmonds York University COSC 6111
Fourier Transformation are a change of basis from the time basis to sine/cosine basis JPG or polynomial basis Purposes: Signal Processing Compressing data (eg images with .jpg) Multiplying integers in n logn loglogn time. Error correcting codes. …. Amazingly once you include complex numbers, the FFT code for sine/cosines and for polynomials are the SAME. http://www.dspguide.com/ch8.htm The Scientist and Engineer's Guide to Digital Signal Processing By Steven W. Smith, Ph.D.
Fourier Transformation Sine &Cosine Basis A continuous periodic function t time y(t) Swings, capacitors, and inductors all resonate at a given frequency, which is how the circuit picks out the contribution of a given frequency. Find the contribution of each frequency Purposes: Signal Processing Compressing data (eg images with .jpg) Multiplying integers in n logn loglogn time. Error correcting cods. ….
Fourier Transformation Sine &Cosine Basis y(x) = x y(x) 2 sin(x) - sin(2x) + 2/3 sin(3x) Surely this can’t be expressed as sum of sines and cosines.
Fourier Transformation Sine &Cosine Basis y(x) = x2 y(x) -4 sin(x) + sin(2x) - 4/9 sin(3x)
Fourier Transformation Sine &Cosine Basis A continuous periodic function t time y(t) A discrete periodic function Time Domain j[0,n-1] y[j] y[] is an array of real values j 1 2 3 4 5 6 7 8 9 10 6
Fourier Transformation Sine &Cosine Basis A continuous periodic function t time y(t) A discrete periodic function Time Domain j[0,n-1] y[j] The function y is expressed as a linear combination of the basis functions cf and sf. y[] YRe[] YIm[] Y[] 7
Fourier Transformation Sine &Cosine Basis A continuous periodic function t time y(t) A discrete periodic function Time Domain j[0,n-1] y[j] The function y is expressed as a linear combination of the basis functions cf and sf. YRe [f] is the coefficient of the bases function cf. YIm [f] is the coefficient of the bases function sf. Recall: ei = cos() + i sin() Hence, the coefficients of cosine are subscripted “Real” And of sine with “Imaginary”. 8
Fourier Transformation Single Pulse Lets consider a few examples and a few properties. Time Domain y Frequency Domain Y Cosine wave Cosine with f=8 Delta function Impulse at Yre[8] One non-zero coefficient in the real frequency domain arises from what in the time domain? One cosine wave of the stated frequency.
Fourier Transformation Single Pulse Time Domain y Frequency Domain Y Cosine wave Cosine with f=8 Delta function Impulse at Yre[8] In the discrete world, delta has one nonzero value, which is delta(8) = 1. In the continuous world, delta has the value 1/ for a width for a total area of one.
Fourier Transformation Single Pulse Time Domain y Frequency Domain Y Cosine wave Cosine with f=4 Delta function Impulse at Yre[4] A different frequency. A different coefficient?
Fourier Transformation Linearity of FT Time Domain y Frequency Domain Y Scale up the amplitude in the time domain. c 2c -c -2c Cosine wave Cosine with f=4 Scales up the amplitude in the frequency domain. c 2c -c Delta function Impulse at Yre[4] Linearity of FT: If Y=FT(y), then cY=FT(cy)
Fourier Transformation Linearity of FT Time Domain y Frequency Domain Y Linearity of FT: FT(c1∙y1+c2∙y2) = c1∙FT(y1) + c2∙FT(y2)
Fourier Transformation Periodic Signals Time Domain y Frequency Domain Y j y[j] j Sine and cosine waves are periodic. The time domain is composed of these. Hence, the time domain is periodic. Strangely, the frequency domain is periodic too.
Fourier Transformation Periodic Signals Time Domain y Frequency Domain Y One frequency solution Frequency This set of discrete points needs to be interpreted as a continuous function. This is the obvious solution. Leading to this frequency decomposition. This is another. Another. Cos(-) = Cos() = Cos(+2 ) =…
Fourier Transformation Duality of FT Time Domain y Frequency Domain Y Cosine wave Cosine with f=4 Delta function Impulse at Yre[4] Delta function Impulse at y[4] Ok, we know this. But what if we switch them?
Fourier Transformation Duality of FT Time Domain y Frequency Domain Y Cosine wave Cosine with f=4 Delta function Impulse at Yre[4] Delta function Impulse at y[4] Cosine wave Cosine with f=4 A single pulse in the time domain is produced from what sum of sines and cosines? Not clear! The curve of the coefficients forms a sine/cosine wave! Strange but true!
[ ][ ] =[ ] Fourier Transformation 1/n Time Domain y Duality of FT Time Domain y Frequency Domain Y Cosine wave Cosine with f=4 Delta function Impulse at Yre[4] Delta function Impulse at y[4] Cosine wave Cosine with f=4 Arises from the duality of FT and its inverse. [ ][ ] =[ ] s1[1] s1[2] s2[1] s2[2] Y[1] Y[2] 1/n y[1] y[2] Sorry! we do this later.
Fourier Transformation Duality of FT Time Domain y Frequency Domain Y Cosine wave Cosine with f=4 Delta function Impulse at Yre[4] Delta function Impulse at y[4] Cosine wave Cosine with f=4 Arises from the duality of FT and its inverse. Duality of FT: If Y=FT(y), then y=FT(Y)
Fourier Transformation Duality of FT Time Domain y Frequency Domain Y Square wave Sinc function How do you get these corner? Sinc function Square wave ? Duality of FT: If Y=FT(y), then y=FT(Y)
Fourier Transformation Duality of FT Time Domain y Frequency Domain Y Gaussian Duality of FT: If Y=FT(y), then y=FT(Y)
Fourier Transformation Change of Basis Change of Basis: T([a1,a2,…,ad]) = [A1,A2,…,Ad] Changes the basis used to describe an object. The Time basis of a vector space: A tuple <w1,w2,…,wd> of basis objects Linearly independent Spans the space uniquely "v $[a1,a2,…,ad], v = a1w1+a2w2 +… + adwd The Fourier basis of a vector space: A tuple <W1,W2,…,Wd> of basis objects Linearly independent Spans the space uniquely "v $[A1,A2,…,Ad], v = A1W1+A2W2 +… + AdWd 22
Fourier Transformation Change of Basis Change of Basis: T([a1,a2,…,ad]) = [A1,A2,…,Ad] Changes the basis used to describe an object. "v $[a1,a2,…,ad], v = a1w1 +a2w2 +… + adwd "v $[A1,A2,…,Ad], v = A1W1+A2W2 +… + AdWd =[w1,w2] =[ , ] Standard Basis New Basis =[W1,W2] =[ , ] W1[1] W1[2] W2[1] W2[2] v = v = [a1,a2] = [3,2] [A1,A2] = [11/5,32/5] [ ][ ] =[ ] W1[1] W1[2] W2[1] W2[2] a1 a2 A1 A2 23
Fourier Transformation Change of Basis Change of Basis: T(y[0],y[1],…,y[n-1]) = [YRe[0],…,YIm[n/2]] Changes the basis used to describe an object. "y $[y[0],y[1],…,y[n-1]], y = y[0]I1 +y[1]I2 +… + y[n-1]In The time basis j’ Ij[j’] zero one j Time Basis =[ , ] =[?,?] =[I1,I2,…] ? = y = y[0]=3 y[1]=2 ? A discrete periodic function j y[j] 24
Fourier Transformation Change of Basis Change of Basis: T(y[0],y[1],…,y[n-1]) = [YRe[0],…,YIm[n/2]] Changes the basis used to describe an object. "y $[y[0],y[1],…,y[n-1]], y = y[0]I1 +y[1]I2 +… + y[n-1]In y = YRe[0]∙c1+YIm[0]∙s1+ ,…,YRe[n/2]∙sn/2+YIm[n/2]∙sn/2 Time Basis =[ , ] =[I1,I2,…] y = Fourier Basis =[ , ] =[?,?] =[c1,s1,..] c1 sn/2 cn/2 s1 y = y[0]=3 y[1]=2 ? YRe[0] =11/5 YIm[0] =32/5 A discrete periodic function j y[j] 25
Fourier Transformation Change of Basis Change of Basis: T(y[0],y[1],…,y[n-1]) = [YRe[0],…,YIm[n/2]] Changes the basis used to describe an object. "y $[y[0],y[1],…,y[n-1]], y = y[0]I1 +y[1]I2 +… + y[n-1]In y = YRe[0]∙c1+YIm[0]∙s1+ ,…,YRe[n/2]∙sn/2+YIm[n/2]∙sn/2 Time Basis =[ , ] =[I1,I2,…] y = Fourier Basis =[ , ] =[c1,s1,..] c1 sn/2 cn/2 s1 y = y[0]=3 y[1]=2 YRe[0] =11/5 YIm[0] =32/5 s1[1] s1[2] s2[1] s2[2] Y[1] Y[2] y[1] y[2] [ ] [ ] =[ ] s1[1] s2[1] s1[2] s2[2] Y[1] Y[2] [ ] [ ] =[ ] y[1] y[2] -1 26
Fourier Transformation Orthogonal Basis Time Basis =[ , ] =[I1,I2,…] Fourier Basis =[ , ] =[c1,s1,..] This frequency basis vectors are orthogonal and of (almost) uniform length: Evenly spaced points within an integer number of full cycles. Sum of squares 27
Fourier Transformation Orthogonal Basis Time Basis =[ , ] =[I1,I2,…] Fourier Basis =[ , ] =[c1,s1,..] This frequency basis vectors are orthogonal and of (almost) uniform length: Evenly spaced points within an integer number of full cycles. Sum of products 28
Fourier Transformation Orthogonal Basis Time Basis =[ , ] =[I1,I2,…] Fourier Basis =[ , ] =[c1,s1,..] This frequency basis vectors are orthogonal and of (almost) uniform length: [ ][ ]=[ ] s1[1] s1[2] s2[1] s2[2] n/2 [ ] = [ ] s1[1] s1[2] s2[1] s2[2] -1 2/n 29
[ ][ ] =[ ] Fourier Transformation =[I1,I2,…] =[c1,s1,..] =[ , ] Orthogonal Basis Time Basis =[ , ] =[I1,I2,…] Fourier Basis =[ , ] =[c1,s1,..] This frequency basis vectors are orthogonal and of (almost) uniform length: [ ][ ] =[ ] s1[1] s1[2] s2[1] s2[2] Y[1] Y[2] 2/n y[1] y[2] This is the correlation between y and cf 30
Fourier Transformation Change of Basis Viewing correlation a different way: YRe[0] =11/5 YIm[0] =32/5 y = y = y[0]=3 y[1]=2 c1 y = cos() = y∙c1 |y||c1| A1 = |y|cos() = y∙c1 = j y[j]c1[j] This is the correlation between y and cf 31
Fourier Transformation Rectangular vs Polar vs Complex The contribution of a given frequency f can be expressed in different ways. Rectangular Notation j There are two basis (hence two coefficients) for each frequency f. n/2 frequencies are considered There are n values in the Time Domain and as needed n in the Frequency Domain. Note 2(n/2 +1)=n+2, because s1 = sn/2 = 0 32
Fourier Transformation Rectangular vs Polar vs Complex The contribution of a given frequency f can be expressed in different ways. Rectangular Notation j Any function y in the Time Domain can be expressed as a linear combination of these n basis functions. Inverse Fourier Transform: 33
Fourier Transformation Rectangular vs Polar vs Complex The contribution of a given frequency f can be expressed in different ways. Rectangular Notation Polar Notation The contribution from frequency f is: As needed, there are still n in the Frequency Domain. 34
Fourier Transformation Rectangular vs Polar vs Complex The contribution of a given frequency f can be expressed in different ways. Complex Notation eθi = cosθ + isinθ i -i -1 1 θ f’(0) = g’(0) Proof: f(θ) = g(θ) f(0) = g(0) f’’(θ) = -f(θ) g’’(θ) = -g(θ) 35
Fourier Transformation Rectangular vs Polar vs Complex The contribution of a given frequency f can be expressed in different ways. There are two basis (hence two coefficients) for each frequency f[0,n/2]. Rectangular Notation Complex Notation There is one basis (hence one coefficients) for each frequency f[0,n-1]. There are n values in the Time Domain. Hence n are needed in the Frequency Domain. eθi = cosθ + isinθ 36
Fourier Transformation Rectangular vs Polar vs Complex The contribution of a given frequency f can be expressed in different ways. Rectangular Notation Complex Notation Any function y in the Time Domain can be expressed as a linear combination of these n basis functions. Inverse Fourier Transform: 37
Fourier Transformation Rectangular vs Polar vs Complex The contribution of a given frequency f can be expressed in different ways. Rectangular Notation Complex Notation Surprisingly the two techniques give the same numbers when the time domain values are real. 38
Fourier Transformation Continuous Functions A continuous periodic function t time y(t) -½T 0 ½T y(t) is an function from real time to a real values. The basis functions are the same. Except they also need to be continuous. Stretched so that a period of the basis lines up with a period of the signal. If this is the dominate musical note of frequency = 2/T, then all the other basis functions are its harmonics frequencies: Frequency: Note on the Piano: , 2, 3, 4, 5, 6, ... C G E 39
Fourier Transformation Continuous Functions A continuous periodic function t time y(t) -½T 0 ½T y(t) is an function from real time to a real values. The third basis, as before will cycle 3 times. Before the signal had n values so we needed n bases functions. Now we need n. Continue for ever. 40
Fourier Transformation Continuous Functions A continuous periodic function t time y(t) -½T 0 ½T y(t) is an function from real time to a real values. Both the signal and the basis functions repeat every T units. Discrete: (previous defn) Continuous: (new defn) Called a Fourier Sequence. 41
Fourier Transformation Continuous Functions A continuous periodic function t time y(t) -½T 0 ½T y(t) is an function from real time to a real values. Both the signal and the basis functions repeat every T units. Discrete: (previous defn) Continuous: (new defn) Called a Fourier Sequence. 42
Fourier Transformation Continuous Functions -½T 0 ½T y(t) A continuous non-periodic function time t We want T. 43
Fourier Transformation Continuous Functions -½T 0 ½T y(t) A continuous non-periodic function time t We want T. Continuous Periodic: (previous defn) Continuous Non-Periodic: (new defn) Continuous Non-Periodic Complex: (new defn) 44
Fourier Transformation Continuous Functions These four definitions of Fourier Transformations look very different, but we will argue that they give pretty much the same answer. Discrete: Continuous Periodic: Continuous Non-Periodic: Continuous Non-Periodic Complex: 45
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() = (-,) A single square y(t) -T/2 T/2 1 -1 Continuous Non-Periodic: The th real Fourier coefficient is the correlation between 1 and cos(t).
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() f = 0, 1, 2,3, …, A single square y(t) -T/2 T/2 1 -1 Continuous Periodic: This technique can’t be applied because the signal is not periodic.
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() f = 0, 1, 2,3, …, YRe[f] A square wave y(t) -T/2 T/2 1 -1 Continuous Periodic: is a real number f is an integer
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() f = 0, 1, 2,3, …,n YRe[f] A square wave y(t) -T/2 T/2 1 -1 Discrete: is a real number f is an integer
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() = (-,) A square wave y(t) -T/2 T/2 1 -1 Continuous Non-Periodic: The Y[f] discrete signal at f is replaced at =2f/T with a continuous delta with value Y[f]/dt = of width dt for a total area of Y[f]. else zero. is a real number f is an integer
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() = (-,) A square wave y(t) -T/2 T/2 1 -1 The Y[f] discrete signal at f is replaced at =2f/T with a continuous delta with value Y[f]/dt = of width dt for a total area of Y[f]. else zero.
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() = (-,) A square wave y(t) -T/2 T/2 1 -1 The Y[f] discrete signal at f is replaced at =2f/T with a continuous delta with value Y[f]/dt = of width dt for a total area of Y[f]. else zero.
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() = (-,) A square wave y(t) -T/2 T/2 1 -1 Continuous Non-Periodic: The Y[f] discrete signal at f is replaced at =2f/T with a continuous delta with value Y[f]/dt = of width dt for a total area of Y[f]. else zero.
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y Sinc function YRe() = (-,) A square wave y(t) -T/2 T/2 1 -1 Continuous Non-Periodic: The Y[f] discrete signal at f is replaced at =2f/T with a continuous delta with value Y[f]/dt = of width dt for a total area of Y[f]. else zero. Continuous Periodic: The two techniques are equivalent!
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y How do you get these corner? Zero function 2 YIm() - A square wave y(t) -T/2 T/2 1 -1 Continuous Non-Periodic:
Fourier Transformation Continuous Functions YIm() Time Domain y Frequency Domain Y y(t) = t Continuous Non-Periodic:
Fourier Transformation Continuous Functions YIm() Time Domain y Frequency Domain Y y(t) = t Continuous Non-Periodic:
Fourier Transformation Continuous Functions Time Domain y Frequency Domain Y y(t) = t y(x) 2 sin(x) - sin(2x) + 2/3 sin(3x) Continuous Non-Periodic: Alternatively only do the FT on part of the signal.
Fourier Transformation Radio Signals Time Domain y Frequency Domain Y Sound Signal ie how far out is the speaker drum at each point in time. Sound is low frequency High frequencies filtered out. This constant function in the time domain translates to a delta at f=0 in the frequency domain Note signal is not centered at zero, but has a constant added to it.
Fourier Transformation Radio Signals Time Domain y Frequency Domain Y Radio Carrier Signal ie A wave of magnetic field that can travel far. One high frequency signal
Fourier Transformation Radio Signals Time Domain y Frequency Domain Y Carrier signal Audio Signal (shifted) Audio Signal (shifted &flipped) Modulation: Their product y(i) = y1(i) y2(i)
Fourier Transformation Radio Signals Time Domain y Frequency Domain Y Because FT is periodic, a flipped version is in the “negative” frequencies. Carrier signal Audio Signal (shifted) Negative copy (shifted) Modulation: Their product y(i) = y1(i) y2(i)
Fourier Transformation * Convolution Time Domain y Frequency Domain Y Modulation: Their product Y = Y1*Y2 Convolution y = y1 y2
Fourier Transformation * Convolution “Proof”: Complex Notation y = y1 y2 Y = Y1*Y2 64
Fourier Transformation * Convolution “Proof”: y = y1 y2 Y = Y1*Y2
Fourier Transformation * Convolution “Proof”: y = y1 y2 Y = Y1*Y2
Fourier Transformation Linear Filter Filters also use of Convolution: Because x[] y[] Independent of time: x[_+dt] y[_+dt] Linear: c1∙x1[]+c2∙x2[] c1∙y1[]+y2∙x2[] This system takes in a signal and outputs transformed signal.
Fourier Transformation h[] = Linear Filter Filters also use Convolution: h[] = In order understand this transformation, we put in a single pulse. [] = h[] This response h[] identifies the system.
Fourier Transformation h[] = Linear Filter Feed in any signal x[] = Sum of contributions from each separate pulse. h[] Sum
Fourier Transformation h[] = Fourier Transformation Linear Filter Where does y[k] come from? x[] = h[]
Fourier Transformation Linear Filter x[] = Input x[]*h[] = Output h[] = Impulse Response This takes O(nm) time. Too much! How can we do it faster?
Fourier Transformation * Convolution Time Domain y Frequency Domain Y y = y1 y2 Product Y = Y1*Y2 Convolution Duality of FT: If Y=FT(y), then y=FT(Y) Y = X H Product y = x*h Convolution
Fourier Transformation * Convolution Time Domain y Frequency Domain Y x[] = Input h[] = Impulse Response Oops Fourier Transform takes O(n2) time. X[] H[] Fast Fourier Transform takes O(nlogn) time! X[]H[] x[]*h[] = Output Multiplication takes O(n) time. Y = X H Product y = x*h Convolution
Fourier Transformation * Convolution Time Domain y Frequency Domain Y Impulse Response h[] = H[] = Impulse Response x[] = Input X[] x[]*h[] = Output X[]H[] Multiplying zeros low and high frequencies in input. Filters out low and high frequencies in input. Not clear what system does to input Y = X H Product y = x*h Convolution
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y Theorem: abcdefgh y[0]=a, y[1]=b, y[2]=c, y[3]=d, … ABCDEFGH Fast Fourier Transform takes O(nlogn) time!
Fourier Transformation FFT nlogn Time The Fast Fourier Transform (FFT) is a very efficient algorithm for performing a discrete Fourier transform FFT principle first used by Gauss in 18?? (But was not interesting without computers) FFT algorithm published by Cooley & Tukey in 1965 In 1969, the 2048 point analysis of a seismic trace took 13 ½ hours. Using the FFT, the same task on the same machine took 2.4 seconds!
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y Lemma 1: y = abcd Y = ABCD If y’ = a0b0c0d0 Y’ = ½∙ABCDABCD then Proof 1:
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y Lemma 1: y = abcd Y = ABCD If y’ = a0b0c0d0 Y’ = ½∙ABCDABCD then “Proof” 2:
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y 16 32 48 64 Lemma 1: Impulse at f = n/2 16 32 48 64 Impulse at f = 0 ½c16+½c16= 16 32 48 64 ½I16+½I16= Impulse at f = 0 & n/2 ½ 1 = = 10101010 ½∙10001000
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y Lemma 1: y = abcd Y = ABCD If Interpolate intermediate values Shouldn’t change frequencies. aabbccdd ABCD = * = 10101010 ½∙10001000 y’ = a0b0c0d0 Y’ = ½∙ABCDABCD
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y Lemma 2: y = abcd Y = ABCD If y’ = a0b0c0d0 Y’ = ½ s1 ∙ABCDABCD then Proof: a0b0c0d0 ½∙ABCDABCD By Lemma 1 * Impulse at f = 1 ? = 1 -1 s1 ? = 0a0b0c0d ½s1∙ABCDABCD The value Y[j] is scaled by B1[j].
Fast Fourier Transformation Divide & Conquer - Friends - Recursion. Trust your friends to solve any subinstance: as long as smaller and is an instance to the same problem. My instance My friend’s Instance My friend’s Instance
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y FFT Alg: y = abcdefgh FFT Input: Split into even & odd indexes yeven = aceg yodd = bdfh Yeven = ACEG Yodd = BDFH Recurse Twice on input of half the size. By Lemmas ½∙ACEGACEG ½s1∙BDFHBDFH Copy 0b0d0f0h a0c0e0g0
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y FFT Alg: y = abcdefgh FFT Input: + = + = ½∙ACEGACEG +½s1∙BDFHBDFH 0b0d0f0h a0c0e0g0 ½∙ACEGACEG ½s1∙BDFHBDFH
Fourier Transformation FFT nlogn Time Time Domain y Frequency Domain Y FFT Alg: T(n) = 2 T(n/2) + O(n) Time to solve FFT on input of size n. Two recursive friends Time to solve FFT on input of size n/2. Time spent by top stack frame. = O(nlog(n)) Same as merge sort.
Fourier Transformation FFT Code Time Domain y Frequency Domain Y Y = ABCDEFGH y = abcdefgh FFT Input: FFT Alg: Algorithm FFT(y, , n): Input: y = [y[0],y[1],y[2],…, y[n-1]] (Time Domain) = e2i 1/n (nth root of unity) n = # of samples (2r) Output: Y = [Y[0],Y[1],Y[2],…, Y[n-1]] (Frequency Domain)
Fourier Transformation FFT Code Time Domain y Frequency Domain Y FFT Alg: y = abcdefgh FFT Input: Split into even & odd indexes yeven = aceg yodd = bdfh Yeven = ACEG Yodd = BDFH Recurse Twice on input of half the size. % Separate even and odd indices yeven = [y[0],y[2],y[4],…, y[n-2]] yodd = [y[1],y[3],y[5],…, y[n-1]] % Recurse Yeven =FFT(yeven, n/2) Yodd =FFT(yodd , n/2)
Fourier Transformation FFT Code Time Domain y Frequency Domain Y FFT Alg: y = abcdefgh FFT Input: + = ½∙ACEGACEG +½s1∙BDFHBDFH Yeven = ACEG ½∙ACEGACEG ½s1∙BDFHBDFH Yodd = BDFH 1 -1 s1 The value Y[j] is scaled by B1[j]. %Combining For i = 0 to n/2-1 Y[i] = Yeven[i] + i ∙Yodd[i] Y[i+n/2] = Yeven[i] + i+n/2 ∙Yodd[i] Return(Y)
Fourier Transformation FFT Code Time Domain y Frequency Domain Y FFT Alg: y = abcdefgh FFT Input: + = ½∙ACEGACEG +½s1∙BDFHBDFH Yeven = ACEG ½∙ACEGACEG ½s1∙BDFHBDFH Yodd = BDFH 1 -1 s1 The value Y[j] is scaled by B1[j]. % Recurse Yeven =FFT(yeven, 2, n/2) Yodd =FFT(yodd , 2, n/2) Algorithm FFT(y, , n): Input: = e2i 1/n = e2i 1/n (nth root of unity) 2 = e2i 2/n (n/2 th root of unity)
Fourier Transformation FFT Code Algorithm FFT(y, , n): Input: y = [y[0],y[1],y[2],…, y[n-1]] (Time Domain) = e2i 1/n (nth root of unity) n = # of samples (2r) Output: Y = [Y[0],Y[1],Y[2],…, Y[n-1]] (Frequency Domain) % Separate even and odd indices yeven = [y[0],y[2],y[4],…, y[n-2]] yodd = [y[1],y[3],y[5],…, y[n-1]] % Recurse Yeven =FFT(yeven, 2, n/2) (2 = e2i 2/n ) Yodd =FFT(yodd , 2, n/2) %Combining For i = 0 to n/2-1 Y[i] = Yeven[i] + i ∙Yodd[i] Y[i+n/2] = Yeven[i] + i+n/2 ∙Yodd[i] Return(Y)
Fast Fourier Transformation
Fourier Transformation FFT Butterfly Time Domain y Frequency Domain Y FFT Alg: y = abcdefgh FFT Input: + = + = ½∙ACEGACEG +½s1∙BDFHBDFH 0b0d0f0h a0c0e0g0 ½∙ACEGACEG ½s1∙BDFHBDFH
Fourier Transformation FFT Butterfly O(log(n)) levels
Fourier Transformation JPG Image Compression JPEG (Image Compression) JPEG is two dimensional Fourier Transform exactly as done before.
Fourier Transformation JPG Image Compression Each 88 block of values from the image is encoded separately.
Fourier Transformation JPG Image Compression Each basis function has a coefficient, giving the contribution of this basis function to the image. Each 88 block of values from the image is encoded separately. It is decomposed as a linear combination of basis functions.
Fourier Transformation JPG Image Compression Because the image domain is indexed by x,y [0,7], so is each basis function. Each 88 block of values from the image is encoded separately. It is decomposed as a linear combination of basis functions.
Fourier Transformation JPG Image Compression They are indexed similarly by u,v [0,7] Each 88 block of values from the image is encoded separately. It is decomposed as a linear combination of basis functions. Because the image domain is specified by 64 values, the frequency domain must also be specified by 64 values And hence 64 basis functions are needed.
Fourier Transformation JPG Image Compression Each 88 block of values from the image is encoded separately. It is decomposed as a linear combination of basis functions. Each of the 64 basis functions is a two dimensional cosine.
Fourier Transformation JPG Image Compression The first basis is constant. Its coefficient gives the average value in within block. Because many images have large blocks of the same colour, this one coefficient gives much of the key information!
Fourier Transformation JPG Image Compression The second basis “slopes” left to right Its (pos or neg) coefficient gives whether left to right the value tends to increase or decrease.
Fourier Transformation JPG Image Compression The second basis “slopes” left to right Because many images have have a gradual change in colour, this one coefficient gives more key information!
Fourier Transformation JPG Image Compression A similar basis for top to bottom.
Fourier Transformation JPG Image Compression The <0,2> basis is Its coefficient gives whether the value tends to be smaller in the middle. This helps display the horizontal lines in images
Fourier Transformation JPG Image Compression As seen, the low frequency components of a signal are more important. Removing 90% of the bits from the high frequency components might remove, only 5% of the encoded information.
Fourier Transformation Polynomial Basis A continuous periodic function t time y(t) A discrete periodic function Time Domain j[0,n-1] y[j] The function y is expressed as a linear combination of the basis functions cf and sf. What we did before y[] YRe[] YIm[] Y[] 106
Fourier Transformation Polynomial Basis Instead of using sine and cosines as the basis, 107
Fourier Transformation Polynomial Basis Instead of using sine and cosines as the basis, We use polynomials. 108
Fourier Transformation Polynomial Basis Change of Basis: T([a1,a2,…,ad]) = [A1,A2,…,Ad] Changes the basis used to describe an object. The Time basis of a vector space: A tuple <w1,w2,…,wd> of basis objects Linearly independent Spans the space uniquely "v $[a1,a2,…,ad], v = a1w1+a2w2 +… + adwd The Polynomial basis of a vector space: A tuple <W1,W2,…,Wd> of basis objects Linearly independent Spans the space uniquely "v $[A1,A2,…,Ad], v = A1W1+A2W2 +… + AdWd 109
Fourier Transformation Polynomial Basis Change of Basis: T([a1,a2,…,ad]) = [A1,A2,…,Ad] Changes the basis used to describe an object. "v $[a1,a2,…,ad], v = a1w1 +a2w2 +… + adwd "v $[A1,A2,…,Ad], v = A1W1+A2W2 +… + AdWd =[w1,w2] =[ , ] Standard Basis New Basis =[W1,W2] =[ , ] W1[1] W1[2] W2[1] W2[2] v = v = [a1,a2] = [3,2] [A1,A2] = [11/5,32/5] [ ][ ] =[ ] W1[1] W1[2] W2[1] W2[2] a1 a2 A1 A2 110
Fourier Transformation Polynomial Basis Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a1,a2,…,an-1] Changes the basis used to describe an object. f = a0∙B0+a1∙B1+a2∙B2 +… + an-1∙Bn-1 f(x) = a0 +a1x +a2x2 + …+ an-1xn-1 a0 =11/5 a1 =32/5 Polynomial Basis =[ , ] =[B0,B1,..] [a1,a2,…,an-1] = coefficients of the polynomial f = A discrete function x f(x) 111
Fourier Transformation Polynomial Basis Have you seen Taylor Expansions of a Function? They show that functions f(x) can be expressed by specifying the coefficients of the polynomial. F(x) = a0+a1x +a2x2 +a3x3 + … Eg: f(x) = 1/(1-x) F(x) = 1+x +x2 +x3 + …
Fourier Transformation Polynomial Basis Have you seen Taylor Expansions of a Function? They show that functions f(x) can be expressed by specifying the coefficients of the polynomial. F(x) = a0+a1x +a2x2 +a3x3 + …
Fourier Transformation Polynomial Basis Have you seen Taylor Expansions of a Function? They show that functions f(x) can be expressed by specifying the coefficients of the polynomial. F(x) = a0+a1x +a2x2 +a3x3 + …
Fourier Transformation Polynomial Basis Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a1,a2,…,an-1] Changes the basis used to describe an object. "y $[y[0],y[1],…,y[n-1]], y = y[0]I0 +y[1]I1 +… + y[n-1]In-1 The time basis x Ij[x] zero one j Time Basis =[ , ] =[?,?] =[I0,I1,…] f = y[0]=3 y[1]=2 y[j] = the value f(j) of the function at x=j. the value f(j) of the function at x=j. A discrete function x f(x) 115
Fourier Transformation Polynomial Basis Instead of evaluating the polynomials on j = 0, 1, 2,…, n-1 y[j] = the value f(j) of the function at x=j. We evaluate them on x0, x1, x2, …, xn-1 the value f(j) of the function at x=j. These xj are special fixed values. Later we set xj = e2i j/n 116
Fourier Transformation Polynomial Basis Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a1,a2,…,an-1] Changes the basis used to describe an object. "y $[y[0],y[1],…,y[n-1]], y = y[0]I0 +y[1]I1 +… + y[n-1]In-1 The time basis x Ij[x] zero one xj Time Basis =[ , ] =[I0,I1,…] =[?,?] f = y[0]=3 y[1]=2 y[j] = the value f(xj) of the function at x=j. the value f(j) of the function at x=j. A discrete function x f(x) x0 x1 x2 x3 x4 … xn-1 117
Fourier Transformation Evaluating & Interpolating A Fourier Transform is a change in basis. It changes the representation of a function from the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1 x to the value f(xi) at key values xi. x0 x1 x2 x3 x4 … xn-1 y0 y1 y2 y3 y4 … yn-1 yi = f(xi) This amounts to evaluating f at these points. 118
Fourier Transformation Evaluating & Interpolating A Fourier Transform is a change in basis. It changes the representation of a function from the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1 (x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = (x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1 (xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1 (x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1 yi = f(xi) Vandermonde matrix Invertible if xi distinct. This amounts to evaluating f at these points. 119
Fourier Transformation Evaluating & Interpolating An Inverse Fourier Transform is the reverse. It changes the representation of a function x from the value f(xi) at key values xi. x0 x1 x2 x3 x4 … xn-1 y0 y1 y2 y3 y4 … yn-1 yi = f(xi) to the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1 This amounts to interpolating these points. 120
Fourier Transformation Evaluating & Interpolating An Inverse Fourier Transform is the reverse. It changes the representation of a function Given a set of n points in the plane with distinct x-coordinates, there is exactly one (n-1)-degree polynomial going through all these points. to the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1 This amounts to interpolating these points. 121
Fourier Transformation Evaluating & Interpolating My output: f(x) = a0+a1x +a2x2 + … + an-1xn-1 My input: (a0,a1,a2,…,an-1) & x 1+2+3+..+n-1 multiplications? f(x) = a0+x(a1+x(a2 + … + x(an-1) … )) n multiplications and n additions. Given n distinct values (x0,x1,x2,…,xn-1). Can they all be done in less than n×O(n) time?
Fourier Transformation Evaluating & Interpolating My output: f(x) = a0+a1x +a2x2 + … + an-1xn-1 My input: (a0,a1,a2,…,an-1) & x (x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = (x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1 (xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1 (x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1 yi = f(xi) I hope so Given n distinct values (x0,x1,x2,…,xn-1) Can they all be done in less than n×O(n) time?
Fast Fourier Transformation FFT nlogn Time The Fast Fourier Transform (FFT) is a very efficient algorithm for performing a discrete Fourier transform FFT principle first used by Gauss in 18?? (But was not interesting without computers) FFT algorithm published by Cooley & Tukey in 1965 In 1969, the 2048 point analysis of a seismic trace took 13 ½ hours. Using the FFT, the same task on the same machine took 2.4 seconds!
Fast Fourier Transformation FFT nlogn Time Divide & Conquer - Friends - Recursion. Trust your friends to solve any subinstance: as long as smaller and is an instance to the same problem. My instance My friend’s Instance My friend’s Instance My friend’s Instance
Fast Fourier Transformation FFT nlogn Time My output: f(x) = a0+a1x +a2x2 + … + an-1xn-1 My input: (start with one x) (a0,a1,a2,…,an-1) & x 1st friend’s input? feven: (a0,a2,a4,…,an-2) & ? 2nd friend’s input? fodd: (a1,a3,a5,…,an-1) & ? feven(z) = a0+a2z+a4z2+a6z3+ … + an-2zn/2-1 f(x) = a0+a1x +a2x2 + … + an-1xn-1 = a0+a2x2+a4x4 + … + an-2xn-2 + a1x+a3x3+a5x5 + … + an-1xn-1 = a0+a2x2+a4x4 + … + an-2xn-2 + x( a1+a3x2+a5x4 + … + an-1xn-2 ) = feven(x2) + x( fodd(x2) ) f(x) = feven(x2) + x fodd(x2)
Fast Fourier Transformation FFT nlogn Time My output: f(x) = a0+a1x +a2x2 + … + an-1xn-1 My input: (start with one x) (a0,a1,a2,…,an-1) & x 1st friend’s input? feven: (a0,a2,a4,…,an-2) & x2 2nd friend’s input? fodd: (a1,a3,a5,…,an-1) & x2 2nd friend’s output: yodd = fodd(x2) 1st friend’s output: yeven = feven(x2) My output: f(x) = yeven + x yodd T(n) = 2 T(n/2) + O(1) = O(n) Ok. So it takes O(n) time to evaluate. f(x) = feven(x2) + x fodd(x2)
Fast Fourier Transformation FFT nlogn Time My input: (a0,a1,a2,…,an-1) (x0,x1,x2,…,xn-1) My output: (y0,y1,y2,…,yn-1) yi = f(xi) 1st friend’s input? feven: (a0,a2,a4,…,an-2) (x02,x12,x22,…,xn-12) 2nd friend’s input? fodd: (a1,a3,a5,…,an-1) (x02,x12,x22,…,xn-12) 2nd friend’s output: i y<odd,i> = fodd(xi2) 1st friend’s output: i y<even,i> = feven(xi2) My output: i f(xi) = y<even,i> + xi y<odd,i> T(n) = 2 T(n/2) + O(n) = O(n log n) Wow! That was easy.
Fast Fourier Transformation FFT nlogn Time My input: (a0,a1,a2,…,an-1) (x0,x1,x2,…,xn-1) n coefficients n values of x n/2 coefficients 1st friend’s input? feven: (a0,a2,a4,…,an-2) (x02,x12,x22,…,xn-12) Does not meet precondition! Oops
Fast Fourier Transformation FFT nlogn Time My input: (a0,a1,a2,…,an-1) (x0,x1,x2,…,xn-1) My output: (y0,y1,y2,…,yn-1) yi = f(xi) 1st friend’s input? feven: (a0,a2,a4,…,an-2) (x02,x12,x22,…,xn/2-12) 2nd friend’s input? fodd: (a1,a3,a5,…,an-1) (x02,x12,x22,…,xn/2-12) 3rd friend’s input? feven: (a0,a2,a4,…,an-2) (xn/22,xn/2+12,…,xn-12) 4th friend’s input? fodd: (a1,a3,a5,…,an-1) (xn/22,xn/2+12,…,xn-12) That’s no good. My output: i f(xi) = y<even,i> + xi y<odd,i> T(n) = 4 T(n/2) + O(n) = O(n2)
Fast Fourier Transformation Roots of Unity The values (x0,x1,x2,…,xn-1) are said to be special if: There are n distinct values. When you square each of them: The set collapses to n/2 distinct values. Eg: …, -3, -2, -1, 1, 2, 3, … square each of them …, 9, 4, 1, 1, 4, 9, … collapse the set 1, 4, 9, … half as many elements.
Fast Fourier Transformation Roots of Unity My input: (a0,a1,a2,…,an-1) Special (x0,x1,x2,…,xn-1) My output: (y0,y1,y2,…,yn-1) yi = f(xi) -3 3 1st friend’s input? feven: (a0,a2,a4,…,an-2) (x02,x12,x22,…,xn-12) n/2 distinct values 2nd friend’s input? fodd: (a1,a3,a5,…,an-1) (x02,x12,x22,…,xn-12) n/2 distinct values 9 2nd friend’s output: i y<odd,i> = fodd(xi2) 1st friend’s output: i y<even,i> = feven(xi2) feven(9) fodd(9) My output: i f(xi) = y<even,i> + xi y<odd,i> T(n) = 2 T(n/2) + O(n) = O(n log n) That’s better 3 f(3) -3 f(-3) feven(9) fodd(9)
Fast Fourier Transformation Roots of Unity My input: (a0,a1,a2,…,an-1) Special (x0,x1,x2,…,xn-1) My output: (y0,y1,y2,…,yn-1) yi = f(xi) To meet precondition these also need to be special 1st friend’s input? feven: (a0,a2,a4,…,an-2) (x02,x12,x22,…,xn-12) n/2 distinct values 2nd friend’s input? fodd: (a1,a3,a5,…,an-1) (x02,x12,x22,…,xn-12) n/2 distinct values
Fast Fourier Transformation Roots of Unity The values (x0,x1,x2,…,xn-1) are said to be special if: There are n distinct values. When you square each of them: The set collapses to n/2 distinct values. Which are also special Eg: …, -3, -2, -1, 1, 2, 3, … square each of them …, 9, 4, 1, 1, 4, 9, … collapse the set 1, 4, 9, … square each of them 2, 16, 81, … But these are not special.
Fast Fourier Transformation Roots of Unity The values (x0,x1,x2,…,xn-1) are said to be special if: There are n distinct values. When you square each of them: The set collapses to n/2 distinct values. Which are also special Eg: -i, -1, 1, i square each of them -1, 1, 1, -1 = -i,-1, 1, i are said to be 4th roots of unity Because 4 = 1 collapse the set -1, 1 square each of them 1, 1 collapse the set 1
Fast Fourier Transformation Roots of Unity is said to be an nth root of unity (in a field) if n = 1 (There should be n solutions of this polynomial) Fermat’s Little Theorem: b≠0 bp-1 =mod p 1 says every nonzero element is an nth root of unity when n=p-1.
Fast Fourier Transformation Roots of Unity is said to be an nth root of unity (in a field) if n = 1 is said to be a generator of the field if the numbers 1,,2, …,n-1 are all distinct 1,,2, …,n-1 are then special (when n is even) 1st half 2nd half n/2+n/2-1,…,n/2+3,n/2+2,n/2+1,n/2+0 ,0,1,2,3,…,n/2-1 square each of them n+n-2, …, n+6, n+4, n+2, n+0, 0,2,4,6, …,n-2 use n = 1 n-2, …, 6, 4, 2, 0, 0,2,4, 6, …,n-2 collapse the set 0,2,4, 6, …,n-2 We need these to be n/2 special values.
Fast Fourier Transformation Roots of Unity 16th roots of unity (n/4)2 = n/2 = -1 i These could be Z mod 17 or complex numbers × 164 165 163 166 162 rr 167 161 (n/2)2 = 1 r θ 168 -1 = 160 = 1616 = 1 1615 169 1614 1610 1613 1611 1612 reθi = rcosθ + irsinθ reθi × seαi = (rs)e(θ+α)i (3n/4)2 = n/2 = -1 -i
Fast Fourier Transformation Roots of Unity Goal: Proof f(θ) = g(θ) f(0) = g(0) f’(0) = g’(0) f’’(θ) = -f(θ) g’’(θ) = -g(θ) Proof by induction (over the reals) that f(θ) = g(θ) f(θ) g(θ) For this θ, f(θ) = g(θ) and f’(θ) = g’(θ) For next θ+, f(θ+) = g(θ+) f’’(θ) = -f(θ) = -g(θ) =g’’(θ) For next θ+, f’(θ+) = g’(θ+)
Fast Fourier Transformation Roots of Unity g(θ) = rcosθ + irsinθ f(θ) = reθi Goal: Proof f(θ) = g(θ) g(0) = rcos0 + irsin0 = r f(0) = re0i = r f(0) = g(0) g’(θ) = -rsinθ + ircosθ f’(θ) = ireθi g’(0) = -rsin0 + ircos0 = ir f’(0) = ire0i = ir f’(0) = g’(0) g’’(θ) = -rcosθ - rsinθ f’’(θ) = -reθi = -f(θ) = -g(θ)
Fast Fourier Transformation Roots of Unity 16th roots of unity square each of them and collapse 164 165 163 166 162 167 161 168 160 1615 169 1614 1610 1613 1611 1612
Fast Fourier Transformation Roots of Unity 16th roots of unity square each of them and collapse 164 Are these special? 166 162 168 160 1614 1610 1612
Fast Fourier Transformation Roots of Unity 8th roots of unity square each of them and collapse 82 Are these special? 83 81 84 80 87 85 86
Fast Fourier Transformation Roots of Unity 4th roots of unity square each of them and collapse 41 Are these special? 42 40 43
Fast Fourier Transformation Roots of Unity 2th roots of unity square each of them and collapse Are these special? 21 20 = 1
Fast Fourier Transformation Roots of Unity My input: (a0,a1,a2,…,an-1) (nth roots of unity ni) My output: (y0,y1,y2,…,yn-1) yi = f(ni) 1st friend’s input? feven: (a0,a2,a4,…,an-2) (n/2th roots of unity n/2i) 2nd friend’s input? fodd: (a1,a3,a5,…,an-1) (n/2th roots of unity n/2i) 2nd friend’s output: i y<odd,i> = fodd(n/2i) 1st friend’s output: i y<even,i> = feven(n/2i) My output: i f(xi) = y<even,i> + xi y<odd,i> T(n) = 2 T(n/2) + O(n) = O(n log n) Excellent
Fourier Transformation FFT Code Algorithm FFT(y, , n): Input: y = [a0,a1,a2,…,an-1] (Time Domain) = e2i 1/n (nth root of unity) n = # of samples (2r) Output: Y = [y0,y1,y2,…,yn-1] (Frequency Domain) % Separate even and odd indices aeven = [a0,a2,a4,…,an-2] aodd = [a1,a1,a5,…,an-1] % Recurse yeven =FFT(aeven, 2, n/2) (2 = e2i 2/n ) yodd =FFT(aodd , 2, n/2) %Combining For i = 0 to n/2-1 y[i] = yeven[i] + i ∙yodd[i] y[i+n/2] = yeven[i] + i+n/2 ∙yodd[i] Return(Y)
Fast Fourier Transformation FFT Code
Fast Fourier Transformation FFT Butterfly
Fast Fourier Transformation Inverse FFT A inverse Fourier transform changes the representation of a function x0 x1 x2 x3 x4 … xn-1 y0 y1 y2 y3 y4 … yn-1 yi = f(xi) from the value f(xi) at key values xi. to the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1 This amounts to interpolating these points. 150
Fast Fourier Transformation Inverse FFT A Fourier Transform is a change in basis. It changes the representation of a function from the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1 (x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = (x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1 (xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1 (x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1 (xi)j yi = f(xi) This amounts to evaluating f at these points. 151
xi = i (x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0 a1 a2 a3 … an-1 y0 y1 y2 Inverse FFT (x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = (x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1 (xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1 (x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1 (xi)j xi = i (0)0 (0)1 (0)2 (0)3 … (0)n-1 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = (1)0 (1)1 (1)2 (1)3 … (1)n-1 (2)0 (2)1 (2)2 (2)3 … (2)n-1 (3)0 (3)1 (3)2 (3)3 … (3)n-1 (n-1)0 (n-1)1 (n-1)2 (n-1)3 … (n-1)n-1 (i)j 152
(i)j = ij Vandermonde matrix (0)0 (0)1 (0)2 (0)3 … (0)n-1 a0 a1 Inverse FFT (0)0 (0)1 (0)2 (0)3 … (0)n-1 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = (1)0 (1)1 (1)2 (1)3 … (1)n-1 (2)0 (2)1 (2)2 (2)3 … (2)n-1 (3)0 (3)1 (3)2 (3)3 … (3)n-1 (n-1)0 (n-1)1 (n-1)2 (n-1)3 … (n-1)n-1 (i)j (i)j = ij 0 0 0 0 … 0 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = 0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3 0 n-1 2n-2 3n-3 … (n-1)(n-1) ij Vandermonde matrix 153
V a = y a = V-1 y V-1 = 1/n V -1 Vandermonde matrix Inverse FFT 0 0 0 0 … 0 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = 0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3 0 n-1 2n-2 3n-3 … (n-1)(n-1) ij Vandermonde matrix V a = y a = V-1 y V-1 = 1/n V -1 154
1/n Vandermonde matrix V-1 = 1/n V 0 0 0 0 … 0 a0 a1 a2 a3 … Inverse FFT 0 0 0 0 … 0 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = 0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3 0 n-1 2n-2 3n-3 … (n-1)(n-1) ij Vandermonde matrix V-1 = 1/n V -1 0 0 0 0 … 0 y0 y1 y2 y3 … yn-1 a0 a1 a2 a3 an-1 = 0 -1 -2 -3 … -(n-1) 0 -2 -4 -6 … -(2n-2) 0 -3 -6 -9 … -3(n-3) 0 -(n-1) -(2n-2) -(3n-3) … -(n-1)(n-1) -ij 1/n 155
Fast Fourier Transformation If w is an nth root of unity and a generator of the field 161 160 162 163 164 165 166 167 168 169 1610 1611 1612 1613 1614 1615 = 1616 = 1 -1 = rr
Fast Fourier Transformation Inverse FFT The inverse w-1 of w is then w-1=wn-1 And is also an nth root of unity and a generator of the field Proof: ww-1 = wwn-1 = wn=1 161 160 162 163 164 165 166 167 168 169 1610 1611 1612 1613 1614 1615 = 1616 = 1 -1 = rr
Fast Fourier Transformation Inverse FFT Cancellation Property: 161 160 162 163 164 165 166 167 168 169 1610 1611 1612 1613 1614 1615 = 1616 = 1 -1 = rr
1/n Vandermonde matrix V-1 = 1/n V 0 0 0 0 … 0 a0 a1 a2 a3 … Inverse FFT Vandermonde matrix 0 0 0 0 … 0 a0 a1 a2 a3 … an-1 y0 y1 y2 y3 yn-1 = 0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3 0 n-1 2n-2 3n-3 … (n-1)(n-1) ij V-1 = 1/n V -1 0 0 0 0 … 0 0 -1 -2 -3 … -(n-1) 0 -2 -4 -6 … -(2n-2) 0 -3 -6 -9 … -3(n-3) 0 -(n-1) -(2n-2) -(3n-3) … -(n-1)(n-1) -ij 1/n 159
Fast Fourier Transformation Inverse FFT The FFT and inverse FFT really are inverse operations Proof: Let A=D -1D. We want to show that A=I, where If i=j, then If i and j are different, then
Fast Fourier Transformation Inverse FFT The FFT and inverse FFT can use the same hardware FFT Input: <1, , a0,a1,a2,…,an-1 > Output: <y0,y1,y2,…,yn-1 > Inverse FFT Input: <1/n, -1, y0,y1,y2,…,yn-1 > Output: <a0,a1,a2,…,an-1 >
Polynomial Multiplication f(x) = a0+a1x +a2x2 + … + an-1xn-1 g(x) = b0+b1x +b2x2 + … + bn-1xn-1 [f×g](x) = c0+c1x +c2x2 + … +c2n-2x2n-2 x5 coefficient: c5= a0×b5+a1×b4 + a2×b3 + … + a5×b0 Convolution Too much O(n2) Time =
Polynomial Multiplication Pad with n 0's [ a , 1 2 ,..., -1 ] b DFT ,0,0,...,0] y z Component Multiply inverse DFT c (Convolution) yi = f(xi) zi = g(xi) yi×zi = [g×f](xi)
Multiplying Big Integers X = 11…10100011101100010010 (N bits) Y = 10…01001100011001001111 X×Y = 10…1110110101001001010100010100110010011110 The high school algorithm takes O(N2) bit operations. Can we do it faster? I hope so See Recursion for one way to do it faster. This is another.
Grade School Revisited: How To Multiply Two Numbers Multiplying Big Integers Grade School Revisited: How To Multiply Two Numbers
Multiplying Big Integers X = 0011 … 1010 0011 1011 0001 0010 m Break into m = O(log N) bit blocks
Multiplying Big Integers X = 0000 … 0000 0000 0011 1010 0011 1011 0001 0010 n=2r blocks m O(log p) an … a7 a6 a5 a4 a3 a2 a1 a0 Break into m = O(log N) bit blocks Pad with zero 2N bits to hold product n blocks where n is a power of 2, ie n=2r. Let p be a prime log p ≥ block size = m p-1 is divisible by n, so Z mod p has n nth roots of unity. View each block as a finite field element in Z mod p. (no actual work)
Multiplying Big Integers X = 0000 … 0000 0000 0011 1010 0011 1011 0001 0010 m f(x) = an-1xn-1 + … + a5x5 + a4x4 + a3x3 + a2x2 + a1x + a0 g(x) = bn-1xn-1 + … + b5x5 + b4x4 + b3x3 + b2x2 + b1x + b0 View as coefficients of a polynomial. Note X = f(2m). Same for Y = g(2m). Multiply g×f using FFT in time O(nlogn). Note X×Y = [g×f](2m). Evaluate [g×f](2m) in time O(n) operations, but each op could be on O(n) bit numbers for a total of O(n2) time.
Multiplying Big Integers Evaluate [g×f](2m) in time O(n). [g×f](x) = cn-1xn-1 + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0 X×Y = 0011 1010 0011 1011 0001 0010 1110 0011 0010 m O(log p) Some texts say the ci can just be shifted and joined. Problem: The field elements may be too big.
Multiplying Big Integers Evaluate [g×f](2m) in time O(n). [g×f](x) = cnxn + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0 Shift each ci by im. Add m O(log p) 101011 001101 110001 000111 100100 010011 X×Y = 01 0101 0110 1011 1010 0001 1111 1011 Adding n numbers each n bits long takes O(n2) but here the numbers are sparse.
Multiplying Big Integers Evaluate [g×f](2m) in time O(n). [g×f](x) = cnxn + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0 Shift each ci by im. Add m O(log p) 101011 001101 110001 000111 100100 010011 X×Y = 01 0101 0110 1011 1010 0001 1111 1011 At each point, at most two numbers overlap Carry is at most one O(N) bit operations.
Multiplying Big Integers X = 11…10100011101100010010 (N bits) Y = 10…01001100011001001111 X×Y = 10…1110110101001001010100010100110010011110 Suppose N is really really big. How many bit operations are needed? O(N logN) O(N logN loglogN) O(N logN loglogN logloglogN loglogloglogN …) FFT time Time stated in text Time as far as I can see
Multiplying Big Integers X = …101000111011000100101010001001010 … N’ Input size = N bits Field element size = N’ = log(N) bits # ai = n = N/N’ # of field ops = O(nlogn) Time for × field op = ? X’ = 1010 0111 0110 0010 0101 0100 0100 1010 N’’ Input size = N’ bits Field element size = N’’ = log(N’) bits # ai = n’ = N’/N’’ # of field ops = O(n’logn’) Time for × field op = ? Total time: O(N’ logN’ loglogN’ logloglogN’ …) And so on …
Multiplying Big Integers X = …101000111011000100101010001001010 … N’ Input size = N bits Field element size = N’ = log(N) bits # ai = n = N/N’ # of field ops = O(nlogn) Time for × field op = ? Total time: = O( n logn ) × O(N’ logN’ loglogN’ logloglogN’ …) = O(N/N’ logN/N’) × O(N’ loglogN logloglogN loglogloglogN …) = O( N logN loglogN logloglogN loglogloglogN …) O(N’ logN’ loglogN’ logloglogN’ …)
The End
eθi = cosθ + isinθ i -i -1 1 θ 176
B1[2]=2 B1[3]=3 B1[1]=1 =e2i 1/n B1[4]=4 B1[0]=0 B1[7]=7 B1[5]=5 B1[6]=6
Fourier Transformation The 0th basis vector will be the constant function 1. B0[j]=j0 j 178
Fourier Transformation The 1st basis vector will be the linear function x. B1[j]=j1 j 179
Fourier Transformation The 2nd basis vector will be the quadratic function x2. B2[j]=j2 j 180
Bn-1[j]=jn-1 j 181