1. Applied Symbolic Computation1 Applied Symbolic Computation (CS 567) The Fast Fourier Transform (FFT) and Convolution Jeremy R. Johnson TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA A A A

2. Applied Symbolic Computation2 Introduction Objective: To derive the fast Fourier transform (FFT) and view it as a factorization of the Vandermonde matrix. To introduce the convolution operator and relate it to polynomial and matrix algebra. To use the Chinese Remainder Theorem to prove the convolution theorem and rederive the FFT. –Polynomial multiplication using interpolation –Vandermonde Matrices –Factoring the Vandermonde matrix using even/odd symmetry –Convolution Theorem –Deriving the FFT using the Chinese Remainder Theorem References: Lipson, Tolimieri, Cormen et al.

3. Applied Symbolic Computation3 Polynomial Multiplication using Interpolation Compute C(x) = A(x)B(x), where degree(A(x)) = m, and degree(B(x)) = n. Degree(C(x)) = m+n, and C(x) is uniquely determined by its value at m+n+1 distinct points. [Evaluation] Compute A(  i ) and B(  i ) for distinct  i, i=0,…,m+n. [Pointwise Product] Compute C(  i ) = A(  i ) * B(  i ) for i=0,…,m+n. [Interpolation] Compute the coefficients of C(x) = c n x m+n + … + c 1 x +c 0 from the points C(  i ) = A(  i ) * B(  i ) for i=0,…,m+n.

4. Applied Symbolic Computation4 Horner’s Method Let A(x) = a 3 x 3 + a 2 x 2 + a 1 x + a 0 A(  ) = ((a 3  + a 2 )  + a 1 )  + a 0 In general, let A 0 = a m A i = A i-1  + a m-i A m = A(  ) The number of operations is 2m (m additions, m multiplications)

5. Applied Symbolic Computation5 Evaluation Utilizing Symmetry The cost of evaluation at two points can be reduced if one is the negative of the other. Let A(x) = A 1 (x 2 )x + A 0 (x 2 ), where the coefficients of A 1 (x) are the odd coefficients of A(x) and the coefficients of A 0 (x) are the even coefficients of A(x) –Since (-  ) 2 =  2, A 0 (  2 ) = A 0 (-  2 ) and A 1 (  2 ) = A 1 (-  2 ) –A(  ) = A 0 (  2 ) +  A 1 (  2 ) –A(-  ) = A 0 (  2 ) -  A 1 (  2 ) Example –A(x) = a 5 x 5 + a 4 x 4 + a 3 x 3 + a 2 x 2 + a 1 x + a 0 = A 1 (x 2 )x + A 0 (x 2 ) –A 1 (x) = a 5 x 2 + a 3 x + a 1 –A 0 (x) = a 4 x 2 + a 2 x + a 0

6. Applied Symbolic Computation6 Evaluation Utilizing Symmetry To evaluate A(x), with deg(A) = m, at , requires –4m operations using Horner’s method for  and -  –4(m/2) + 3 = 2m + 3 [approx. half] operations (using  symmetry) To evaluate a polynomial of degree N-1 at N=2m points –2N 2 + o(N 2 ) –N 2 + o(N 2 )

7. Applied Symbolic Computation7 Vandermonde Matrix

8. Applied Symbolic Computation8 Discrete Fourier Transform If  1,…,  n are equal to the nth roots of unity, the Vandermonde matrix becomes the DFT matrix. Let  i =  i, where  is a primitive n th root of unity.

9. Applied Symbolic Computation9 Discrete Fourier Transform

10. Applied Symbolic Computation10 Properties of Roots of Unity Lemma:  -1 = . Lemma: Let n = 2m, and  be a primitive n th root of unity. Then  2 is a primitive m th root of unity. Lemma: Let n = 2m, and  be a primitive n th root of unity. Then  m = -1 and  m+k = -  k. Therefore, F 2m is a Vandermonde matrix where half the points are negatives of the other half. Thus, we can utilize the previous factorization to compute the DFT. Moreover, if n=2 k this property can be used recursively.

11. Applied Symbolic Computation11 FFT Input: N = 2 k, a = (a 0,a 1,…,a n-1 ) = a 0 +a 1 x + … + a n-1 x n-1 Output: A = (A 0,A 1,…,A n-1 ) with A k = a(  k ) Procedure FFT(N,a, ,A) if N = 1 then A0 := a0; else n := N/2; b := (a 0,a 2,…,a 2(n-1) ); c := (a 1,a 3,…,a 2n-1 ); FFT(n,b,  2,B); FFT(n,c,  2,C); for k from 0 to n-1 do A k := B k +  k × C k ; A n+k := B k -  k × C k ; end do; end if; End;

12. Applied Symbolic Computation12 Fast Fourier Transform Assume that n = 2m, then Let T(n) be the computing time of the FFT and assume that n=2 k, then T(n) = 2T(n/2) +  (n) T(n) =  (nlogn)

13. Applied Symbolic Computation13 Example FFT Factorization

14. Applied Symbolic Computation14 Polynomial Multiplication using Interpolation Compute C(x) = A(x)B(x), where degree(A(x)) = m, and degree(B(x)) = n. Degree(C(x)) = m+n, and C(x) is uniquely determined by its value at m+n+1 distinct points. [Evaluation] Compute A(  i ) and B(  i ) for distinct  i, i=0,…,m+n. [Pointwise Product] Compute C(  i ) = A(  i ) * B(  i ) for i=0,…,m+n. [Interpolation] Compute the coefficients of C(x) = c n x m+n + … + c 1 x +c 0 from the points C(  i ) = A(  i ) * B(  i ) for i=0,…,m+n.

15. Applied Symbolic Computation15 Inverse DFT Proof: The (i,j) element of If i = j, then the sum is equal to n, and if i  j, then the sum is 0, since x n -1 = (x-1)(x n-1 + … + x + 1)

16. Applied Symbolic Computation16 Linear Convolution Definition: Let u and v be two vectors of size m and n respectively. The linear convolution of u and v is equal to Example

17. Applied Symbolic Computation17 Linear Convolution and Polynomial Multiplication Linear convolution is the same as polynomial multiplication Let u(x) = u m x m + … + u 1 x + u 0 and v(x) = v n x n + … + v 1 x + v 0 Then u(x)*v(x) = (u*v) m+n x m+n + … + (u*v) 1 x + (u*v) 0 Example –u(x)v(x) = (u 2 x 2 + u 1 x + u 0 )(v 2 x 2 + v 1 x + v 0 ) = (u 2 v 2 )x 4 + (u 1 v 2 + u 2 v 1 )x 3 + (u 0 v 2 + u 1 v 1 + u 2 v 0 )x 2 + (u 0 v 1 + u 1 v 0 )x + u 0 v 0

18. Applied Symbolic Computation18 Cyclic Convolution Definition: Let u and v be two vectors of size n respectively. The n-point cyclic convolution of u and v is equal to Example

19. Applied Symbolic Computation19 Cyclic Convolution and Polynomial Multiplication N-point cyclic convolution is the same as polynomial multiplication modulo x n -1. Let u(x) = u m x m + … + u 1 x + u 0 and v(x) = v n x n + … + v 1 x + v 0 Then u(x)*v(x) (mod x n -1) = (u*v) n-1 x n-1 + … + (u*v) 1 x + (u*v) 0 Example u(x)v(x) = (u 2 x 2 + u 1 x + u 0 )(v 2 x 2 + v 1 x + v 0 ) = (u 2 v 2 )x 4 + (u 1 v 2 + u 2 v 1 )x 3 + (u 0 v 2 + u 1 v 1 + u 2 v 0 )x 2 + (u 0 v 1 + u 1 v 0 )x + u 0 v 0 = (u 2 v 2 )x + (u 1 v 2 + u 2 v 1 ) + (u 0 v 2 + u 1 v 1 + u 0 v 2 )x 2 + (u 0 v 1 + u 1 v 0 )x + u 0 v 0 = (u 0 v 2 + u 1 v 1 + u 2 v 0 )x 2 + (u 0 v 1 + u 1 v 0 + u 2 v 2 )x + (u 0 v 0 + u 1 v 2 + u 2 v 1 )

20. Applied Symbolic Computation20 Circulant Matrices Definition: A circulant matrix C(u 0,…,u n ) is obtained by cyclically rotating the vector u 0,…,u n. Multiplication by a circulant matrix corresponds to cyclic convolution. The (i,j) element of C(u 0,…,u n ) is equal to u i-j mod n Example

21. Applied Symbolic Computation21 Shift Matrices Definition: The n-point shift matrix S n is the permutation matrix that cyclically shifts the elements of a vector. The (i,j) element of S n is equal to 1 when i-j  1 (mod n) Example

22. Applied Symbolic Computation22 Generating Circulant Matrices A circulant matrix is equal to a linear combination of powers of the shift matrix.

23. Applied Symbolic Computation23 Convolution Theorem Theorem: F n (u * v) = F n (u)  F n (v) u * v = F n -1 (F n (u)  F n (v)) This theorem provides an O(nlogn) algorithm for performing cyclic convolution provided F n is computed with the FFT. We will prove this theorem two different ways –Show that F n diagonalizes a circulant matrix –Use the Chinese remainder theorem

24. Applied Symbolic Computation24 Diagonalizing the Shift Matrix Theorem: F n S n = W n F n, where W n = diag(1, ,…,  n-1 )

25. Applied Symbolic Computation25 Diagonalizing a Circulant Matrix Theorem: F n C n (u) = diag(F n (u)) F n

26. Applied Symbolic Computation26 First Proof of the Convolution Theorem Theorem: F n (u * v) = F n (u)  F n (v) u * v = F n -1 (F n (u)  F n (v)) Proof: F n (u * v) = F n (C(u) v) = diag(F n (u)) F n v = F n (u)  F n (v)

27. Applied Symbolic Computation27 Polynomial Version of the Chinese Remainder Theorem Theorem: Let f(x) and g(x) be polynomials in F[x] (coefficients in a field). Assume that gcd(f(x),g(x)) = 1. For any A 1 (x) and A 2 (x) there exist a polynomial A(x) with A(x)  A 1 (x) (mod f(x)) and A(x)  A 2 (x) (mod g(x)). Theorem: F[x]/(f(x)g(x))  F[x]/(f(x))  F[x]/(g(x)). I.E. There is a 1-1 mapping from F[x]/(f(x)g(x)) onto F[x]/(f(x))  F[x]/(g(x)) that preserves arithmetic. A(x)  (A(x) mod f(x), A(x) mod g(x))

28. Applied Symbolic Computation28 Multifactor CRT The CRT can be generalized to the case when we have n pairwise relatively prime polynomials. If f 1 (x),…,f n (x) are pairwise relatively prime, i.e. gcd(f i (x),f j (x)) = 1 for i  j, then given A 1 (x),…,A n (x) there exists a polynomial A(x) such that A  A i (x) (mod f i (x)). Moreover, there exist a system of orthogonal idempotents: E 1 (x),…,E n (x), such that E i (x)  1 (mod f i (x)) and E i (x)  0 (mod f j (x)) for i  j. A(x) = A 1 (x)E 1 (x) + … + A n (x)E n (x) F[x]/(A 1 (x)…A n (x)  F[x]/(A 1 (x))  …  F[x]/(A n (x))

29. Applied Symbolic Computation29 CRT and the Convolution Theorem Since x n -1 = (x-1)(x-  )…(x-  n-1 ) and gcd(x-  i, x-  j ) = 1 for i  j C[x]/(x n -1)  C[x]/(x-1)  …  C[x]/(x-  n-1 ) Let u(x) and v(x) be elements of C[x]/(x n -1) The map C[x]/(x n -1) onto C[x]/(x-1)  …  C[x]/(x-  n-1 ) is F n –u(x) gets mapped to (u(1),…,u(  n-1 )) Multiplication in C[x]/(x n -1) is cyclic convolution Multiplication in C[x]/(x-1)  …  C[x]/(x-  n-1 ) is a point-wise product First multiplying u(x) and v(x) in C[x]/(x n -1) and then applying F n is the same as applying F n to u(x) and F n to v(x) and then multiplying in C[x]/(x-1)  …  C[x]/(x-  n-1 ). Therefore, F n (u * v) = F n (u)  F n (v)

30. Applied Symbolic Computation30 CRT and the FFT Factor F n by projecting onto C[x]/(x-1)  …  C[x]/(x-  n-1 ) in stages Let n = 2m, then x n -1 = (x m -1)(x m +1) and gcd(x m -1,x n -1) = 1 C[x]/(x n -1)  C[x]/(x m -1)  C[x]/(x m +1) –This mapping is F 2  I m C[x]/(x m -1)  C[x]/(x-1)  C[x]/(x-  2 )  …  C[x]/(x-  2(m-1) ) –This mapping is F m C[x]/(x m +1)  C[x]/(x-  )  …  C[x]/(x-  2m-1 ) –This mapping is W m F m (F m  W m F m )(F 2  I m ) A(x)  (A(1),A(  2),…,A(  2(m-1) ),A(  ),…,A(  n-1) ))

31. Applied Symbolic Computation31 CRT and the FFT (F m  W m F m )(F 2  I m ) A(x)  (A(1),A(  2),…,A(  2(m-1) ),A(  ),…,A(  n-1) )) F 2m A(x)  (A(1), A(  ),A(  2),…,A(  n-1) )) Therefore,