Advanced Algorithms Analysis and Design By Dr. Nazir Ahmad Zafar Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Lecture No. 43 Polynomials and Fast Fourier Transform Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

Definitions A Field is a set F with two binary operations + : F × F → F and * : F × F → F such that (F, +) is an abelian group with identity element 0 (F\{0}, *) is an abelian group with identity element 1 Multiplication distributes over addition a*(b + c) = (a*b) + (a*c) (a + b)*c = (a*c) + (b*c) Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Definitions Polynomial A polynomial in the variable x over an algebraic field F is a representation of a function A(x) as a formal sum A(x) = a0 + a1x1 + a2x2 + . . .+ anxn Coefficients Values a0, a1,..., an are coefficients of polynomial, and drawn from a field F, typically set of complex numbers. Degree A polynomial A(x) is said to have degree n if its highest coefficient an is nonzero Degree Bound Any integer strictly greater than the degree of a polynomial is a degree-bound of that polynomial. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Addition of two Polynomials: Brute Force Addition of two polynomials of degree n takes Θ(n) time, Example 1 A (x) = a0 + a1x1 + a2x2 + . . .+ anxn B (x) = b0 + b1x1 + b2x2 + . . .+ bnxn C (x) = (a0 + b0) + (a1 + b1) x1 + . . .+(an + bn)xn Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Multiplication of Two Polynomial: Brute Force Multiplication of two polynomials of degree n takes Θ(n2) Example 2 A (x) = a0 + a1x1 + a2x2 + . . .+ anxn B (x) = b0 + b1x1 + b2x2 + . . .+ bnxn a0b0 + a1b0x1 + . . .+ (anb0)xn a0b1x1 + a1b1x2 + . . .+ (anb1)xn+1 . . . a0bn xn + a1bnxn+1 + . . .+ (anbn)xn+n C (x) = (a0b0) + (a1b0 + a0b1)x1 + . . . + (anbn)xn+n Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Polynomial Representation The Coefficient Representation Point Value Presentation Note The method for multiplying polynomials equations as above take (n2) time when the polynomials are represented in coefficient form But (n) time when represented in point value form We can multiply using coefficient representation in only (n log n) time converting between two forms This lecture make much use of complex numbers, the symbol i has same meaning, you know sqr(-1) Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

1. Coefficient Representation A coefficient representation of a polynomial degree bound n is a vector of coefficients: a = (a0, a1,…, an-1) Vectors as Column In this lecture, we will treat vector as column vector Convenient Way The coefficients representation is convenient for certain operations on polynomials Example: Computing A(x) at x0 Operation of evaluating polynomials A(x) at given point x0 consists of computing value of A(x0). Evaluation takes time (n) using Horner’s rule Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Evaluation and Addition using Coefficient Form Operation 1: Horner’s Rule A(x0)= a0 + x0( a1 + x0( a2 +…x 0(an-2 +x0(an-1))… ) Operation 2: Addition of Two Polynomials Similarly adding two polynomials represented by the coefficient vectors: a = (a0, a1 ,…, an-1 ) and b = (b0, b1, … ,bn-1) takes (n) times We just produce the coefficient vector: c = (c0, c1,…, c n-1) where cj = aj + bj,  j = 1 ,…, n - 1 Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Multiplication using Coefficient Representation Operation 3: Multiplication of Two Polynomials Consider multiplication of A(x) and B(x), with degree bounds n, represented in coefficient form If we use the method described above polynomials multiplication takes time O(n2). Since each coefficient in vector a must be multiplied by each coefficients in the vector b. Operation of multiplying polynomials in coefficient form seems to be considerably more difficult than that of evaluating or adding two polynomials. The resulting coefficient vector c, also called the convolution of the input vectors a and b. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

2. Point–value Representation A point value representation of a polynomial A(x) of degree bound n is a set of n point value pairs. { (x0, y0), (x1, y1),. . . ,(x n-1, y n-1) }, all of the x k are distinct and y k = A(x k), for k = 0,1, . . ., n-1. Polynomial has various point value representations, since any set of n distinct points x0 ,x1 ,…,x n-1 can be used as a basis for the representation. Conversion: From a Coefficient Form to Point Value Form Computing a point value representation for a polynomials given in coefficient form is in principle straight forward , This is because select n distinct points x0 ,x1 ,…,x n-1 and then evaluate A(x k) for k = 0,1 ,…, n-1. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

2. Point–value Representation With Horner’s rule, n-point evaluation takes q(n2). This is because for x = x0, evaluation cost is q(n). And since there are n number of points, hence there will be q(n2) cost for evaluating all of the n number of points using Horner’s rule. Clever Choose of xk We shall see that if we choose xk cleverly, this computation can be accelerated to run in q(n log n) Inverse of evaluating coefficient form of polynomial from point value representation called interpolation. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Theorem Uniqueness of an interpolating polynomial For any set { (x0, y0),(x1, y1),. . ., (x n-1, y n-1) } of n point–value pairs such that all xk values distinct, there is a unique polynomial A(x) of degree bound n such that yk = A(xk) for k = 0,1, . . ., n-1. Proof Proof is based on existence of inverse of a matrix. Let us suppose that A(x) is required polynomial A(x) = a0 + a1x1 + a2x2 +. . .+ anxn Equation: yk = A(xk) is equivalent to the matrix equation given in the next slide. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Theorem 1 x0 x0 … x0 a0 y0 1 x1 x1 … x1 a1 y1 . . . … . . . 1 x x … x a y This matrix on the left side is called vander-monde matrix and is denoted V(x0, x1, …..xn-1) The determinant of this this matrix is If xk are distinct then it is nonsingular. The coefficient aj can be uniquely determined a = V(x0, x1, …..xn-1)-1 y Dr Nazir A. Zafar Advanced Algorithms Analysis and Design Dr Nazir A. Zafar Advanced Algorithms Analysis and Design Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Representation of polynomials Solving The Equation in Proof of Theorem Using LU decomposition algorithms, we can solve these equation in O(n3) A faster algorithm, in Θ(n2), for n-point interpolation is based on Lagrange's formula: Dr Nazir A. Zafar Advanced Algorithms Analysis and Design Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Representation of polynomials Addition using Point Value Form The point-value representation is quite convenient for many operations on polynomials. For addition: C(x) = A(x) + B(x)  C(xk) = A(xk) + B(xk) More precisely, if point value representation for A {(x0, y0), (x1, y1),. . ., (xn-1, yn-1)}, And for B: {(x0, y’0), (x1, y’1),. . ., (xn-1, y’n-1)}, Then a point-value representation for C is {(x0, y0+y’0), (x1, y1+y’1),. . ., (xn-1, yn-1+y’n-1)}, Thus, the time to add two polynomials of degree-bound n in point-value form is Θ(n). Dr Nazir A. Zafar Advanced Algorithms Analysis and Design Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Multiplication using Point Value Form Similarly, point-value representation is convenient for multiplying polynomials as well. C(x) = A(x) B(x)  C(xk) = A(xk)B(xk) for any xk, We can multiply a point value representations for A and B to obtain a point-value representation for C. A standard point-value representation for A and B consists of n point-value pairs for each polynomial Multiplying these, we must extended point-value representations for A and B of 2n point-value each. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Multiplication using Point Value Form Given an extended point-value representation for A, {(x0, y0), (x1, y1),..., (x2n-1, y2n-1)}, And extended point-value representation for B, {(x0, y’0), (x1, y’1),..., (x2n-1, y’2n-1)}, Then a point-value representation for C is {(x0, y0 y’0), (x1, y1 y’1),..., (xn-1, yn-1 y’n-1)} Finally, we consider how to evaluate a polynomial given in point-value form at a new point. Apparently no simpler approach than converting polynomial to coefficient form, and then evaluating it Dr Nazir A. Zafar Advanced Algorithms Analysis and Design Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Discrete Fourier Transform We can use any points as evaluation points, but by choosing evaluation points carefully, we can convert between representations in only Θ(n lg n) time. If we take “complex roots of unity” evaluation points, we can produce a point-value representation taking Discrete Fourier Transform of coefficient vector. The inverse operation, interpolation, can be performed by taking “inverse DFT” of point-value pairs, yielding a coefficient vector. We will show how FFT performs the DFT and inverse DFT operations in Θ(n lg n) Multiplication procedure is shown in the next slide Dr Nazir A. Zafar Advanced Algorithms Analysis and Design Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Fast multiplication of polynomials in coefficient form representation Point-value representation Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Procedure: Multiplication of Polynomials in n lg n We assume n is a power of 2; this requirement can always be met by adding zero coefficients. Double degree-bound: Create coefficient representations of A(x) and B(x) as degree bound 2n polynomials by adding n high-order zero coefficients to each. Evaluate: Compute point-value representations of A(x) and B(x) of length 2n through two applications of FFT of order 2n. These representations contain the values of the two polynomials at the (2n)th roots of unity. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Multiplication of Polynomials in n lg n Point wise multiply: Compute point-value form for polynomial C(x) = A(x)B(x) by multiplying these together point wise. This representation contains the value of C(x) at each (2n)th root of unity. Interpolate: Create coefficient representation of C(x) through a single application of an FFT on 2n point-value pairs to compute inverse DFT. Steps (1) and (3) take time Θ(n), and steps (2) and (4) take time Θ(n lg n). Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Complex Roots of Unity and Their Properties We claimed that if we use complex roots of unit we can evaluate and interpolate polynomials in Θ(nlgn) time. Here, we define complex roots of unity and study their properties. Define the DFT,and then show how the FFT computes the DFT and its inverse in just Θ(nlgn) time. Complex root of unity A complex nth root of unity is a complex number ω such that ωn =1 There are exactly n complex nth roots of unity: e2πik/n for k=0,1,…,n-1 eiu =cos(u) + isin(u). Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Complex Roots of Unity and Their Properties Values of ω08 , ω18 ,. . ., ω78 in complex plane are shown where ω8 = e2πi/8 is the principal 8th root of unity. Complex roots of unity form a cyclic group Complex roots of unity have interesting properties Some of those are discussed in the next ω18 ω48 ω38 ω28 ω68 ω58 ω78 ω08 =ω88 i -i -1 1 Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Properties: Complex Roots of Unity Lemma 1 (cancellation lemma) For any integers n ≥ 0,k ≥0,and d > 0, ωdkdn = ωkn . Proof : the lemma follows directly from ωn =e2πi/n ,since ωdkdn = (e2πi/dn)dk =(e2πi/n)k = ωkn Corollary 1 (cancellation lemma) For any even integer n > 0, ωn/2n = -1 Proof: We know that ωn =e2πi/n Now ωn/2n = ωn/22.n/2 = ω2 = ωn =e2πi/2 = ωn =eπi = -1 Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Properties: Complex Roots of Unity Lemma 2 (Halving Lemma) If n > 0 is even, then squares of n complex nth root of unity are the n/2 complex (n/2)th root of unity. Proof: By the cancellation lemma, we have: (ωnk )2 = ωkn/2 For any nonnegative integer k, note that if we square all of complex nth root of unity, then each (n/2)th root of unity is obtained exactly twice, since (ωnk +n/2)2 = ω2k+nn/2 = ω2knωnn = ω2kn Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design Continue Halving Lemma is Essential in Reducing Cost Thus ωkn and ωk+n/2n/2 have the same square. This property can also be proved using corollary, ωnn/2 = ω2 = -1 Since ωn/2n = -1 implies ωk+n/2n = - ωkn and thus (ωnk +n/2)2 = (ωnk)2 As we shall see, the halving lemma is essential to our divide-and-conquer approach of converting between coefficient and point-value representation of polynomials Since it guarantees that the recursive sub problems are only half as large. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Properties: Complex Roots of Unity Lemma 3 (Summation Lemma) For any integer n ≥ 1 and nonnegative integer k not divisible by n, n-1 ∑ (ωkn)j = 0 j=0 Proof: ∑ (ωkn)j = (ωnk)n-1 j=0 ωnk-1 Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Properties: Complex Roots of Unity = (ωnn)k-1 ωnk-1 = (1)k-1 = 0 Requiring that k not be divisible by n ensures that the denominator is not 0, since ωnk = 1 only when k is divisible by n. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design The DFT Recall that we wish to evaluate a polynomial n-1 A(x) = ∑ ajxj j=0 of degree-bound n at ωn0 ,ωn1, ωn2, ……..ωnn-1 ( that is, at the n complex nth roots of unity). Without loss of generality, we assume that n is a power of 2, since a given degree-bound can always be raised we can always add new high-order zero coefficients as necessary. Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design The DFT We assume that A is given in coefficient from a = ( a0, a1,. . ., an-1). Let us define the results yk for k = 0,1,. . ., n-1, by yk = A(ωkn) n-1 = ∑ aj ωk jn j=0 The vector y = (y0, y1,. . ., yn-1) is Discrete Fourier Transform (DFT) of the coefficient vector a = ( a0, a1,. . ., an-1). We can also write y = DFTn(a) Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design The FFT And hence (A) (A) Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design The FFT For k = 0, 1. . ., n/2-1 Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

FFT Recursive Algorithm Dr Nazir A. Zafar Advanced Algorithms Analysis and Design