Chapter 2 Divide-and-Conquer algorithms Break up problem into several parts. Solve each part recursively. Combine solutions to sub-problems into overall solution. Most common usage. Break up problem of size n into two equal parts of size ½n. Solve two parts recursively. Combine two solutions into overall solution in linear time. Consequence. Brute force: n2. Divide-and-conquer: n log n. 1

Example 1 Integer Multiplication "Divide and conquer", Caesar's other famous quote "I came, I saw, I conquered" Divide-and-conquer idea dates back to Julius Caesar. Favorite war tactic was to divide an opposing army in two halves, and then assault one half with his entire force.

Integer Arithmetic Add. Given two n-digit integers a and b, compute a + b. O(n) bit operations. Multiply. Given two n-digit integers a and b, compute a × b. Brute force solution: (n2) bit operations. 1 * Multiply 1 + Add 3

Divide-and-Conquer Multiplication: Warmup To multiply two n-digit integers: Multiply four ½n-digit integers. Add two ½n-digit integers, and shift to obtain result. assumes n is a power of 2 4

Karatsuba’s algorithm To multiply two n-digit integers: Add two ½n digit integers. Multiply three ½n-digit integers. Add, subtract, and shift ½n-digit integers to obtain result. Theorem. [Karatsuba-Ofman, 1962] Can multiply two n-digit integers in O(n1.585) bit operations. Karatsuba: also works for multiplying two degree N univariate polynomials A B A C C 5

Recursion Tree . . . . . . T(n) n T(n/2) T(n/2) T(n/2) 3(n/2) T(n/4) T(n / 2k) 3k (n / 2k) . . . . . . T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2) 3 lg n (2) 6

Example 2: Mergesort "Divide and conquer", Caesar's other famous quote "I came, I saw, I conquered" Divide-and-conquer idea dates back to Julius Caesar. Favorite war tactic was to divide an opposing army in two halves, and then assault one half with his entire force.

Sorting Sorting. Given n elements, rearrange in ascending order. Obvious sorting applications. List files in a directory. Organize an MP3 library. List names in a phone book. Display Google PageRank results. Problems become easier once sorted. Find the median. Find the closest pair. Binary search in a database. Identify statistical outliers. Find duplicates in a mailing list. Non-obvious sorting applications. Data compression. Computer graphics. Interval scheduling. Computational biology. Minimum spanning tree. Supply chain management. Simulate a system of particles. Book recommendations on Amazon. Load balancing on a parallel computer. . . . 8

Mergesort Mergesort. Divide array into two halves. Recursively sort each half. Merge two halves to make sorted whole. Jon von Neumann (1945) A L G O R I T H M S divide O(1) sort 2T(n/2) merge O(n) 9

Merging Merging. Combine two pre-sorted lists into a sorted whole. How to merge efficiently? Linear number of comparisons. Use temporary array. Challenge for the bored. In-place merge. [Kronrud, 1969] A G L O R H I M S T A G H I using only a constant amount of extra storage 10

A Useful Recurrence Relation Def. T(n) = number of comparisons to mergesort an input of size n. Mergesort recurrence. Solution. T(n) = O(n log2 n). Assorted proofs. We describe several ways to prove this recurrence. Initially we assume n is a power of 2 and replace  with =. 11

Proof by Recursion Tree T(n) n T(n/2) T(n/2) 2(n/2) T(n/4) T(n/4) T(n/4) T(n/4) 4(n/4) log2n . . . T(n / 2k) 2k (n / 2k) . . . T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2) n/2 (2) n log2n 12

Proof by Telescoping Claim. If T(n) satisfies this recurrence, then T(n) = n log2 n. Pf. For n > 1: assumes n is a power of 2 13

Proof by Induction Claim. If T(n) satisfies this recurrence, then T(n) = n log2 n. Pf. (by induction on n) Base case: n = 1. Inductive hypothesis: T(n) = n log2 n. Goal: show that T(2n) = 2n log2 (2n). assumes n is a power of 2 14

Analysis of Mergesort Recurrence Claim. If T(n) satisfies the following recurrence, then T(n)  n lg n. Pf. (by induction on n) Base case: n = 1. Define n1 = n / 2 , n2 = n / 2. Induction step: assume true for 1, 2, ... , n–1. log2n 15

16

17

18

Randomized Selection Algorithm Input: An array A, low, high, k Output: the k-th smallest element in A[low], A[low+1] … A[high] Assume: 1<= k <= high – low + 1 Algorithm RandomSelect(A, low, high, k): Step 1: m = partition(A, low, high); Step 2: if (m – low + 1 == k) return A[m] else if (m – low + 1 > k) return RandomSelect(A, low, m, k) else return RandomSelect(A, m+1, high, _________ ) End. 19

20

21

The Fast Fourier Transform FFT 22

Outline and Reading Polynomial Multiplication Problem Primitive Roots of Unity (page 63) The FFT Algorithm (page 64) Integer Multiplication ([AHU] Section 8.1) Java FFT Integer Multiplication FFT 23

Polynomials Polynomial: In general, FFT 24

Polynomial Evaluation Horner’s Rule: Given coefficients (a0,a1,a2,…,an-1), defining polynomial Given x, we can evaluate p(x) in O(n) time using the equation Eval(A,x): [Where A=(a0,a1,a2,…,an-1)] If n=1, then return a0 Else, Let A’=(a1,a2,…,an-1) return a0+x*Eval(A’,x) 25

Polynomial Multiplication Problem Given coefficients (a0,a1,a2,…,an-1) and (b0,b1,b2,…,bn-1) defining two polynomials, p() and q(), and number x, compute p(x)q(x). We need to compute the coefficients ci where A straightforward evaluation would take O(n2) time. FFT will do it in O(n log n) time. FFT 26

Polynomial Interpolation & Polynomial Multiplication Given a set of n points in the plane with distinct x-coordinates, there is exactly one (n-1)-degree polynomial going through all these points. Alternate approach to computing p(x)q(x): Calculate p() on 2n x-values, x0,x1,…,x2n-1. Calculate q() on the same 2n x values. Find the (2n-1)-degree polynomial that goes through the points {(x0,p(x0)q(x0)), (x1,p(x1)q(x1)), …, (x2n-1,p(x2n-1)q(x2n-1))}. Unfortunately, a straightforward evaluation would still take O(n2) time, as we would need to apply an O(n)-time Horner’s Rule evaluation to 2n different points. FFT will do it in O(n log n) time, by picking 2n points that are easy to evaluate… FFT 27

Primitive Roots of Unity A number w is a primitive n-th root of unity, for n > 1, if wn = 1 The numbers 1, w, w2, …, wn-1 are all distinct Example 1: Z*11: 2, 6, 7, 8 are 10-th roots of unity in Z*11 22=4, 62=3, 72=5, 82=9 are 5-th roots of unity in Z*11 2-1=6, 3-1=4, 4-1=3, 5-1=9, 6-1=2, 7-1=8, 8-1=7, 9-1=5 Example 2: The complex number e2pi/n is a primitive n-th root of unity, where 28

Properties of Primitive Roots of Unity Inverse Property: If w is a primitive root of unity, then w -1=wn-1 Proof: wwn-1=wn=1 Cancellation Property: For non-zero -n<k<n, Proof: Reduction Property: If w is a primitive (2n)-th root of unity, then w2 is a primitive n-th root of unity. Proof: If 1,w,w2,…,w2n-1 are all distinct, so are 1,w2,(w2)2,…,(w2)n-1 Reflective Property: If n is even, then wn/2 = -1. Proof: By the cancellation property, for k=n/2: Corollary: wk+n/2= -wk. FFT 29

Fourier transform viewed as polynomial evaluation

The Discrete Fourier Transform Given coefficients (a0,a1,a2,…,an-1) for an (n-1)-degree polynomial p(x) The Discrete Fourier Transform is to evaluate p at the values 1,w,w2,…,wn-1 We produce (y0,y1,y2,…,yn-1), where yj=p(wj) That is, Matrix form: y=Fa, where F[i,j]=wij. The Inverse Discrete Fourier Transform recovers the coefficients of an (n-1)-degree polynomial given its values at 1,w,w2,…,wn-1 Matrix form: a=F -1y, where F -1[i,j]=w-ij/n. FFT 34

Correctness of the inverse DFT The DFT and inverse DFT really are inverse operations Proof: Let A=F -1F. We want to show that A=I, where If i=j, then If i and j are different, then FFT 35

Convolution The DFT and the inverse DFT can be used to multiply two polynomials So we can get the coefficients of the product polynomial quickly if we can compute the DFT (and its inverse) quickly… FFT 36

The Fast Fourier Transform The FFT is an efficient algorithm for computing the DFT The FFT is based on the divide-and-conquer paradigm: If n is even, we can divide a polynomial into two polynomials and we can write FFT 37

The FFT Algorithm The running time is O(n log n). [inverse FFT is similar] FFT 38

The FFT Algorithm FFT 39

FFT 40

FFT 41

Using the 1D FFT for filtering Signal = sin(7t) + .5 sin(5t) at 128 points Noise = random number bounded by .75 Filter by zeroing out FFT components < .25 CS267 Lecture 21 03/17/2011

Using the 2D FFT for image compression Image = 200 x 320 matrix of values Compress by keeping largest 2.5% of FFT components Similar idea used by jpeg Jpeg breaks an image into 8x8 pixel blocks, does a DCT, quantizes the coefficients (round them so they can be Stored in very few bits), and stored these bits slide courtesy of Jim Demmel, UCB 03/17/2011