Divide-and-Conquer UNC Chapel HillZ. Guo

Divide-and-Conquer It’s a technique instead of an algorithm Recursive in structure – Divide the problem into several smaller sub- problems that are similar to the original but smaller in size – Conquer the sub-problems by solving them recursively. If they are small enough, just solve them in a straightforward manner. – Combine the solutions to create a solution to the original problem UNC Chapel HillZ. Guo

An Example: Merge Sort Sorting Problem: Sort a sequence or n elements into non-decreasing order. Divide: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each Conquer: Sort the two subsequences recursively using merge sort. Combine: Merge the two sorted subsequences to produce the sorted answer. UNC Chapel Hill

Z. Guo Merge Sort – Example UNC Chapel Hill

Z. Guo Merge Sort – Example Original Sequence Sorted Sequence UNC Chapel Hill

Z. Guo Merge-Sort (A, p, r) INPUT: a sequence of n numbers stored in array A OUTPUT: an ordered sequence of n numbers MergeSort (A, p, r) // sort A[p..r] by divide & conquer 1if p < r 2 then q   (p+r)/2  3 MergeSort (A, p, q) 4 MergeSort (A, q+1, r) 5 Merge (A, p, q, r) // merges A[p..q] with A[q+1..r] Initial Call: MergeSort(A, 1, n) UNC Chapel Hill

Z. Guo Procedure Merge Merge(A, p, q, r) 1 n 1  q – p n 2  r – q 3for i  1 to n 1 4 do L[i]  A[p + i – 1] 5for j  1 to n 2 6 do R[j]  A[q + j] 7L[n 1 +1]   8R[n 2 +1]   9i  1 10j  1 11for k  p to r 12 do if L[i]  R[j] 13 then A[k]  L[i] 14 i  i else A [k]  R[j] 16 j  j + 1 Sentinels, to avoid having to check if either subarray is fully copied at each step. Input: Array containing sorted subarrays A[p..q] and A[q+1..r]. Output: Merged sorted subarray in A[p..r]. UNC Chapel Hill

Z. Guo j Merge – Example … … A k k k k k k k k i i i i   i j j j j k L R UNC Chapel Hill

Z. Guo Correctness of Merge Merge(A, p, q, r) 1 n 1  q – p n 2  r – q 3for i  1 to n 1 4 do L[i]  A[p + i – 1] 5for j  1 to n 2 6 do R[j]  A[q + j] 7L[n 1 +1]   8R[n 2 +1]   9i  1 10j  1 11for k  p to r 12 do if L[i]  R[j] 13 then A[k]  L[i] 14 i  i else A [k]  R[j] 16 j  j + 1 Loop Invariant for the for loop At the start of each iteration of the for loop: Subarray A[p..k – 1] contains the k – p smallest elements of L and R in sorted order. L[i] and R[j] are the smallest elements of L and R that have not been copied back into A. Initialization: Before the first iteration: A[p..k – 1] is empty. i = j = 1. L[1] and R[1] are the smallest elements of L and R not copied to A. UNC Chapel Hill

Z. Guo Correctness of Merge Merge(A, p, q, r) 1 n 1  q – p n 2  r – q 3for i  1 to n 1 4 do L[i]  A[p + i – 1] 5for j  1 to n 2 6 do R[j]  A[q + j] 7L[n 1 +1]   8R[n 2 +1]   9i  1 10j  1 11for k  p to r 12 do if L[i]  R[j] 13 then A[k]  L[i] 14 i  i else A [k]  R[j] 16 j  j + 1 Maintenance: Case 1 : L[i]  R[j] By LI, A contains p – k smallest elements of L and R in sorted order. By LI, L[i] and R[j] are the smallest elements of L and R not yet copied into A. Line 13 results in A containing p – k + 1 smallest elements (again in sorted order). Incrementing i and k reestablishes the LI for the next iteration. Similarly for L[i] > R[j]. Termination: On termination, k = r + 1. By LI, A contains r – p + 1 smallest elements of L and R in sorted order. L and R together contain r – p + 3 elements. All but the two sentinels have been copied back into A. UNC Chapel Hill

Z. Guo Analysis of Merge Sort Running time T(n) of Merge Sort: Divide: computing the middle takes  (1) Conquer: solving 2 subproblems takes 2T(n/2) Combine: merging n elements takes  (n) Total : T(n) =  (1) if n = 1 T(n) = 2T(n/2) +  (n) if n > 1  T(n) =  (n lg n) (CLRS, Chapter 4) UNC Chapel Hill MergeSort (A, p, r) // sort A[p..r] by divide & conquer 1if p < r 2 then q   (p+r)/2  3 MergeSort (A, p, q) 4 MergeSort (A, q+1, r) 5 Merge (A, p, q, r) // merges A[p..q] with A[q+1..r]

Recurrence Relations Recurrences (Chapter 4) – Substitution Method – Iteration Method – Master Method Arising from Divide and Conquer (e.g. MERGE-SORT) T(n) =  (1) if n  c T(n) = a T(n/b) + D(n) + C(n) otherwise UNC Chapel HillZ. Guo

Substitution Method Guessing the form of the solutions, then using mathematical induction to find the constants and show the solution works. It works well when it is easy to guess. But, there is no general way to guess the correct solution. UNC Chapel HillZ. Guo

An Example Solve: T(n) = 3T(  n/3  ) + n T(n)   =  c n lg n UNC Chapel HillZ. Guo GUESS

An Example Solve: T(n) = 3T(  n/3  ) + n T(n)  3c  n/3  lg  n/3  + n  c n lg (n/3) + n = c n lg n - c n lg3 + n = c n lg n - n (c lg 3 - 1)  c n lg n * The last step is true for c  1 / lg3. UNC Chapel HillZ. Guo

Making a Good Guess Guessing a similar solution to the one that you have seen before – T(n) = 3T(  n/3  + 5 ) + n similar to T(n) = 3T(  n/3  ) + n when n is large, the difference between n/3 and ( n/3 + 5 ) is insignificant Another way is to prove loose upper and lower bounds on recurrence and then reduce the range of uncertainty. – Start with T(n) =  (n) & T(n) = O(n 2 )  T(n) =  (n log n) UNC Chapel HillZ. Guo

Subtleties When the math doesn’t quite work out in the induction, try to adjust your guess with a lower-order term. For example: – We guess T(n)  O(n) for T(n) = 3T(  n/3  )+ 4, but we have T(n)  3c  n/3  + 4 = c n + 4 – New guess is T(n)  c n - b, where b  0 T(n)  3(c  n/3  - b)+4 = c n - 3b + 4 = c n - b - (2b-4) Therefore, T(n)  c n - b, if 2b - 4  0 or if b  2 UNC Chapel HillZ. Guo

Changing Variables Use algebraic manipulation to turn an unknown recurrence similar to what you have seen before. – Consider T(n) = 2T(  n 1/2  ) + lg n UNC Chapel HillZ. Guo

Changing Variables Use algebraic manipulation to turn an unknown recurrence similar to what you have seen before. – Consider T(n) = 2T(  n 1/2  ) + lg n – Rename m = lg n and we have T(2 m ) = 2T(2 m/2 ) + m – Set S(m) = T(2 m ) and we have S(m) = 2S(m/2) + m  S(m) = O(m lg m) – Changing back from S(m) to T(n), we have T(n) = T(2 m ) = S(m) = O(m lg m) = O(lg n lg lg n) UNC Chapel HillZ. Guo

Avoiding Pitfalls Be careful not to misuse asymptotic notation. For example: – We can falsely prove T(n) = O(n) by guessing T(n)  c n for T(n) = 2T(  n/2  ) + n T(n)  2c  n/2  + n  c n + n = O(n)  Wrong! – The err is that we haven’t proved T(n)  c n UNC Chapel HillZ. Guo

Exercises Solution of T(n) = T(  n/2  ) + 1 is O(lg n) Solution of T(n) = 2T(  n/2  + 17) + n is O(n lg n) Solve T(n) = 2T(n 1/2 ) + 1 by making a change of variables. Don’t worry whether values are integral. UNC Chapel HillZ. Guo

UNC Chapel HillZ. Guo Iteration Method Expand (iterate) the recurrence and express it as a summation of terms dependent only on n and the initial conditions The key is to focus on 2 parameters – the number of times the recurrence needs to be iterated to reach the boundary condition – the sum of terms arising from each level of the iteration process Techniques for evaluating summations can then be used to provide bounds on solution.

UNC Chapel HillZ. Guo An Example Solve: T(n) = 3T(n/4) + n T(n) = n + 3T(n/4) = n + 3[ n/4 + 3T(n/16) ] = n + 3[n/4] + 9T(n/16) = n + 3[n/4] + 9 [n/16] + 27T(n/64) T(n)  n + 3n/4 + 9n/ n/64 + … + 3 log 4 n  (1)  n  (3/4) i +  (n log 4 3 ) = 4n+ o(n) = O(n)

UNC Chapel HillZ. Guo Recursion Trees Keep track of the time spent on the subproblems of a divide and conquer algorithm A convenient way to visualize what happens when a recursion is iterated Help organize the algebraic bookkeeping necessary to solve the recurrence

Comp 550 Recursion-tree Method Making a good guess is sometimes difficult... Use recursion trees to devise good guesses. Recursion Trees – Show successive expansions of recurrences using trees. – Keep track of the time spent on the subproblems of a divide and conquer algorithm. – Help organize the algebraic bookkeeping necessary to solve a recurrence.

Comp 550 Recursion Tree – Example Running time of Merge Sort: T(n) =  (1) if n = 1 T(n) = 2T(n/2) +  (n) if n > 1 Rewrite the recurrence as T(n) = c if n = 1 T(n) = 2T(n/2) + cn if n > 1 c > 0: Running time for the base case and time per array element for the divide and combine steps.

Comp 550 Recursion Tree for Merge Sort For the original problem, we have a cost of cn, plus two subproblems each of size (n/2) and running time T(n/2). cn T(n/2) Each of the size n/2 problems has a cost of cn/2 plus two subproblems, each costing T(n/4). cn cn/2 T(n/4) Cost of divide and merge. Cost of sorting subproblems. T(n) = 2T(n/2) + cn

Comp 550 Recursion Tree for Merge Sort Continue expanding until the problem size reduces to 1. cn cn/2 cn/4 cccccc lg n cn Total : cnlgn+cn

Comp 550 Recursion Tree for Merge Sort Continue expanding until the problem size reduces to 1. cn cn/2 cn/4 cccccc Each level has total cost cn. Each time we go down one level, the number of subproblems doubles, but the cost per subproblem halves  cost per level remains the same. There are lg n + 1 levels, height is lg n. (Assuming n is a power of 2.) Can be proved by induction. Total cost = sum of costs at each level = (lg n + 1)cn = cnlgn + cn =  (n lgn).

UNC Chapel HillZ. Guo Recursion Trees and Recurrences Useful even when a specific algorithm is not specified – For T(n) = 2T(n/2) + n 2, we have

UNC Chapel HillZ. Guo Recursion Trees T(n) =  (n 2 )

Comp 550 Recursion Trees – Caution Note Recursion trees only generate guesses. – Verify guesses using substitution method. If careful when drawing out a recursion tree and summing the costs, can be used as direct proof.

UNC Chapel HillZ. Guo Recursion Trees Exercise T(n) = T(n/2) + T(n/4) + n 2

Comp 550 The Master Method Based on the Master theorem. “Cookbook” approach for solving recurrences of the form T(n) = aT(n/b) + f(n) a  1, b > 1 are constants. f(n) is asymptotically positive. T(n) is defined for nonnegative integers n/b may not be an integer, but we ignore floors and ceilings. Why?

Comp 550 The Master Theorem Theorem 4.1 Let a  1 and b > 1 be constants, let f(n) be a function, and Let T(n) be defined on nonnegative integers by the recurrence T(n) = aT(n/b) + f(n), where we can replace n/b by  n/b  or  n/b . T(n) can be bounded asymptotically in three cases: 1.If f(n) = O(n log b a–  ) for some constant  > 0, then T(n) =  ( n log b a ). 2.If f(n) =  (n log b a ), then T(n) =  ( n log b a lg n). 3.If f(n) =  (n log b a+  ) for some constant  > 0, and if, for some constant c < 1 and all sufficiently large n, we have a·f(n/b)  c f(n), then T(n) =  (f(n)). Theorem 4.1 Let a  1 and b > 1 be constants, let f(n) be a function, and Let T(n) be defined on nonnegative integers by the recurrence T(n) = aT(n/b) + f(n), where we can replace n/b by  n/b  or  n/b . T(n) can be bounded asymptotically in three cases: 1.If f(n) = O(n log b a–  ) for some constant  > 0, then T(n) =  ( n log b a ). 2.If f(n) =  (n log b a ), then T(n) =  ( n log b a lg n). 3.If f(n) =  (n log b a+  ) for some constant  > 0, and if, for some constant c < 1 and all sufficiently large n, we have a·f(n/b)  c f(n), then T(n) =  (f(n)). We’ll return to recurrences as we need them… If none of these three cases apply, you’re on your own.

UNC Chapel HillZ. Guo Master Theorem Let a  1 and b > 1 be constants and let T(n) be the recurrence T(n) = a T(n/b) + c n k defined for n  If a > b k, then T(n) =  ( n log b a ). 2. If a = b k, then T(n) =  ( n k lg n ). 3. If a < b k, then T(n) =  ( n k ).

UNC Chapel HillZ. Guo Examples T(n) = 16T(n/4) + n – a = 16, b = 4, thus n log b a = n log 4 16 =  (n 2 ) – f(n) = n = O(n log  ) where  = 1  case 1. – Therefore, T(n) =  (n log b a ) =  (n 2 ) T(n) = T(3n/7) + 1 – a = 1, b=7/3, and n log b a = n log 7/3 1 = n 0 = 1 – f(n) = 1 =  (n log b a )  case 2. – Therefore, T(n) =  (n log b a lg n) =  (lg n)

UNC Chapel HillZ. Guo Examples (Cont.) T(n) = 3T(n/4) + n lg n – a = 3, b=4, thus n log b a = n log 4 3 = O(n ) – f(n) = n lg n =  (n log  ) where   0.2  case 3. – Therefore, T(n) =  (f(n)) =  (n lg n) T(n) = 2T(n/2) + n lg n – a = 2, b=2, f(n) = n lg n, and n log b a = n log 2 2 = n – f(n) is asymptotically larger than n log b a, but not polynomially larger. The ratio lg n is asymptotically less than n  for any positive . Thus, the Master Theorem doesn’t apply here.

UNC Chapel HillZ. Guo Exercises Use the Master Method to solve the following:  T(n) = 4T(n/2) + n  T(n) = 4T(n/2) + n 2  T(n) = 4T(n/2) + n 3 Try this if you finished the ones above: T(n) = 4T(n/2) + n 2 /lgn

An Example of “on your own” T(n) = – (A) T(n) = O(n lgn) – (B) T(n) = O(n) – (C) T(n) = O( ) – (D) T(n) = O(n lglgn) UNC Chapel HillZ. Guo