CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Recurrences

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Recurrences Coming up Merge Sort (Chap 2.3) Recurrences (Chap )

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Merge Sort Merge Sort : divide-and-conquer approach Insertion Sort: Incremental approach The Divide-and-Conquer Paradigm: Divide: Divide the problem into a number of subproblems. Conquer: If the subproblem sizes are small enough, solve them directly. Otherwise solve them recursively. Combine: Combine the solutions to the subproblems into the solution for the original problem.

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Merge Sort Suppose there are some people called Mr. MergeSort. They are identical. They don’t know how to do sorting. But each of them has a secretary called Mr. Merge, who can merge 2 sorted sequences into one sorted sequence.

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Merge Sort At the beginning, a Mr. MergeSort is called to sort: Then 2 other Mr. MergeSorts are called to sort: Both of them say “Still complicated! I’ll split them and call other Mr. MergeSorts to handle.” Then 4 other Mr. MergeSorts are called to sort: All of them say “Still complicated! I’ll split them and call other Mr. MergeSorts to handle.” Then 8 other Mr. MergeSorts are called to sort: “So complicated!!, I’ll split them and call other Mr. MergeSorts to handle.” All of them say ‘This is easy. No need to do anything.’

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Merge Sort Then the first Mr. MergeSort succeeds and returns. Then each of the 2 Mr. MergeSorts returns the merged numbers. Then the 4 Mr. MergeSorts returns the merged numbers. Then the 8 Mr. MergeSorts return All of them say ‘This is easy. No need do anything.’ Both Mr. MergeSorts call their secretaries Mr. Merge to merge the returned numbers The 4 Mr. MergeSorts call their secretaries Mr. Merge to merge the returned numbers The first Mr. MergeSort calls his secretary Mr. Merge to merge the returned numbers

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Merge Sort MERGE-SORT (A,p,r) 1if p < r 2Then q   (p+r)/2  3MERGE-SORT(A,p,q) 4MERGE-SORT(A,q+1,r) 5MERGE (A,p,q,r) The MERGE-SORT(A,p,r) procedure sorts the elements A[p,..r]: A = p th r th  x  : “Floor” The least integer greater than x  x  :“Ceiling” The greatest integer less than x MERGE-SORT (A,p,r) 1if p < r 2Then q   (p+r)/2  3MERGE-SORT(A,p,q) 4MERGE-SORT(A,q+1,r) 5MERGE (A,p,q,r)

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Merge Sort The MERGE(A,p,q,r) procedure merges 2 sorted sequences: A[p..q] and A[q+1..r] A = p th r th q th Step 1 : Copy A[p..q], A[q+1..r] to 2 temporary arrays L and R. Step 2 :Adds a largest value,  (a sentinel: ending condition), to each of L and R. Step 3 :Continuously remove the smallest one from L and R back to A until finished L= R=  L=  R= A = p th r th  L=  R= A = p th r th  L=  R= A = p th r th  L=  R= A = p th r th  L= 3 6  R= A = p th r th  L= 6  R= A = p th r th 5 7  L= 6  R= A = p th r th 7  L= 6  R= A = p th r th 7  L=  R= A = p th r th  L=  R=

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Merge Sort MERGE (A,p,q,r) 1n 1  q-p+1 2n 2  r-q 3create L[1..n 1 +1], R[1..n 2 +1] 4for i  1 to n 1 5do L[i]  A[p+i-1] 6for j  1 to n 2 7do R[j]  A[q+j] 8L[n 1 +1]   9R[n 2 +1]   10i  1 11j  1 12for k  p to r 13do if L[i]  R[j] 14then A[k]  L[i] 15i  i+1 16else 17j  j+1 Step 1 : Copy A[p..q], A[q+1..r] to 2 temporary arrays L and R. Step 2 :Adds a largest value,  (a sentinel: ending condition), to each of L and R. Step 3 :Continuously remove the smallest one from L and R back to A until finished. The MERGE(A,p,q,r) procedure merges 2 sorted sequences: A[p..q] and A[q+1..r]

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Analysis of MERGE Procedure Let n=n 1 + n 2 Line 1-3 and 8-11 takes constant time. So,  (1). Line 4-5 and Line 6-7 takes  (n 1 +n 2 ) =  (n) time. In line 12-17, the loop iterates n times, each of which takes constant time. So,  (n). Conclusion: The MERGE procedure runs in  (1) +  (n) +  (n) =  (n) time. MERGE (A,p,q,r) 1n 1  q-p+1 2n 2  r-q 3create L[1..n 1 +1], R[1..n 2 +1] 4for i  1 to n 1 5do L[i]  A[p+i-1] 6for j  1 to n 2 7do R[j]  A[q+j] 8L[n 1 +1]   9R[n 2 +1]   10i  1 11j  1 12for k  p to r 13do if L[i]  R[j] 14then A[k]  L[i] 15i  i+1 16else 17j  j+1

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Analysis of Merge Sort MERGE-SORT has a recursive call to itself, plus a call to MERGE. The Running time: Recurrence Equation MERGE-SORT (A,p,r) 1if p < r 2Then q   (p+r)/2  3MERGE-SORT(A,p,q) 4MERGE-SORT(A,q+1,r) 5MERGE (A,p,q,r)  (1)if n=1 T  n/2  + T  n/2  +  (n)if n>1 T(n) = Or simplified: cif n=1 2T(n/2)+cnif n>1 T(n) = Then, what is the complexity of Merge Sort? To sort A[1..n] using Merge Sort, we call MERGE-SORT(A,1,n)

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences cif n=1 2T(n/2)+cnif n>1 T(n) = Solving Recurrences When an algorithm contains recursive call(s) to itself, (eg. Divide-and-conquer approaches), its running time can often be described by a recurrence equation. 3 methods to solve recurrences: Recursion-tree method Can directly prove or help to guess MI Substitution method guess a bound + prove by mathematical induction Master method Based on the Master Theorem => T(n) =  (?)

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Recursion-tree method) T(n) Expanding the recursion tree: Recursion-tree method cn T(n/2) cn cn/2 T(n/4) cn/2 T(n/4) cif n=1 2T(n/2)+cnif n>1 T(n) =

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Recursion-tree method) Fully Expanded recursion tree: cn cn/2 cn/4 cn/2 cn/4 c*1 cn n Total: cn lg n + cn ie. T(n) =  (n lg n)

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Substitution method) Step 1. Guess the form of the solution Example. Given: Substitution method 1if n=1 2T(  n/2  )+nif n>1 T(n) = We guess that T(n)=O(n lg n) To prove this we need to show “There exists positive constants c and n 0 such that...T(n)  cn lg n, for all n  n 0 ” Step 2 MI

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Substitution method) Step 2. Use mathematical induction to find the constants and prove. Assume it holds for  n/2 , ie. There exists positive constants c and n 0 such that T(  n/2  )  c  n/2  lg (  n/2  ). By substitution: T(n)  2 (c  n/2  lg (  n/2  )) + n  cn lg (n/2) + n = cn (lg(n)-lg(2))+n = cn (lg(n)-1)+n = cn lg n -cn + n  cn lg n T(n)=O(n lg n) “There exists a positive constants c and n 0 such that T(n)  cn lg n, for all n  n 0 ” 1if n=1 2T(  n/2  )+nif n>1 T(n) = This holds as long as c  1 MI

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Substitution method) Step 2 (cont’d). Then we need to find n 0. Try n 0 =1When n=1,T(n)= 1 cn lg n = (c*1) lg (1) = c * 0 = 0 => T(n)  cn lg n ? Try n 0 =2When n=2,T(n) = 2T(1) + 2 = 2(1) + 2 = 4 cn lg n = (c*2) lg (2) = c*2*1 = 2c => T(n)  cn lg n ? Try n 0 =3When n=3,T(n) = 2T(1) + 3 = 2(1) + 3 = 5 cn lg n = (c*3) lg (3) = 3*1.585*c = 4.755*c => T(n)  cn lg n ? Since n 0 =2 and n 0 =3 form the base cases for all n>=2, we thus completed the prove: T(n)  cn lg n is true when c>=2 and n>=2. T(n)=O(n lg n) “There exists a positive constants c and n 0 such that T(n)  cn lg n, for all n  n 0 ” 1if n=1 2T(  n/2  )+nif n>1 T(n) = MI

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Master method) A cookbook method For solving recurrences of the form: T(n) = a T(n/b) + f(n) Where a >= 1 and b > 1 and f(n) is an asymptotically positive function These algorithms work recursively by dividing a problem of size n into a subproblems, each of size n/b. f(n) = cost of dividing the problem and combining the results. We omit floors and ceilings (eg. interpret n/b to mean either  n/b  or  n/b . ) Master method

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Master method) Master Theorem Let T(n) be defined as T(n) = a T(n/b) + f(n) where a >=1 and b > 1. Then If f(n) = O(n log b a-  ) for some constant  >0, then T(n) =  (n log b a ) If f(n) =  (n log b a ), then T(n) =  (n log b a lg n) If f(n) =  (n log b a+  ) for some constant  >0, and if a f (n/b) <= c f(n) for some constant c <1 and all sufficiently large n, then T(n) =  (f(n)) The master method can be used only if f(n) satisfies any of the 3 cases.

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Master method) T(n) = 9T(n/3) + n a = 9, b = 3 => n log b a = n log 3 9 = n 2 Sincef(n)= n = n (2-1) = O(n log  ) Where  =1 => Case 1 => T(n) =  (n log b a ) => T(n) =  (n 2 ) Example 1: T(n) = T(2n/3) + 1 a = 1, b = 3/2 => n log b a = n log 3/2 1 = n 0 = 1 Since f(n) = 1 =  (n log b a ) => Case 2 => T(n) =  (n log b a lg n) => T(n) =  (lg n) Example 2: Master Theorem Master Theorem For T(n) = a T(n/b) + f(n) where a >=1 and b > 1. If f(n) = O(n log b a-  ) for some constant  >0, then T(n) =  (n log b a ) If f(n) =  (n log b a ), then T(n) =  (n log b a lg n) If f(n) =  (n log b a+  ) for some constant  >0,.. then T(n) =  (f(n))

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Solving Recurrence (Master method) Example 3: T(n) = 3T(n/4) + n lg n a=3, b=4 => n log b a = n log 4 3 = n Since f(n) = n lg n When n  2, f(n)  n lg 2 = n = n 1 = n , where  =0.201 => f(n) =  (n log b a+  ) a f(n/b) = a (n/b * lg(n/b)) = 3 (n/4 * lg (n/4)) = 3/4 n lg(n/4) = 3/4 (n lg(n)) - 3/4 (n lg(4))  3/4 f(n) => a f(n/b)  c f(n) where c=3/4. => Case 3. ie. T(n) =  (f(n)) =  (n lg n) Master Theorem Master Theorem For T(n) = a T(n/b) + f(n) where a >=1 and b > 1. If f(n) = O(n log b a-  ) for some constant  >0, then T(n) =  (n log b a ) If f(n) =  (n log b a ), then T(n) =  (n log b a lg n) If f(n) =  (n log b a+  ) for some constant  >0, and if a f (n/b) <= c f(n) for some constant c <1 and all sufficiently large n, then T(n) =  (f(n))

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 4. Recurrences Recurrences Summary The Divide-and-Conquer Paradigm Merge Sort ( MERGE-SORT Procedure + MERGE Procedure ) Analysis of MERGE:  (n) Analysis of MERGE-SORT: A recurrence problem  (n lg n)) 3 Methods to solve recurrences: Recursion-tree method Substitution method Master method