Divide and Conquer Strategy
General Concept of Divide & Conquer Given a function to compute on n inputs, the divide- and-conquer strategy consists of: splitting the inputs into k distinct subsets, 1kn, yielding k subproblems. solving these subproblems combining the subsolutions into solution of the whole. if the subproblems are relatively large, then divide & conquer is applied again. if the subproblems are small, the are solved without splitting.
Divide & Conquer
The Divide and Conquer Algorithm Divide_Conquer(problem P) { if Small(P) return S(P); else { divide P into smaller instances P1, P2, …, Pk, k1; Apply Divide_Conquer to each of these subproblems; return Combine(Divide_Conque(P1), Divide_Conque(P2),…, Divide_Conque(Pk)); }
Divde_Conquer recurrence relation The computing time of Divide_Conquer is T(n) is the time for Divide_Conquer on any input size n. g(n) is the time to compute the answer directly (for small inputs) f(n) is the time for dividing P and combining the solutions. n small otherwise
Three Steps of The Divide and Conquer Approach The most well known algorithm design strategy: Divide the problem into two or more smaller subproblems. Conquer the subproblems by solving them recursively. Combine the solutions to the subproblems into the solutions for the original problem.
A Typical Divide and Conquer Case a problem of size n subproblem 1 of size n/2 subproblem 2 of size n/2 a solution to subproblem 1 a solution to subproblem 2 a solution to the original problem
An Example: Calculating a0 + a1 + … + an-1 ALGORITHM RecursiveSum(A[0..n-1]) //Input: An array A[0..n-1] of orderable elements //Output: the summation of the array elements if n > 1 return ( RecursiveSum(A[0.. n/2 – 1]) + RecursiveSum(A[n/2 .. n– 1]) ) Efficiency: (for n = 2k) A(n) = 2A(n/2) + 1, n >1 A(1) = 0;
Code int sumArray(int anArray[],int start,int end){ if(start==end) return anArray[start]; if(start<end){ int mid=(start+end)/2; int lsum=sumArray(anArray,start,mid-1); int rsum=sumArray(anArray,mid+1,end); return lsum+rsum+anArray[mid]; } return 0;
Bubble Sort BubbleSort(array A, int n) for i 0 to n-1 for j 0 to n-2-i if A[j] > A[j+1] swap(A[j],A[j+1]) Animate
How many swaps does bubble sort make? O(log n) O(n) O(n log n) O(n2)
In Practice Bubble Sort in practice: “The bubble sort has little to recommend it, except a catchy name” – D. Knuth “the generic bad algorithm” – The Jargon File Poor interaction with CPUs – causes cache misses Number of swaps is O(n2)
Selection Sort SelectionSort(array A, int n) for i ← 0 to n-2 do min ← i for j ← (i + 1) to n-1 do if A[j] < A[min] min ← j swap A[i] and A[min] Animate
How many swaps does selection sort make? O(log n) O(n) O(n log n) O(n2)
In Practice Selection Sort in practice: On average, takes about 40% the time of Bubble Sort Requires O(n) swaps Very fast on small lists (< ~20) – better than the O(n log n) sorts Good as a base case
Insertion Sort insertionSort(array A, int n) for i = 1 to n-1 do value = A[i] j = i-1 while j >= 0 and A[j] > value do A[j + 1] = A[j] j = j-1 A[j+1] = value Animate
How many swaps does insertion sort make? O(log n) O(n) O(n log n) O(n2)
In Practice Insertion Sort in practice: On average, takes about 20% the time of Bubble Sort Requires O(n2) swaps Finishes a sorted list in O(n) time Very fast on small lists (< ~20) – better than the O(n log n) sorts Good as a base case
Sorting Characteristics In-place sort: A sort that does not require an extra array to hold “scratch work” Stable sort: A sort that preserves the relative order of elements with the same value Content sensitive: A sort that will tailor its execution to the data – runs faster for certain types of data
Comparison Runtime In place Stable Content Sensitive Bubble O(n2) Yes Selection No Insertion
Merge Sort
What is the runtime of bubble sort that sorts only the first 20 elements of a list of n elements? (1) (log n) (n) (n log n) (n2)
MergeSort MergeSort(array A, int i, int j) (1) if (j – i < 20) (2) InsertionSort(A[i…j], j-i+1) (3) else (4) int mid (i+j)/2 (5) MergeSort(A, i, mid) (6) MergeSort(A, mid+1, j) (7) Merge(A,i,j)
Merge 1 1 3 2 4 3 6 4 2 5 5 6 7 7 8 8 1 2 3 4 5 6 7 8
Merge Function O(1) O(n) O(n) O(n) O(n) Merge(array A, int i, int j, int mid) (1) array tmp; (2) p1 i, p2 mid+1, k 0 (3) while (p1 mid && p2 j) (4) if (A[p1] < A[p2]) (5) tmp[k++] = A[p1++] (6) else (7) tmp[k++] = A[p2++] (8) while (p1 mid) (9) tmp[k++] = A[p1++] (10) while (p2 j) tmp[j++] = A[p2++] copy tmp to A O(1) O(n) O(n) O(n) O(n)
MergeSort (1) T(n/2) T(n/2) cn T(n) = 2T(n/2) + cn Let T(n) be the runtime on an array of size n. MergeSort MergeSort(array A, int i, int j) (1) n j – i (2) if (n < 20) (3) InsertionSort(A[i…j], j-i+1) (4) else (5) int mid (i+j)/2 (6) MergeSort(A, i, mid) (7) MergeSort(A, mid+1, j) (8) Merge(A,i,j) (1) T(n/2) T(n/2) cn T(n) = 2T(n/2) + cn
Bounding T(n) T(n) = 2T(n/2) + f(n) where f(n) = O(n) 2T(n/2) + cn = 2(2T(n/4) + c(n/2)) + cn = 4T(n/4) + 2cn = 4(2T(n/8) + c(n/4)) + 2cn = 8T(n/8) + 3cn = … = 2iT(n/2i ) + icn
Bounding T(n) T(n) = 2iT(n/2i) + icn, T(1) = c’ If i = log2 n then: T(n) = 2i T(n/2i) + icn = 2log n T(n/2log n) + cnlog2n = nT(1) + cnlog2n = nc’ + cnlog2n = O(n log n)
Merge Sort Characteristics Stable: Yes – no reason to swap elements unless the first is larger In-place: No – Merge requires an extra array Can you write an O(n) in-place merge? Content Sensitive: No – will behave in exactly the same way, regardless of execution
Comparison Runtime In place Stable Content Sensitive Bubble O(n2) Yes Selection No Insertion Merge O(n log n)
QuickSort
QuickSort QuickSort(array A, int i, int j) if i < j q Partition(A, i, j) QuickSort(A, i, q-1) QuickSort(A, q+1, j)
Partition Algorithm: Example 3 3 8 8 1 8 4 1 2 7 7 4 5 8 6 7 2 8 5 6 7 6 Return this index
Partition O(n) Partition(array A, int i, int j) int pivot A[j] for r i to j-1 if A[r] ≤ pivot p p+1 swap(A[p], A[r]) swap(A[p+1], p[j]) return p+1 O(n)
QuickSort QuickSort(array A, int i, int j) if i < j q Partition(A, i, j) QuickSort(A, i, q-1) QuickSort(A, q+1, j) T(n) = 2T(???) + cn
Runtime Suppose the list is already sorted: T(n) = T(0) + T(n-1) + cn = 2T(0) + T(n-2) + cn + c(n-1) = 3T(0) + T(n-3) + cn + c(n-1) + c(n-2) = ... = iT(0) + T(n-i) + cn + … + c(n-i+1) = nT(0) + T(0) + cn + .. + 1 = O(n2)
QuickSort QuickSort(array A, int i, int j) if i < j q Partition(A, i, j) QuickSort(A, i, q-1) QuickSort(A, q+1, j) What is the recurrence if we manage to get the median as the pivot (in O(1) time)?
What is the recurrence for QuickSort if we could pick the median as the pivot in O(n) time? T(n) = 3T(n/2) + O(n) T(n) = T(n/3) + O(n) T(n) = 2T(n/4) + O(n) T(n) = T(n1/2) + O(n) T(n) = 2T(n/2) + O(n)
Runtime Suppose we can pick the “middle” element each time? T(n) = 2T(n/2) + cn = O(n log n) Like merge sort
Picking a Median Can we pick a median for the pivot? Sure – sort and pick the “middle”: O(n log n) Now what is the recurrence?
What is the recurrence for QuickSort if we pick the median by sorting? T(n) = T(n)/2 + O(n) T(n) = 2T(n/2) + O(n) T(n) = 3T(n/2) + O(n) T(n) = T(n/2) + O(nlog n) T(n) = 2T(n/2) + O(nlog n) T(n) = 3T(n/2) + O(nlog n)
Picking a Median Can we pick a median for the pivot? Sure – sort and pick the “middle”: O(n log n) Problem: If T(n) = 2T(n/2) + cn log n then T(n) = (n log n) We need to pick the median in O(n) time if we want the O(n log n) quick sort time This is possible – but difficult We will tackle this next week
Picking a Pivot Suppose the pivot were a random value from the list We can show that the average runtime for QuickSort is then O(n log n) So we do exactly that: Pick a random element Swap the value to the end Proceed with Partition
Partition Partition(array A, int i, int j) swap(A[j], A[random(i,j)]) int pivot A[j] int p i-1 for r i to j-1 if A[r] ≤ pivot p p+1 swap(A[p], A[r]) swap(A[p+1], p[j]) return p+1
Quick Sort Characteristics Stable: No – partition shuffles everything around In-place: Yes Content Sensitive: No (depends on pivot choice)
Comparison Runtime In place Stable Content Sensitive Bubble O(n2) Yes Selection No Insertion Merge O(n log n) QuickSort O(n log n)* * Average case in standard implementation – worst case of O(n2)