Design and Analysis of Algorithms Quicksort Haidong Xue Summer 2012, at GSU

Review of insertion sort and merge sort Insertion sort – Algorithm – Worst case number of comparisons = O(?) Merge sort – Algorithm – Worst case number of comparisons = O(?)

Sorting Algorithms AlgorithmWorst TimeExpected TimeExtra Memory Insertion sortO(1) Merge sortO(n) Quick sortO(1) Heap sortO(1)

Quicksort Algorithm Input: A[1, …, n] Output: A[1,.., n], where A[1]<=A[2]…<=A[n] Quicksort: 1.if(n<=1) return; 2.Choose the pivot p = A[n] 3.Put all elements less than p on the left; put all elements lager than p on the right; put p at the middle. (Partition) 4. Quicksort(the array on the left of p) 5. Quicksort(the array on the right of p)

Quicksort Algorithm Quicksort example Current pivots Previous pivots Quicksort Hi, I am nothing Nothing Jr. Nothing 3rd

Quicksort Algorithm More detail about partition Input: A[1, …, n] (p=A[n]) Output: A[1,…k-1, k, k+1, … n], where A[1, …, k-1] A[k], A[k]=p Partition: 1.t = the tail of smaller array 2.from i= 1 to n-1{ if(A[i]<p) { exchange A[t+1] with A[i]; update t to the new tail; } 3.exchange A[t+1] with A[n];

Quicksort Algorithm Partition example tail Exchange 2 with A[tail+1] Do nothing

Quicksort Algorithm Partition example tail Exchange 1 with A[tail+1] tail Exchange 3 with A[tail+1] Do nothing tail Do nothing Final step: exchange A[n] with A[tail+1]

Quicksort Algorithm The final version of quick sort: Quicksort(A, p, r){ if(p<r){ //if(n<=1) return; q = partition(A, p, r); //small ones on left; //lager ones on right Quicksort(A, p, q-1); Quicksort(A, q+1, r); }

Analysis of Quicksort Time complexity – Worst case – Expected Space complexity - extra memory – 0 = O(1)

Analysis of Quicksort

Strict proof of the worst case time complexity

Strict proof of the expected time complexity

Java implementation of Quicksort Professional programmers DO NOT implement sorting algorithms by themselves We do it for practicing algorithm implementation techniques and understanding those algorithms Code quicksort – Sort.java // the abstract class – Quicksort_Haydon.java // my quicksort implementation – SortingTester.java // correctness tester

Design and Analysis of Algorithms Review on time complexity, “in place”, “stable” Haidong Xue Summer 2012, at GSU

Time complexity of algorithms Execution time? – Pros: easy to obtain; Cons: not accurate Number Instructions? – Pros: very accurate; Cons: calculation is not straightforward Number of certain operations? – Pros: easy to calculate, generally accurate; Cons: not very calculate Asymptotic Notations Pros? Cons?

Time complexity of algorithms In the worst case for(int i=0; i<A.length; i++) C1: cost of “assign a value ” C2: cost of “<” C3: cost of “increment by 1” C4: cost of “==” C5: cost of “return” C1 + (length+1)*C2+ length*C3 length*C4 0*C5 C5 Assuming C4 >> C1, C2, C3, C5 Worst case T(n) = n*C4 =  (n) Worst case T(n) = C1+C2+ C5 + n(C2+C3+C4)+ =  (n)

“in place” and “stable” in sorting algorithms AlgorithmWorst TimeExpected TimeExtra MemoryStable Insertion sortO(1) (in place)Can be Merge sortO(n)Can be Quick sortO(1) (in place)Can be Heap sortO(1) (in place)No Stable Not stable

Design and Analysis of Algorithms Heapsort Haidong Xue Summer 2012, at GSU

Max-Heap A complete binary tree, and … Yes No every level is completely filled, except possibly the last, which is filled from left to right

Max-Heap Satisfy max-heap property: parent >= children Since it is a complete tree, it can be put into an array without lose its structure information.

Max-Heap

Max-Heap Use an array as a heap For element at i: Parent index =parent(i)= floor(i/2); Left child index = left(i)=2*i; Right child index =right(i)=2*i +1 Last non-leaf node = floor(length/2) i=3 2*i = 6 2*i+1=7 floor(i/2)=floor(1.5)=1 floor(length/2)=4

Max-Heapify Input: A compete binary tree rooted at i, whose left and right sub trees are max-heaps; last node index Output: A max-heap rooted at i. Algorithm: 1. Choose the largest node among node i, left(i), right(i). 2. if(the largest node is not i){ – Exchange i with the largest node – Max-Heapify the affected subtree }

Max-Heapify Example

Heapsort for a heap Input: a heap A Output: a sorted array A Algorithm: 1. Last node index l = A’s last node index 2. From the last element to the second{ exchange (current, root); l--; Max-Heapify(A, root, l); }

Heapsort example

Heapsort example

Heapsort example

Heapsort example

Heapsort example

Heapsort example

Heapsort example

Array -> Max-Heap Input: a array A Output: a Max-Heap A Algorithm: Considering A as a complete binary tree, from the last non-leaf node to the first one{ Max-Heapify(A, current index, last index); }

Build Heap Example

Heapsort Input: array A Output: sorted array A Algorithm: 1. Build-Max-Heap(A) 2. Last node index l = A’s last node index 3. From the last element to the second{ exchange (current, root); l--; Max-Heapify(A, root, l); } Let’s try it

Analysis of Heapsort Input: array A Output: sorted array A Algorithm: 1. Build-Max-Heap(A) 2. Last node index l = A’s last node index 3. From the last element to the second{ exchange (current, root); l--; Max-Heapify(A, root, l); } O(n) or O(nlgn) O(nlgn) O(n)

Design and Analysis of Algorithms Non-comparison sort (sorting in linear time) Haidong Xue Summer 2012, at GSU

Comparison based sorting Algorithms that determine sorted order based only on comparisons between the input elements AlgorithmWorst TimeExpected TimeExtra MemoryStable Insertion sortO(1) (in place)Can be Merge sortO(n)Can be Quick sortO(1) (in place)Can be Heap sortO(1) (in place)No What is the lower bound?

Lower bounds for comparison based sorting <? Y N Y …… ………………………….. Done N Y For n element array, how many possible inputs are there? Factorial of n n! What is the shortest tree can have n! leaves? As a result ……

Sorting in linear time Can we sort an array in linear time? Yes, but not for free E.g. sort cards with 13 slots What if there are more than one elements in the same slot?

Counting Sort Input: array A[1, …, n]; k (elements in A have values from 1 to k) Output: sorted array A Algorithm: 1.Create a counter array C[1, …, k] 2.Create an auxiliary array B[1, …, n] 3.Scan A once, record element frequency in C 4.Calculate prefix sum in C 5.Scan A in the reverse order, copy each element to B at the correct position according to C. 6.Copy B to A

Counting Sort A: C: B: Position indicator:

Analysis of Counting Sort Input: array A[1, …, n]; k (elements in A have values from 1 to k) Output: sorted array A Algorithm: 1.Create a counter array C[1, …, k] 2.Create an auxiliary array B[1, …, n] 3.Scan A once, record element frequency in C 4.Calculate prefix sum in C 5.Scan A in the reverse order, copy each element to B at the correct position according to C. 6.Copy B to A Time O(n) O(k) O(n) O(n+k)=O(n) (if k=O(n)) Space O(k) O(n) O(n+k)=O(n) (if k=O(n))

Radix-Sort Input: array A[1, …, n]; d (number of digit a element has) Output: sorted array A Algorithm: for each digit{ use a stable sort to sort A on a digit } T(n)=O(d(n+k))

Summary AlgorithmWorst TimeExpected TimeExtra MemoryStable Insertion sortO(1) (in place)Yes Merge sortO(n)Yes Quick sortO(1) (in place)Yes Heap sortO(1) (in place)No Counting sortYes Design strategies: Divide and conquer Employ certain special data structure Tradeoff between time and space

Knowledge tree Algorithms Analysis Design Algorithms for classic problems Classic data structure Asymptotic notations Probabilisti c analysis Sorting Shortest path Matrix multiplication … Divide & Conquer Greedy Dynamic Programming O(), o(),  (),  (),  () Heap, Hashing, Binary Tree, RBT, …. Quicksort, Heapsort, Mergesort, … ……… … … … ………… … … … … … ……… …