The father of algorithm analysis Donald E. Knuth (1938---) The father of algorithm analysis popularizing asymptotic notation. The author of TAOCP -- The Art of Computer Programming Volume 1: Fundamental Algorithms Volume 2: Seminumerical Algorithms Volume 3: Sorting and Searching (among tons of others) “ I have been a happy man ever since January 1, 1990, when I no longer had an email address.” 4/20/2019 IT 279

The Art of Computer Programming TAOCP – The Art of Computer Programming Volume 1: Fundamental Algorithms Basic Concepts and Information Structures Volume 2: Seminumerical Algorithms Random numbers and Arithmetic Volume 3: Sorting and Searching Sorting and Searching Volume 4: Combinatorial Algorithms Combinatorial Searching and Recursion Volume 5: Syntactical Algorithms Lexical Scanning and Parsing Techniques Volume 6: Theory of Languages Mathematical Linguistics Volume 7: Compiler Programming Language Translation Donald E. Knuth Highlighted in the preface of Vol. 1 4/20/2019 IT 279

“An algorithm must be seen to be believed” -- Donald E. Knuth -- Sorting algorithms 4/20/2019 IT 279

Sorting: Arranging items into a certain order. Items can be stored in an one dimensional array or linked list, or trees. We prefer using one dimensional array Why? 4/20/2019 IT 279

Insert an item into a sorted the right place. Insertion Sort: Insert an item into a sorted the right place. sorted 1 3 4 6 8 9 10 12 7 4/20/2019 IT 279

Insert an item into the right place. Insertion Sort: Insert an item into the right place. sorted 1 3 4 6 8 9 10 12 7 2 4/20/2019 IT 279

Insert an item into the right place. Insertion Sort: Insert an item into the right place. sorted 1 3 4 6 8 9 10 12 7 2 7 8 9 10 12 7 4/20/2019 IT 279

Insert an item into the right place. 6 12 3 1 10 8 9 7 2 6 12 3 1 10 8 Insertion Sort: Insert an item into the right place. 6 12 3 1 10 8 9 7 2 6 12 3 1 10 8 9 7 2 3 6 12 1 10 8 9 7 2 1 3 6 12 10 8 9 7 2 1 3 6 10 12 8 9 7 2 1 3 6 8 10 12 9 7 2 1 3 6 8 9 10 12 7 2 1 3 6 7 8 9 10 12 2 1 2 3 6 7 8 9 10 12 4/20/2019 IT 279

Select the minimum item into the end of the sorted segment. Selection Sort: Select the minimum item into the end of the sorted segment. sorted 1 3 4 9 12 7 8 10 the minimum one 1 3 4 7 12 9 8 10 the minimum one 4/20/2019 IT 279

Selection Sort: Select the minimum item into the end of the sorted segment. 6 12 3 1 10 8 9 7 2 1 12 3 6 10 8 9 7 2 1 2 3 6 10 8 9 7 12 1 2 3 6 10 8 9 7 12 1 2 3 6 10 8 9 7 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12 4/20/2019 1 2 3 6 7 8 9 10 12 IT 279

Analysis of Insertion/Selection Sorts: T(n) = 1 +2 + 3 + ... + n = O(n2) Shell Sort D. L. Shell, (1959) O(n3/2) O(n log n) Lower bound of sorting algorithms 4/20/2019 IT 279

Incremental insertion (selection) sort. Shell Sort: h = 4 7 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4 3 5 8 14 6 1 16 2 7 10 9 13 11 15 12 4 3 1 8 14 6 5 16 2 7 10 9 13 11 15 12 4 3 1 8 14 6 5 9 2 7 10 12 13 11 15 16 4 3 1 8 2 6 5 9 4 7 10 12 13 11 15 16 14 4/20/2019 IT 279

Shell Sort: hk-sorted h = 4 4-sorted 2-sorted 1-sorted 3 1 8 2 6 5 9 4 7 10 12 13 11 15 16 14 4-sorted 3 1 6 2 7 4 8 5 9 10 11 13 12 14 16 15 2-sorted 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1-sorted 4/20/2019 IT 279

Incremental insertion Shell Sort: Incremental insertion with Increment sequence, hk-sorted 1-sorted ht-sorted ht-1-sorted hk-1-sorted  (implies) hk-sorted hk-1-sorted 10 9 30 20 hk-sorted Shell suggested: Not a good suggestion, worst case: O(n2) 4/20/2019 IT 279

Shell Sort: using Shell’s suggestion Worst case analysis: O(n2) Idea: Right before the final sort, let the smallest n/2 be distributed in the even position. e.g. 2-sorted 9 1 10 2 11 3 12 4 13 5 14 6 15 7 16 8 1 2 3 4 5 6 7 8 final sort 4/20/2019 IT 279

Shell Sort: using Shell’s suggestion What kind of initial array will result in such 2-sorted? 9 1 10 2 11 3 12 4 13 5 14 6 15 7 16 8 13 5 12 1 14 4 10 7 11 3 9 8 15 6 16 2 Let Ah[s] = A[s], A[s+h], A[s+2h], .... odd positions Ah[1] Ah[0] even positions 5 13 1 11 3 Ah[2] 12 7 4 15 9 10 8 Ah[3] 6 2 14 16 Ah[5] Ah[6] Ah[4] Ah[7] elements will never cross the even-/odd-cell before the final sort 8-sort , 4-sort , 2-sort 4/20/2019 IT 279

bad strategy Shell sort algorithm 7 5 8 14 6 12 9 2 16 1 i j j j j t t void shellsort(int a[], int size) { int h = size/2; while (h > 0) { // h-sort for (int i=h; i<size;i++) { int t=a[i], j=i; while (j >= h && a[j-h] > t) { a[j] = a[j-h]; j -= h; } a[j]=t; h /= 2; } // end increment bad strategy h = 4 h = 4 h = 4 i h = 4 j j j j j-h j-h j,i j, i t t 7 5 8 14 6 12 9 2 16 1 4/20/2019 IT 279

if gcd(p,q) = 1, then any number is a liner combination of p and q hk+1-sorted hk+2-sorted hk+1 hk+1 hk+1 hk+1 hk+1 hk+2 hk+2 hk+2 A[p] A[p-(xhk+2+yhk+1)]  A[p-(xhk+2+yhk+1)] < A[p] If j is a liner combination of hk+2 and hk+1 with positive coefficients, then there is no need to check A[p-j] < A[p] xp+yq= gcd(p,q) if gcd(p,q) = 1, then any number is a liner combination of p and q 4/20/2019 IT 279

A[p-(xhk+2+yhk+1)] < A[p] hk+1-sorted hk+2-sorted hk+1 hk+1 hk+1 hk+1 hk+1 hk+2 hk+2 hk+2 A[p] A[p-j] If j is a liner combination of hk+2 and hk+1 with positive coefficients, then there is no need to check A[p-j] < A[p] while (h > 0) { // h-sort for (int i=h; i<size;i++) { int t=a[i], j=i; while (j >= h && a[j-h] > t) { a[j] = a[j-h]; j -= h; } a[j]=t; while (h > 0) { // h-sort for (int i=h; i<size;i++) { int t=a[i], j=i; while (x > j && j >= h && a[j-h] > t) { a[j] = a[j-h]; j -= h; } a[j]=t; 4/20/2019 IT 279

then for sufficiently large j, we have that A[p-(xhk+2+yhk+1)] < A[p] hk+1-sorted hk+2-sorted hk+1 hk+1 hk+1 hk+1 hk+1 hk+2 hk+2 hk+2 A[p] A[p-j] xp+yq= gcd(p,q) If gcd(hk+1 , hk+2 ) = 1, then for sufficiently large j, we have that j is a liner combination of hk+2 and hk+1 with positive coefficients, there is no need to check A[p-j] < A[p] 4/20/2019 IT 279

gcd(hk+1 , hk+2 ) = 1,   A[p-j] < A[p] 4/20/2019 IT 279

 while (h > 0) { // h-sort for (int i=h; i<size;i++) { int t=a[i], j=i; while (x > j && j >= h && a[j-h] > t) { a[j] = a[j-h]; j -= h; } a[j]=t;  4/20/2019 IT 279

Percolating a non-heap Worst case: O(n) Heapsort Percolating a non-heap Worst case: O(n) 34 2 15 2 5 7 30 7 15 2 6 23 5 9 21 25 30 7 6 23 9 21 6 15 32 5 23 40 11 31 30 21 7 25 15 32 34 40 11 31 30 25 4/20/2019 IT 279

Percolating a non-heap Heapsort O(log n) 2 Percolating a non-heap O(log n-1) 5 Worst case: O(n) O(log n-2) 6 7 2 5 7 6 23 9 21 40 O(log 1) 15 32 34 40 11 31 30 25 O(n+n log n) = O(n log n) 4/20/2019 IT 279

Mergesort Merge two sorted lists 3 5 7 8 1 2 6 O(n) 4/20/2019 IT 279

7 5 3 8 2 1 6 Merge Sort 7 5 3 8 2 1 6 7 5 3 8 2 1 6 O(log n) 7 5 3 8 2 1 5 7 3 8 1 2 O(n log n) 3 5 7 8 1 2 6 1 2 3 5 6 7 8 4/20/2019 IT 279

Quick Sort < < < < < < < < < < < < 7 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4 < 7 5 8 4 6 1 3 2 11 10 9 13 16 15 12 14 < < < 2 3 1 4 6 8 5 7 11 10 9 12 16 15 13 14 < < < < < < < 2 1 3 4 6 5 8 7 9 10 11 12 14 13 15 16 < < < < < < < < < < < < < < < 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 4/20/2019 IT 279

Quick Sort 7 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4 4 3 2 16 14 8 2 5 4 3 6 1 7 16 11 10 9 13 14 15 12 8 1 2 4 3 6 5 7 8 11 10 9 13 14 15 12 16 1 2 3 4 6 5 7 8 11 10 9 13 14 15 12 16 1 2 3 4 5 6 7 8 9 10 11 13 14 15 12 16 1 2 3 4 5 6 7 8 9 10 11 12 13 15 14 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 4/20/2019 IT 279

Quick Sort /* Quick Sort */ void SwapTwo(int A[], int i, int j){ int temp = A[i]; A[i]=A[j]; A[j]=temp; } void QuickSort(int A[], int h, int t){ // h: head // t: tail if (h == t) return; int i=h+1, j=t; if (i==j) { if (A[h] > A[i]) SwapTwo(A,h,i); return; while (i < j) { while (A[h] >= A[i] && i < t) i++; while (A[h] <= A[j] && j > h) j--; if (i < j) SwapTwo(A,i++,j--); if (i==j && A[h]>A[i]) SwapTwo(A,h,i); QuickSort(A,h,i-1); QuickSort(A,i,t); Quick Sort 4/20/2019 IT 279