1. The father of algorithm analysis
Donald E. Knuth ( )
The father of algorithm analysis
popularizing asymptotic notation.
The author of
TAOCP -- The Art of Computer Programming
Volume 1: Fundamental Algorithms
Volume 2: Seminumerical Algorithms
Volume 3: Sorting and Searching
(among tons of others)
“ I have been a happy man ever since January 1, 1990,
when I no longer had an address.”
4/20/2019
IT 279

2. The Art of Computer Programming
TAOCP –
The Art of Computer Programming
Volume 1: Fundamental Algorithms
Basic Concepts and Information Structures
Volume 2: Seminumerical Algorithms
Random numbers and Arithmetic
Volume 3: Sorting and Searching
Sorting and Searching
Volume 4: Combinatorial Algorithms
Combinatorial Searching and Recursion
Volume 5: Syntactical Algorithms
Lexical Scanning and Parsing Techniques
Volume 6: Theory of Languages
Mathematical Linguistics
Volume 7: Compiler
Programming Language Translation
Donald E. Knuth
Highlighted in the preface of Vol. 1
4/20/2019
IT 279

3. “An algorithm must be seen to be believed”
-- Donald E. Knuth --
Sorting algorithms
4/20/2019
IT 279

4. Sorting: Arranging items into a certain order.
Items can be stored in an one dimensional array or linked list, or trees.
We prefer using one dimensional array
Why?
4/20/2019
IT 279

5. Insert an item into a sorted the right place.
Insertion Sort:
Insert an item into a sorted the right place.
sorted 1 3 4 6 8 9 10 12 7
4/20/2019
IT 279

6. Insert an item into the right place.
Insertion Sort:
Insert an item into the right place.
sorted 1 3 4 6 8 9 10 12 7 2
4/20/2019
IT 279

7. Insert an item into the right place.
Insertion Sort:
Insert an item into the right place.
sorted 1 3 4 6 8 9 10 12 7 2 7 8 9 10 12 7
4/20/2019
IT 279

8. Insert an item into the right place. 6 12 3 1 10 8 9 7 2 6 12 3 1 10 8
Insertion Sort:
Insert an item into the right place. 6 12 3 1 10 8 9 7 2 6 12 3 1 10 8 9 7 2 3 6 12 1 10 8 9 7 2 1 3 6 12 10 8 9 7 2 1 3 6 10 12 8 9 7 2 1 3 6 8 10 12 9 7 2 1 3 6 8 9 10 12 7 2 1 3 6 7 8 9 10 12 2 1 2 3 6 7 8 9 10 12
4/20/2019
IT 279

9. Select the minimum item into the end of the sorted segment.
Selection Sort:
Select the minimum item into the end of the sorted segment.
sorted 1 3 4 9 12 7 8 10
the minimum one 1 3 4 7 12 9 8 10
the minimum one
4/20/2019
IT 279

10. Selection Sort:
Select the minimum item into the end of the sorted segment. 6 12 3 1 10 8 9 7 2 1 12 3 6 10 8 9 7 2 1 2 3 6 10 8 9 7 12 1 2 3 6 10 8 9 7 12 1 2 3 6 10 8 9 7 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12
4/20/2019 1 2 3 6 7 8 9 10 12
IT 279

11. Analysis of Insertion/Selection Sorts:
T(n) = n = O(n2)
Shell Sort
D. L. Shell, (1959)
O(n3/2)
O(n log n)
Lower bound of sorting algorithms
4/20/2019
IT 279

12. Incremental insertion (selection) sort. Shell Sort: h = 47 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4 3 5 8 14 6 1 16 2 7 10 9 13 11 15 12 4 3 1 8 14 6 5 16 2 7 10 9 13 11 15 12 4 3 1 8 14 6 5 9 2 7 10 12 13 11 15 16 4 3 1 8 2 6 5 9 4 7 10 12 13 11 15 16 14
4/20/2019
IT 279

13. Shell Sort: hk-sorted h = 4 4-sorted 2-sorted 1-sorted 3 1 8 2 6 5 9 47 10 12 13 11 15 16 14
4-sorted 3 1 6 2 7 4 8 5 9 10 11 13 12 14 16 15
2-sorted 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1-sorted
4/20/2019
IT 279

14. Incremental insertion
Shell Sort:
Incremental insertion
with
Increment sequence,
hk-sorted
1-sorted
ht-sorted
ht-1-sorted
hk-1-sorted  (implies) hk-sorted
hk-1-sorted
hk-sorted
Shell suggested:
Not a good suggestion, worst case: O(n2)
4/20/2019
IT 279

15. Shell Sort: using Shell’s suggestion
Worst case analysis: O(n2)
Idea:
Right before the final sort, let the smallest n/2 be distributed in the even position.
e.g.
2-sorted 9 1 10 2 11 3 12 4 13 5 14 6 15 7 16 8 1 2 3 4 5 6 7 8
final sort
4/20/2019
IT 279

16. Shell Sort: using Shell’s suggestion
What kind of initial array will result in such 2-sorted? 9 1 10 2 11 3 12 4 13 5 14 6 15 7 16 8 13 5 12 1 14 4 10 7 11 3 9 8 15 6 16 2
Let Ah[s] = A[s], A[s+h], A[s+2h], ....
odd positions
Ah[1]
Ah[0]
even positions 5 13 1 11 3
Ah[2] 12 7 4 15 9 10 8
Ah[3] 6 2 14 16
Ah[5]
Ah[6]
Ah[4]
Ah[7]
elements will never cross the even-/odd-cell before the final sort
8-sort ,
4-sort ,
2-sort
4/20/2019
IT 279

17. bad strategy Shell sort algorithm 7 5 8 14 6 12 9 2 16 1 i j j j j t t
void shellsort(int a[], int size) {
int h = size/2;
while (h > 0) { // h-sort
for (int i=h; i<size;i++) {
int t=a[i], j=i;
while (j >= h && a[j-h] > t) {
a[j] = a[j-h];
j -= h; }
a[j]=t;
h /= 2;
} // end increment
bad strategy
h = 4
h = 4
h = 4 i
h = 4 j j j j
j-h
j-h
j,i j, i t t 7 5 8 14 6 12 9 2 16 1
4/20/2019
IT 279

18. if gcd(p,q) = 1, then any number is a liner combination of p and q
hk+1-sorted
hk+2-sorted
hk+1
hk+1
hk+1
hk+1
hk+1
hk+2
hk+2
hk+2
A[p]
A[p-(xhk+2+yhk+1)]
 A[p-(xhk+2+yhk+1)] < A[p]
If j is a liner combination of hk+2 and hk+1 with positive coefficients, then there is no need to check A[p-j] < A[p]
xp+yq= gcd(p,q)
if gcd(p,q) = 1, then any number is a liner combination
of p and q
4/20/2019
IT 279

19. A[p-(xhk+2+yhk+1)] < A[p]
hk+1-sorted
hk+2-sorted
hk+1
hk+1
hk+1
hk+1
hk+1
hk+2
hk+2
hk+2
A[p]
A[p-j]
If j is a liner combination of hk+2 and hk+1 with positive coefficients, then there is no need to check A[p-j] < A[p]
while (h > 0) { // h-sort
for (int i=h; i<size;i++) {
int t=a[i], j=i;
while (j >= h && a[j-h] > t) {
a[j] = a[j-h];
j -= h; }
a[j]=t;
while (h > 0) { // h-sort
for (int i=h; i<size;i++) {
int t=a[i], j=i;
while (x > j && j >= h && a[j-h] > t) {
a[j] = a[j-h];
j -= h; }
a[j]=t;
4/20/2019
IT 279

20. then for sufficiently large j, we have that
A[p-(xhk+2+yhk+1)] < A[p]
hk+1-sorted
hk+2-sorted
hk+1
hk+1
hk+1
hk+1
hk+1
hk+2
hk+2
hk+2
A[p]
A[p-j]
xp+yq= gcd(p,q)
If gcd(hk+1 , hk+2 ) = 1,
then for sufficiently large j, we have that
j is a liner combination of hk+2 and hk+1 with positive coefficients, there is no need to check A[p-j] < A[p]
4/20/2019
IT 279

21. gcd(hk+1 , hk+2 ) = 1, 
 A[p-j] < A[p]
4/20/2019
IT 279

22.  while (h > 0) { // h-sort for (int i=h; i<size;i++) {
int t=a[i], j=i;
while (x > j && j >= h && a[j-h] > t) {
a[j] = a[j-h];
j -= h; }
a[j]=t; 
4/20/2019
IT 279

23. Percolating a non-heap Worst case: O(n)
Heapsort
Percolating a non-heap
Worst case: O(n) 34 2 15 2 5 7 30 7 15 2 6 23 5 9 21 25 30 7 6 23 9 21 6 15 32 5 23 40 11 31 30 21 7 25 15 32 34 40 11 31 30 25
4/20/2019
IT 279

24. Percolating a non-heap
Heapsort
O(log n) 2
Percolating a non-heap
O(log n-1) 5
Worst case: O(n)
O(log n-2) 6 7 2 5 7 6 23 9 21 40
O(log 1) 15 32 34 40 11 31 30 25
O(n+n log n) = O(n log n)
4/20/2019
IT 279

25. Mergesort
Merge two sorted lists 3 5 7 8 1 2 6
O(n)
4/20/2019
IT 279

26. 7 5 3 8 2 1 6
Merge Sort 7 5 3 8 2 1 6 7 5 3 8 2 1 6
O(log n) 7 5 3 8 2 1 5 7 3 8 1 2
O(n log n) 3 5 7 8 1 2 6 1 2 3 5 6 7 8
4/20/2019
IT 279

27. Quick Sort < < < < < < < < < < < <7 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4
< 7 5 8 4 6 1 3 2 11 10 9 13 16 15 12 14
<
<
< 2 3 1 4 6 8 5 7 11 10 9 12 16 15 13 14
<
<
<
<
<
<
< 2 1 3 4 6 5 8 7 9 10 11 12 14 13 15 16
<
<
<
<
<
<
<
<
<
<
<
<
<
<
< 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
4/20/2019
IT 279

28. Quick Sort 7 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4 4 3 2 16 14 8 2 5 4 3 6 1 7 16 11 10 9 13 14 15 12 8 1 2 4 3 6 5 7 8 11 10 9 13 14 15 12 16 1 2 3 4 6 5 7 8 11 10 9 13 14 15 12 16 1 2 3 4 5 6 7 8 9 10 11 13 14 15 12 16 1 2 3 4 5 6 7 8 9 10 11 12 13 15 14 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
4/20/2019
IT 279

29. Quick Sort /* Quick Sort */ void SwapTwo(int A[], int i, int j){
int temp = A[i];
A[i]=A[j];
A[j]=temp; }
void QuickSort(int A[], int h, int t){ // h: head
// t: tail
if (h == t) return;
int i=h+1, j=t;
if (i==j) {
if (A[h] > A[i]) SwapTwo(A,h,i);
return;
while (i < j) {
while (A[h] >= A[i] && i < t) i++;
while (A[h] <= A[j] && j > h) j--;
if (i < j) SwapTwo(A,i++,j--);
if (i==j && A[h]>A[i]) SwapTwo(A,h,i);
QuickSort(A,h,i-1);
QuickSort(A,i,t);
Quick Sort
4/20/2019
IT 279