1. Sub-Quadratic Sorting Algorithms
Sub-Quadratic = Big O better than O(n2)
CLRS, Sections 2.1, 2.3, 6.4, 7.1 – 7.4

2. Sub-Quadratic Sorts Divide and Conquer Algorithms Data Structure Based
Quicksort
Merge Sort
Data Structure Based
Heapsort
Linear Sorts
Counting Sort
Radix Sort
CS Data Structures

3. Quicksort Quicksort is the most popular, fast sorting algorithm:
It has an average running time case of
Θ(n*log n)
The other fast sorting algorithms, Mergesort and Heapsort, also run in Θ(n*log n) time, but
have fairly large constants hidden in their algorithms.
tend to move data around more than desirable.
CS Data Structures

4. Quicksort Invented by C.A.R. (Tony) Hoare
A Divide-and-Conquer approach that uses recursion:
If the list has 0 or 1 elements, it’s sorted
Otherwise, pick any element p in the list. This is called the pivot value
Partition the list, minus the pivot, into two sub-lists:
one of values less than the pivot and another of those greater than the pivot
equal values go to either
Return the Quicksort of the first list followed by the Quicksort of the second list.
CS Data Structures

5. Quicksort Tree Use binary tree to represent the execution of Quicksort
Each node represents a recursive call of Quicksort
Stores
Unsorted sequence before the execution and its pivot
Sorted sequence at the end of the execution
The leaves are calls on sub-sequences of size 0 or 1 
2 4  2 4
9 7  7 9
2  2 - -
9  9
CS Data Structures

6. Example: Quicksort Execution
Pivot selection – last value in list: 6
CS Data Structures

7. Example: Quicksort Execution
Partition around 6
Recursive call on sub-list with smaller values
CS Data Structures

8. Example: Quicksort Execution
Partition around 2
Recursive call on sub-list of smaller values
1 -
4 3 -
CS Data Structures

9. Example: Quicksort Execution
Base case
Return from sub-list of smaller values
 1 2
1  1
CS Data Structures

10. Example: Quicksort Execution
Recursive call on sub-list of larger values
 1 2
1  1
CS Data Structures

11. Example: Quicksort Execution
Partition around 4
Recursive call on sub-list of smaller values
 1 2
1  1
4 - -
CS Data Structures

12. Example: Quicksort Execution
Base case
Return from sub-list of smaller values
 1 2
1  1
4 3  3 4
4  4 -
CS Data Structures

13. Example: Quicksort Execution
Recursive call on sub-list of larger values
 1 2
1  1
4 3  3 4
4  4 -
CS Data Structures

14. Example: Quicksort Execution
Base case
Return from sub-list of larger values
 1 2
1  1
4 3  3 4
4  4 -
CS Data Structures

15. Example: Quicksort Execution
Return from sub-list of larger values 
1  1
4 3  3 4
4  4 -
CS Data Structures

16. Example: Quicksort Execution
Return from sub-list of smaller values  
1  1
4 3  3 4
4  4 -
CS Data Structures

17. Example: Quicksort Execution
Recursive call on sub-list of larger values  
1  1
4 3  3 4
4  4 -
CS Data Structures

18. Example: Quicksort Execution
Partition around 7
Recursive call on sub-list of smaller values  
1  1
4 3  3 4
7 -
9 -
4  4 3
CS Data Structures

19. Example: Quicksort Execution
Base case
Return from sub-list of smaller values  
7 9 7  7 7
1  1
4 3  3 4
7  7
9 -
4  4
CS Data Structures

20. Example: Quicksort Execution
Recursive call on sub-list of larger values  
7 9 7  7 7
1  1
4 3  3 4
7  7
9 -
4  4
CS Data Structures

21. Example: Quicksort Execution
Base case
Return from sub-list of larger values  
7 9 7  7 7 9
1  1
4 3  3 4
7  7
9  9
4  4
CS Data Structures

22. Example: Quicksort Execution
Return from sub-list of larger values  
7 9 7  7 7 9
1  1
4 3  3 4
7  7
9  9
4  4
CS Data Structures

23. Example: Quicksort Execution
Done  
7 9 7  7 7 9
1  1
4 3  3 4
7  7
9  9
4  4
CS Data Structures

24. Quicksort Algorithm // A: array of values
// p: beginning index of sub-list being sorted
// r: ending index of sub-list being sorted
// Initial call: Quicksort(A, 1, A.length)
CS Data Structures

25. Quicksort Partition Algorithm
CS Data Structures

26. Example: Quicksort Partition
Partition(A, 1, A.length)
x = A[7] = 6
i = p - 1 = 0
j = 1
A[1] = 2 ≤ 6
i = = 1
swap A[1], A[1] 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
CS Data Structures

27. Example: Quicksort Partition
Partition(A, 1, A.length)
j = 2
A[2] = 8 > 6
no change 1 2 3 4 5 6 7 8
CS Data Structures

28. Example: Quicksort Partition
Partition(A, 1, A.length)
j = 3
A[3] = 7 > 6
no change 1 2 3 4 5 6 7 8
CS Data Structures

29. Example: Quicksort Partition
Partition(A, 1, A.length)
j = 4
A[4] = 1 ≤ 6
i = = 2
swap A[2], A[4] 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
CS Data Structures

30. Example: Quicksort Partition
Partition(A, 1, A.length)
j = 5
A[5] = 3 ≤ 6
i = = 3
swap A[3], A[5] 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
CS Data Structures

31. Example: Quicksort Partition
Partition(A, 1, A.length)
j = 6
A[6] = 5 ≤ 6
i = = 4
swap A[4], A[6] 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
CS Data Structures

32. Example: Quicksort Partition
Partition(A, 1, A.length)
j = 7
for loop done
swap A[5], A[7]
return 5 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
CS Data Structures

33. Example: Quicksort Partition
Partition(A, 1, A.length)
Partition complete 1 2 3 4 5 6 7 8
CS Data Structures

34. Runtime Analysis: Best Case
What is best case running time?
Assume keys are random, uniformly distributed.
Recursion:
Partition splits list in two sub-lists of size: (n/2)
Quicksort each sub-list.
Depth of recursion tree? O(log n)
Number of accesses in partition? O(n)
Best case running time: O(n*log n)
CS Data Structures

35. Runtime Analysis: Worst Case
What is worst case running time?
List already sorted.
Recursion:
Partition splits array in two sub-arrays:
one sub-array of size: 0
the other sub-array of size: n-1
Quicksort each sub-array.
Depth of recursion tree? O(n)
Number of accesses per partition? O(n)
Worst case running time: O(n2)
CS Data Structures

36. Average Case for Quicksort
If the list is already sorted, Quicksort is terrible: O(n2)
It is possible to construct other bad cases.
However, Quicksort is usually O(n*log n)
Because the constants are so good, Quicksort is generally the fastest known, comparison-based algorithm.
Most real-world sorting is done by Quicksort.
CS Data Structures

37. Tweaking Quicksort
Almost anything you can try to “improve” Quicksort will actually slow it down.
One good tweak is to switch to a different sorting method when the subarrays get small (say, 10 or 12).
Quicksort has too much overhead for small array sizes.
For large arrays, it might be a good idea to check beforehand if the array is already sorted.
But there is a better tweak than this.
CS Data Structures

38. Picking a Better Pivot
Before, we picked the first element of the sub-list to use as a pivot.
If the array is already sorted, this results in O(n2) behavior.
It’s no better if we pick the last element.
We could do an optimal Quicksort if we always picked a pivot value that exactly cuts array in half.
Such a value is called a median: half of the values in the list are larger, half are smaller.
The easiest way to find the median is to sort the list and pick the value in the middle. (!)
CS Data Structures

39. Median of Three
Obviously, it doesn’t make sense to sort list in order to find median.
Instead, compare just three elements of sub-list: first, last, and middle.
Take the median (middle value) of these three as pivot.
If rearrange (sort) these three numbers so that the smallest is in the first position, the largest in the last position, and the other in the middle.
Simplifies and speeds up the partition loop.
CS Data Structures

40. Mergesort Algorithm
Don Knuth cites John von Neumann as the creator of this algorithm:
If a list has 1 element or 0 elements, it’s sorted.
If a list has more than 1, split into two separate lists.
Perform this algorithm on each of those smaller lists.
Take the 2 sorted lists and merge them together.
CS Data Structures

41. Mergesort When implementing, one temporary array is used instead of
multiple temporary
arrays.
Why?
CS Data Structures

42. Merge-Sort Tree
Use binary tree to represent the execution of Merge-sort
Each node represents a recursive call of Merge-sort.
Stores
Unsorted sequence before the execution and its pivot.
Sorted sequence at the end of the execution.
The leaves are calls on sub-sequences of size 0 or 1. 
7 2  2 7
9 4  4 9
7  7
2  2
9  9
4  4
CS Data Structures

43. Example: Merge-sort Execution
List to be sorted
CS Data Structures

44. Example: Merge-sort Execution
Divide values in half between 3 and 7. /
CS Data Structures

45. Example: Merge-sort Execution
Recursive call on first sub-list.
CS Data Structures

46. Example: Merge-sort Execution
Divide values in half between 9 and 4.
5 9/4 3 -
CS Data Structures

47. Example: Merge-sort Execution
Recursive call on first sub-list.
5 9 -
4 3 -
CS Data Structures

48. Example: Merge-sort Execution
Divide values in half between 5 and 9.
5/9 -
4 3 -
CS Data Structures

49. Example: Merge-sort Execution
Recursive call on first sub-list.
5 9 -
4 3 -
5 -
9 -
CS Data Structures

50. Example: Merge-sort Execution
Base case.
First sub-list returns 5.
5 9 -
4 3 -
5  5
9 -
CS Data Structures

51. Example: Merge-sort Execution
Recursive call on second sub-list.
5 9 -
4 3 -
5  5
9 -
CS Data Structures

52. Example: Merge-sort Execution
Base case.
Second sub-list returns 9.
5 9 -
4 3 -
5  5
9  9
CS Data Structures

53. Example: Merge-sort Execution
Merge 5 and 9.
5 9  5 9
4 3 -
5  5
9  9
CS Data Structures

54. Example: Merge-sort Execution
Return 59 from first sub-list.
5 9  5 9
4 3 -
5  5
9  9
CS Data Structures

55. Example: Merge-sort Execution
Recursive call on second sub-list.
5 9  5 9
5  5
9  9
CS Data Structures

56. Example: Merge-sort Execution
Divide values in half between 4 and 3.
5 9  5 9
4/3 -
5  5
9  9
CS Data Structures

57. Example: Merge-sort Execution
Recursive call on first sub-list.
5 9  5 9
4 3 -
5  5
9  9
4 -
3 -
CS Data Structures

58. Example: Merge-sort Execution
Base case.
First sub-list returns 4.
5 9  5 9
4 3 -
5  5
9  9
4  4
3 -
CS Data Structures

59. Example: Merge-sort Execution
Recursive call on second sub-list.
5 9  5 9
4 3 -
5  5
9  9
4  4
3 -
CS Data Structures

60. Example: Merge-sort Execution
Base case.
First sub-list returns 3.
5 9  5 9
4 3 -
5  5
9  9
4  4
3  3
CS Data Structures

61. Example: Merge-sort Execution
Merge 4 and 3.
5 9  5 9
4 3  3 4
5  5
9  9
4  4
3  3
CS Data Structures

62. Example: Merge-sort Execution
Return 34 from second sub-list.
5 9  5 9
4 3  3 4
5  5
9  9
4  4
3  3
CS Data Structures

63. Example: Merge-sort Execution
Merge 59 and 34. 
5 9  5 9
4 3  3 4
5  5
9  9
4  4
3  3
CS Data Structures

64. Example: Merge-sort Execution
First sub-list returns 3459. 
5 9  5 9
4 3  3 4
5  5
9  9
4  4
3  3
CS Data Structures

65. Example: Merge-sort Execution
Recursive call on second sub-list. 
5 9  5 9
4 3  3 4
5  5
9  9
4  4
3  3
CS Data Structures

66. Example: Merge-sort Execution
Divide values in half between 1 and 2. 
7 1/2 6 -
5 9  5 9
4 3  3 4
5  5
9  9
4  4
3  3
CS Data Structures

67. Example: Merge-sort Execution
Recursive call on first sub-list. 
5 9  5 9
4 3  3 4
7 1 -
2 6 -
5  5
9  9
4  4
3  3
CS Data Structures

68. Example: Merge-sort Execution
Divide values in half between 7 and 1. 
5 9  5 9
4 3  3 4
7/1 -
2 6 -
5  5
9  9
4  4
3  3
CS Data Structures

69. Example: Merge-sort Execution
Recursive call on first sub-list. 
5 9  5 9
4 3  3 4
7 1 -
2 6 -
5  5
9  9
4  4
3  3
7 -
1 -
CS Data Structures

70. Example: Merge-sort Execution
Base case.
First sub-list returns 7. 
5 9  5 9
4 3  3 4
7 1 -
2 6 -
5  5
9  9
4  4
3  3
7  7
1 -
CS Data Structures

71. Example: Merge-sort Execution
Recursive call on second sub-list. 
5 9  5 9
4 3  3 4
7 1 -
2 6 -
5  5
9  9
4  4
3  3
7  7
1 -
CS Data Structures

72. Example: Merge-sort Execution
Base case.
First sub-list returns 1. 
5 9  5 9
4 3  3 4
7 1 -
2 6 -
5  5
9  9
4  4
3  3
7  7
1  1
CS Data Structures

73. Example: Merge-sort Execution
Merge 7 and 1. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6 -
5  5
9  9
4  4
3  3
7  7
1  1
CS Data Structures

74. Example: Merge-sort Execution
First sub-list returns 17. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6 -
5  5
9  9
4  4
3  3
7  7
1  1
CS Data Structures

75. Example: Merge-sort Execution
Recursive call on second sub-list. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6 -
5  5
9  9
4  4
3  3
7  7
1  1
CS Data Structures

76. Example: Merge-sort Execution
Divide values in half between 2 and 6. 
5 9  5 9
4 3  3 4
7 1  1 7
2/6 -
5  5
9  9
4  4
3  3
7  7
1  1
CS Data Structures

77. Example: Merge-sort Execution
Recursive call on first sub-list. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6 -
5  5
9  9
4  4
3  3
7  7
1  1
2 -
6 -
CS Data Structures

78. Example: Merge-sort Execution
Base case.
First sub-list returns 2. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6 -
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6 -
CS Data Structures

79. Example: Merge-sort Execution
Recursive call on second sub-list. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6 -
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6 -
CS Data Structures

80. Example: Merge-sort Execution
Base case.
First sub-list returns 6. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6 -
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6  6
CS Data Structures

81. Example: Merge-sort Execution
Merge 2 and 6. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6  2 6
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6  6
CS Data Structures

82. Example: Merge-sort Execution
Second sub-list returns 26. 
5 9  5 9
4 3  3 4
7 1  1 7
2 6  2 6
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6  6
CS Data Structures

83. Example: Merge-sort Execution
Merge 17 and 26.  
5 9  5 9
4 3  3 4
7 1  1 7
2 6  2 6
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6  6
CS Data Structures

84. Example: Merge-sort Execution
Second sub-list returns 1267.  
5 9  5 9
4 3  3 4
7 1  1 7
2 6  2 6
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6  6
CS Data Structures

85. Example: Merge-sort Execution
Merge 3479 and 1267.   
5 9  5 9
4 3  3 4
7 1  1 7
2 6  2 6
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6  6
CS Data Structures

86. Example: Merge-sort Execution
Done.   
5 9  5 9
4 3  3 4
7 1  1 7
2 6  2 6
5  5
9  9
4  4
3  3
7  7
1  1
2  2
6  6
CS Data Structures

87. Merge Sort Algorithm
CS Data Structures

88. Merge Sort Runtime Analysis
What is running time?
No best, worst case. Runtime independent of data sorting.
Recursion:
Partition splits list in two sub-lists of size: (n/2)
Merge sort each sub-list.
Depth of recursion tree? O(log n)
Cost to merge? O(n)
Running time: O(n*log n)
CS Data Structures

89. Heapsort Algorithm Use Max-Heap to sort list of elements:
Add n elements to Max-Heap.
Swap the first element (maximum) with last element.
Largest element is in last position – where it belongs in sorted order.
However, first n – 1 elements no longer a Max-Heap.
Float element at root down one of its subtrees to return tree to a Max-Heap.
Repeat until all elements sorted.
CS Data Structures

90. Heapsort Algorithm
CS Data Structures

91. Example: Heapsort Begin after Build-Max-Heap 9 5 7 6 2 4 i = 61 3
Begin after Build-Max-Heap
i = 6
swap A[1], A[6]
A.heap-size = 5
Max-Heapify(A, 1) 7 5 6 4 2 1 3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 - 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 -
CS Data Structures

92. Example: Heapsort Begin after Build-Max-Heap 7 5 6 4 2 i = 51 3
Begin after Build-Max-Heap
i = 5
swap A[1], A[5]
A.heap-size = 4
Max-Heapify(A, 1) 6 5 4 2 1 3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 - 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 -
CS Data Structures

93. Example: Heapsort Begin after Build-Max-Heap 6 5 4 2 i = 41 3
Begin after Build-Max-Heap
i = 4
swap A[1], A[4]
A.heap-size = 3
Max-Heapify(A, 1) 5 2 4 1 3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 - 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 -
CS Data Structures

94. Example: Heapsort Begin after Build-Max-Heap 5 2 4 i = 31 3
Begin after Build-Max-Heap
i = 3
swap A[1], A[3]
A.heap-size = 2
Max-Heapify(A, 1) 4 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 - 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 -
CS Data Structures

95. Example: Heapsort Begin after Build-Max-Heap 4 2 i = 2 swap A[1], A[2]
A.heap-size = 1
Max-Heapify(A, 1) 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 - 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 -
CS Data Structures

96. Heapsort Running Time The heapsort algorithm takes time O(n*log2n).
Build-Max-Heap takes O(n) time.
We do n calls to Max-Heapify which takes O(log2n).
CS Data Structures

97. Analysis of Sorting Algorithms, So Far
Best Case
Average Case
Worst Case
Selection Sort
O(n2)
Insertion Sort
O(n)
Bubble Sort
Quicksort
O(n*log n)
Merge Sort
Heapsort
CS Data Structures

98. Comparison of Various Sorts
Num Items
Selection
Insertion
Quicksort
1000 16 5
2000 59 49 6
4000
271
175
8000
1056
686
16000
4203
2754 11
32000
16852
11039 45
64000
expected? 68
128000
158
256000
335
512000
722
1550
times in milliseconds
CS Data Structures

99. Final Comments Language libraries have sorting algorithms Hybrid sorts
Java Arrays and Collections classes
C++ Standard Template Library
Hybrid sorts
When size of unsorted list or portion of array is small:
use Insertion or Selection Sort.
Otherwise:
use O(n log(n)) sort like Quicksort or Heapsort.
Many other special case sorting algorithms exist, like Radix Sort.
CS Data Structures

100. CS Data Structures

101. ShellSort Created by Donald Shell in 1959
Wanted to stop moving data small distances (in the case of insertion sort and bubble sort) and stop making swaps that are not helpful (in the case of selection sort)
Start with sub arrays created by looking at data that is far apart and then reduce the gap size
CS Data Structures

102. ShellSort in practice
Gap of five. Sort sub array with 46, 5, and
Gap still five. Sort sub array with 2 and
Gap still five. Sort sub array with 83 and
Gap still five Sort sub array with 41 and
Gap still five. Sort sub array with 102 and
Continued on next slide:
CS Data Structures

103. Completed Shellsort
Gap now 2: Sort sub array with
Gap still 2: Sort sub array with
Gap of 1 (Insertion sort)
Array sorted
CS Data Structures

104. Shellsort on Another Data Set
Initial gap = length / 2 = 16 / 2 = 8
initial sub arrays indices:
{0, 8}, {1, 9}, {2, 10}, {3, 11}, {4, 12}, {5, 13}, {6, 14}, {7, 15} next gap = 8 / 2 = 4
{0, 4, 8, 12}, {1, 5, 9, 13}, {2, 6, 10, 14}, {3, 7, 11, 15} next gap = 4 / 2 = 2
{0, 2, 4, 6, 8, 10, 12, 14}, {1, 3, 5, 7, 9, 11, 13, 15}
final gap = 2 / 2 = 1
CS Data Structures

105. ShellSort Code
public static void shellsort(Comparable[] list) { Comparable temp; boolean swap;
for(int gap = list.length / 2; gap > 0; gap /= 2) for(int i = gap; i < list.length; i++) { Comparable tmp = list[i];
int j = i;
for( ; j >= gap &&
tmp.compareTo( list[j - gap] ) < 0;
j -= gap )
list[ j ] = list[ j - gap ];
list[ j ] = tmp;
} }
CS Data Structures