1. Computer Algorithms Lecture 11 Sorting in Linear Time Ch. 8
Some of these slides are courtesy of D. Plaisted et al, UNC and M. Nicolescu, UNR

2. Comparison-based Sorting
Comparison sort
Only comparison of pairs of elements may be used to gain order information about a sequence.
Hence, a lower bound on the number of comparisons will be a lower bound on the complexity of any comparison-based sorting algorithm.
What sorts we have analyzed so far have been comparison sorts
The best worst-case complexity so far is (n lg n)
To sort n elements, comparison sorts must make (nlgn) comparisons in the worst case
We prove a lower bound of n lg n, (or (n lg n)) for any comparison sort, implying that merge sort and heapsort are optimal.

3. Decision Tree Model
Full binary tree: every node is either a leaf of has degree 2
Represents the comparisons made by a sorting algorithm on an input of a given size: models all possible execution traces
Control, data movement, other operations are ignored
Count only the comparisons
Decision tree for insertion sort on three elements:
one execution trace
node
leaf:

4. Decision Tree Model
All permutations on n elements must appear as one of the leaves in the decision tree
Worst-case number of comparisons
the length of the longest path from the root to a leaf
the height of the decision tree
n! permutations

5. Decision Tree Model
Goal: finding a lower bound on the running time on any comparison sort algorithm
find a lower bound on the heights of all decision trees for all algorithms

6. Lemma Any binary tree of height h has at most Proof: induction on h
Basis: h = 0  tree has one node, which is a leaf
2h = 1
Inductive step: assume true for h-1 (2h-1 leaves)
Extend the height of the tree with one more level
Each leaf becomes parent to two (complete binary tree) new leaves
No. of leaves for tree of height h =
= 2  (no. of leaves for tree of height h-1)
≤ 2  2h-1
= 2h
2h leaves

7. Lower Bound for Comparison Sorts
Theorem: Any comparison sort algorithm requires (nlgn) comparisons in the worst case.
Proof: How many leaves does the tree have?
At least n! (each of the n! permutations of the input appears as some leaf)  n! ≤ l
At most 2h leaves
 n! ≤ l ≤ 2h
 h ≥ lg(n!) = (nlgn)
(3.18) h
l leaves
What comparison sorts are asymptotically optimal?
Merge sort and heapsort
We can beat the (nlgn) running time if we use other operations than comparisons!

8. Beating the lower bound
We can beat the lower bound if we don’t base our sort on comparisons:
Counting sort for keys in [0..k], k=O(n)
Radix sort for keys with a fixed number of “digits”
Bucket sort for random keys (uniformly distributed)

9. Counting Sort 3 2 5 7 4 2 8 5 3 2 Assumption: Idea: A C B
The elements to be sorted are integers in the range 0 to k
Idea:
Determine for each input element x, the number of elements smaller than or equal to x
Place element x into its correct position in the output array 3 2 5 1 4 6 7 8 A 7 4 2 1 3 5 C 8 5 3 2 1 4 6 7 8 B

10. Counting Sort Alg.: COUNTING-SORT(A, B, n, k) for i ← 0 to k1 j n A
Alg.: COUNTING-SORT(A, B, n, k)
for i ← 0 to k
do C[ i ] ← 0
for j ← 1 to n
do C[A[ j ]] ← C[A[ j ]] + 1
C[i] contains the number of elements equal to i
for i ← 1 to k
do C[ i ] ← C[ i ] + C[i -1]
C[i] contains the number of elements ≤ i
for j ← n downto 1
do B[C[A[ j ]]] ← A[ j ]
C[A[ j ]] ← C[A[ j ]] - 1 k C 1 n B

11. Example 3 2 5 7 4 2 8 3 3 3 3 2 A C C B C B C B C B C A[8]=3 A[7]=0
CountingSort(A, B, k)
1. for i  1 to k
do C[i]  0
3. for j  1 to length[A]
do C[A[j]]  C[A[j]] + 1
5. for i  2 to k
do C[i]  C[i] + C[i –1]
7. for j  length[A] downto 1
do B[C[A[ j ]]]  A[j]
C[A[j]]  C[A[j]]–1 3 2 5 1 4 6 7 8 A C 7 4 2 1 3 5 C 8
A[8]=3
A[7]=0 3 1 2 4 5 6 7 8 B C 3 1 2 4 5 6 7 8 B C
A[6]=3
A[5]=2 3 1 2 4 5 6 7 8 B C 3 2 1 4 5 6 7 8 B C

12. Example (cont.) 3 2 5 3 2 5 3 2 3 2 5 3 2 A B C B C B C B A[4]=02 5 1 4 6 7 8 A
A[4]=0
A[2]=5 3 2 1 4 5 6 7 8 B C 5 3 2 1 4 6 7 8 B C
A[3]=3
A[1]=2 3 2 1 4 5 6 7 8 B C 5 3 2 1 4 6 7 8 B

13. Counting-Sort (A, B, k) O(k) O(n) O(k) O(n) CountingSort(A, B, k)
1. for i  1 to k
do C[i]  0
3. for j  1 to length[A]
do C[A[j]]  C[A[j]] + 1
5. for i  2 to k
do C[i]  C[i] + C[i –1]
7. for j  length[A] downto 1
do B[C[A[ j ]]]  A[j]
C[A[j]]  C[A[j]]–1
O(k)
Stable, but not in place.
No comparisons made: it uses actual values of the elements to index into an array.
O(n)
O(k)
O(n)
Overall time: (n + k)

14. Radix Sort It was used by the card-sorting machines.
Card sorters worked on one column at a time.
It is the algorithm for using the machine that extends the technique to multi-column sorting.
The human operator was part of the algorithm!
Key idea: sort on the “least significant digit” first and on the remaining digits in sequential order. The sorting method used to sort each digit must be “stable”.
If we start with the “most significant digit”, we’ll need extra storage.

15. Radix Sort Considers keys as numbers in a base-k number
A d-digit number will occupy a field of d columns
Sorting looks at one column at a time
For a d-digit number, sort the least significant digit first
Continue sorting on the next least significant digit, until all digits have been sorted
Requires only d passes through the list

16. An Example
Input
After sorting
on LSD
After sorting
on middle
digit
After sorting
on MSD
928    495
  

17. 1 is the lowest order digit, d is the highest-order digit
Radix-Sort(A, d)
RadixSort(A, d)
1. for i  1 to d
do use a stable sort to sort array A on digit I
1 is the lowest order digit, d is the highest-order digit
Correctness of Radix Sort
By induction on the number of digits sorted.
Assume that radix sort works for d – 1 digits.
Show that it works for d digits.
Radix sort of d digits  radix sort of the low-order d – 1 digits followed by a sort on digit d .

18. Correctness of Radix sort
We use induction on the number d of passes through the digits
Basis: If d = 1, there’s only one digit, trivial
Inductive step: assume digits 1, 2, , d-1 are sorted
Now sort on the d-th digit
If ad < bd, sort will put a before b: correct
a < b regardless of the low-order digits
If ad > bd, sort will put a after b: correct
a > b regardless of the low-order digits
If ad = bd, sort will leave a and b in the
same order and a and b are already sorted
on the low-order d-1 digits

19. Analysis of Radix Sort
Given n numbers of d digits each, where each digit may take up to k possible values, RADIX-SORT correctly sorts the numbers in (d(n+k))
One pass of sorting per digit takes (n+k) assuming that we use counting sort
There are d passes (for each digit)

20. Bucket Sort Assumption: Idea:
the input is generated by a random process that distributes elements uniformly over [0, 1)
the discrete () uniform distribution is a discrete probability distribution that can be characterized by saying that all values of a finite set of possible values are equally probable.
Idea:
Divide [0, 1) into n equal-sized buckets
Distribute the n input values into the buckets
Sort each bucket
Go through the buckets in order, listing elements in each one
Input: A[1 . . n], where 0 ≤ A[i] < 1 for all i
Output: elements in A sorted
Auxiliary array: B[0 . . n - 1] of linked lists, each list initially empty

21. Example - Bucket Sort 1
.78
.17
.39
.26
.72
.94
.21
.12
.23
.68 / 2 1
.17 / 3 2
.26
.21 / 4 3 / 5 4 6 5 7 / 6 8 7
.78 / 9 8 10 9 /

22. Example - Bucket Sort Concatenate the lists from
.17
.12
.23
.26
.21
.39
.68
.78
.72 / / 1
.12 / 2
.21
.23 / 3 / 4 5 / 6 7
.72 /
Concatenate the lists from
0 to n – 1 together, in order 8 9 /

23. Analysis Relies on no bucket getting too many values.
All lines except insertion sorting take O(n) altogether.
Intuitively, if each bucket gets a constant number of elements, it takes O(1) time to sort each bucket  O(n) sort time for all buckets.
We “expect” each bucket to have few elements, since the average is 1 element per bucket.
But we need to do a careful analysis.

24. Summary Non-Comparison Based Sorts
Running Time
worst-case average-case best-case in place
Counting Sort O(n + k) O(n + k) O(n + k) no
Radix Sort O(d(n + k')) O(d(n + k')) O(d(n + k')) no
Bucket Sort O(n) no
Counting sort assumes input elements are in range [0,1,2,..,k] and uses array indexing to count the number of occurrences of each value.
Radix sort assumes each integer consists of d digits, and each digit is in
range [1,2,..,k'].
Bucket sort requires advance knowledge of input distribution (sorts n numbers uniformly distributed in range in O(n) time).