1. CS 583 Analysis of Algorithms
Sorting in Linear Time
CS 583
Analysis of Algorithms
5/5/2019
CS583 Fall'06: Sorting in Linear Time

2. CS583 Fall'06: Sorting in Linear Time
Outline
Comparison Sort Algorithms
Lower Bounds for Sorting
Counting Sort
Order Statistics
Minimum and Maximum
Selection
5/5/2019
CS583 Fall'06: Sorting in Linear Time

3. CS583 Fall'06: Sorting in Linear Time
Comparison Sorts
We have seen several algorithms that can sort n numbers in O(n lg n) time.
Merge sort and heapsort achieve this upper bound in the worst case; quicksort achieves it on average.
These algorithms have one common property: the sorted order they determine is based only on comparisons between the input elements. Such sorting algorithms are called comparison sorts. 
We prove that any comparison sort must make (n lg n) comparisons in the worst case to sort n elements.
5/5/2019
CS583 Fall'06: Sorting in Linear Time

4. Comparison Sort: Decision Tree
We assume without loss of generality that all input elements are distinct. In this case, we can simply make comparisons of one form, for example, ai <= aj.
Comparison sorts can be viewed in terms of decision tree. For example, sorting three elements using insertion sort will look as follows:
1:2
<= >
2: :3
<= >
<1,2,3> 1:3
<= >
<1,3,2> <3,1,2> ...
5/5/2019
CS583 Fall'06: Sorting in Linear Time

5. Comparison Sort: Decision Tree (cont.)
In a decision tree each internal node is annotated by i:j for some i and j in the range 1 <= i,j <= n.
Each leaf is annotated by a permutation <(1), ... , (n)>.
The execution of the sorting algorithm corresponds to tracing a path from the root to a leaf.
Any correct sorting algorithm must be able to produce each permutation of its input, hence all n! leaves of the decision tree must be "reachable".
5/5/2019
CS583 Fall'06: Sorting in Linear Time

6. Lower Bound for Comparison Sorts
Theorem 8.1
Any comparison sort algorithm requires (n lg n) comparisons in the worst case.
Proof.
The length of the longest path from the root of a decision tree to any of its reachable trees represents the worst-case number of comparisons that a sorting algorithm performs. Hence, we need to determine the height of the decision tree, where each permutation is a reachable leaf.
5/5/2019
CS583 Fall'06: Sorting in Linear Time

7. Lower Bound for Comparison Sorts (cont.)
Consider a tree of height h and l leaves. Since each permutation appears as a leaf, we have n! <= l. A binary tree of height h has no more than 2h leaves:
n! <= l <= 2h =>
h >= lg(n!)
lg(n!) = (n lg n) (see 3.18) =>
h >= (n lg n) => h = (n lg n)  
5/5/2019
CS583 Fall'06: Sorting in Linear Time

8. CS583 Fall'06: Sorting in Linear Time
Counting Sort
This algorithm assumes that each of the n input elements is an integer in the range 0 to k.
When k = O(n), the sort runs in (n) time. 
The basic idea is to determine for each input element x, the number of elements less than x.
This information can be used to place x directly into its position in the output array.
The algorithm requires an input array A, the output array B, and an intermediate (“counting”) array C.
5/5/2019
CS583 Fall'06: Sorting in Linear Time

9. Counting Sort: Example
n=5, k=2:
C after steps 1-4:
0 1 2
1 1 3
C after loop 6:
1 2 5
B in loop 9:
index 2 1
5/5/2019
CS583 Fall'06: Sorting in Linear Time

10. Counting Sort: Pseudocode
Counting-Sort (A,B,n,k)
1 for i = 0 to k
2 C[i] = 0
3 for i = 0 to n
4 C[A[i]]++
5 // C[i] contains the number of elements = i
6 for i = 1 to k
7 C[i] = C[i] + C[i-1]
8 // C[i] now contains number of elements <= i
9 for i = n to 1
10 B[C[A[i]]] = A[i]
11 C[A[i]]--
12 return
5/5/2019
CS583 Fall'06: Sorting in Linear Time

11. Counting Sort: Performance
After loop 6, the array C contains the first position of an element with value i, which is the same as the number of elements <= i. At each iteration, when the i element is placed into the output array, the position of the next element i will be before the current one.
To calculate the running time, observe that the number of operations is k+1(loop 1) + n+k+n = (k+n). When using k=O(n), we have the running time (n). 
An important quality of the counting sort is that it is stable, numbers with the same value appear in the output array in the same order as they do in the input array. The property of stability is important when satellite date are carried around with the key.
5/5/2019
CS583 Fall'06: Sorting in Linear Time

12. CS583 Fall'06: Sorting in Linear Time
Order Statistics
The ith order statistics of a set of n elements is the ith smallest element.
The minimum element is the first order statistics (i=1).
The maximum element is the last order statistics (i=n).
A median is a the “half point” of the set (i=(n+1)/2).
The selection problem is finding the ith order statistics from a set of n distinct numbers.
It can be solved in O(n lg n) time by sorting elements, and then selecting the ith element from the sorted array.
The fastest algorithm runs in O(n) time in the worst case.
5/5/2019
CS583 Fall'06: Sorting in Linear Time

13. Minimum/Maximum: Pseudocode
MINIMUM(A)
1 min = A[1]
2 for i = 2 to length[A]
3 if A[i] < min
min = A[i]
5 return min
The above algorithm makes (n-1) comparisons. Finding the maximum can be accomplished with (n-1) comparisons as well:
MAXIMUM(A)
1 max = A[1]
3 if A[i] > max
max = A[i]
5 return max
5/5/2019
CS583 Fall'06: Sorting in Linear Time

14. Simultaneous Minimum and Maximum
MINMAX(A)
1 min = A[1]
2 max = A[1]
3 i = 2
4 while (i <= length[A])
5 if (i+1) > length[A]
x_min = A[i]; x_max = A[i]
7 else
if A[i] < A[i+1]
x_min = A[i]; x_max = A[i+1]
else
x_min = A[i+1]; x_max = A[i]
12 if x_max > max
max = x_max
14 if x_min < min
min = x_min
16 i += 2
17 return (min, max)
The above algorithm performs at most 5n/2 comparisons to find both minimum and maximum, and hence runs in (n) time.
5/5/2019
CS583 Fall'06: Sorting in Linear Time

15. CS583 Fall'06: Sorting in Linear Time
General Selection
The general selection algorithm finds an ith order statistics. The algorithm below is modeled after a quicksort algorithm with expected running time O(n).
RANDOMIZED-SELECT (A,p,r,i)
1 if p=r
2 return A[p]
3 q = RANDOMIZED-PARTITION(A,p,r)
4 k = q-p+1
5 if i = k // the pivot element is the answer
6 return A[q]
7 else
8 if i<k
return RANDOMIZED-SELECT(A,p,q-1,i)
10 else
return RANDOMIZED-SELECT(A,q+1,r,i-k)
5/5/2019
CS583 Fall'06: Sorting in Linear Time