1. Ch8: Sorting in Linear Time Ming-Te Chi
Algorithms
Ch8: Sorting in Linear Time
Ming-Te Chi
Ch8 Sorting in Linear Time

2. Ch8 Sorting in Linear Time
Overview
This week topics：sorting
Next topics：searching and related subjects.
Computer is powerful in manipulating data
Store data and perform operations on data
Search/retrieve
Insert
Delete
Other operations
Ch8 Sorting in Linear Time

3. Sorting – how fast can we sort?
Selection or insertion sort
Merge sort
Heap sort
Quick sort
. . .
Complexity (time) is either O(n log n) or O(n2)
Ch8 Sorting in Linear Time

4. Lower bounds for sorting
to examine all the input.
All sorts seen so far are
We’ll show that is a lower bound for comparison sorts.
Ch8 Sorting in Linear Time

5. Sorting based on Comparison
Selection sort, insertion sort, merge sort, heap sort, quick sort
All have time complexity
Based on comparison between the input elements
Comparison sorts can be viewed abstractly in terms of decision tree
Ch8 Sorting in Linear Time

6. Ch8 Sorting in Linear Time
Decision tree
Fully binary tree
Abstraction of any comparison sort.
Represents comparisons made by
a specific sorting algorithm
on inputs of a given size
Abstracts away everything else: control and data movement.
We’re counting only comparisons
Ch8 Sorting in Linear Time

7. Example: Decision Tree for Sorting Three Element
{a, b, c}
a:b
a ≤ b
a > b
a:c
b:c
b ≤ c
a ≤ c
a > c
b > c
b:c
cab
a:c
cba
b ≤ c
a ≤ c
b > c
a > c
abc
acb
bac
bca
Ch8 Sorting in Linear Time

8. Ch8 Sorting in Linear Time
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Ch8 Sorting in Linear Time

9. Ch8 Sorting in Linear Time
Each leaf is labeled by the permutation of orders that the algorithm determines
Any correct sorting algorithm must be able to produce each permutation of its input.
A correct comparison sort algorithm requires that each of the permutations must appear as one of the leaves of the decision tree.
Ch8 Sorting in Linear Time

10. Ch8 Sorting in Linear Time
Each of these leaves must be reachable from the root by a path corresponding to an actual execution of the comparison sort.
The execution of the sorting algorithm corresponds to tracing a path from the root of the decision tree to a leaf.
Ch8 Sorting in Linear Time

11. Ch8 Sorting in Linear Time
How many leaves?
What is the height of the tree?
Ch8 Sorting in Linear Time

12. Properties of the decision tree
At least n! leaves
For any comparison sort
1 tree for each n
View the tree as if the algorithm splits in two at each node, based on the information it has determined up to that point.
The tree models all possible execution traces.
Ch8 Sorting in Linear Time

13. Properties of the decision tree (continue)
Length of the longest path from root to leaf?
The worst-case number of comparisons that the corresponding sorting algorithm performs.
Depends on the algorithm
Insertion sort:
Merge sort:
Ch8 Sorting in Linear Time

14. Ch8 Sorting in Linear Time

15. Ch8 Sorting in Linear Time

16. Bucket Sort: linear time sorting
Allocate sufficient number of buckets &
put element in corresponding buckets
at the end, scan the buckets in order & collect all elements
n elements, ranges from 1 to m  m buckets
O(m+n)
Best case: O(n), Worst case: O(m)
Ch8 Sorting in Linear Time

17. Sorting in linear time: 8.2 Counting sort (similar to bucket sort)
Depends on a key assumption:
n numbers to be sorted are integers in {0, 1, , k}.
Input: A[1 . . n], where A[ j ] ∈ {0, 1, , k} for j = 1, 2, , n.
(Array A and values n and k are given as parameters.)
Output: B[1 . . n], sorted.
(B is assumed to be already allocated and is given as a parameter.)
Auxiliary storage: C[0 . . k]
Ch8 Sorting in Linear Time

18. Sorting in linear time: 8.2 Counting sort
Ch8 Sorting in Linear Time

19. Ch8 Sorting in Linear Time
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Ch8 Sorting in Linear Time

20. Ch8 Sorting in Linear Time

21. Ch8 Sorting in Linear Time
How big a k is practical?
Good for sorting 32-bit values? No.
16-bit? Probably not.
8-bit? Maybe, depending on n.
4-bit? Probably (unless n is really small)
Example: playing cards. (撲克牌)
Counting sort (bucket sort) will be used in radix sort.
Ch8 Sorting in Linear Time

22. Ch8 Sorting in Linear Time
8.3 Radix sort
Used by the card-sorting machines one can now find only in computer museum.
Sort the least significant digit first
RADIX_SORT(A,d)
1 for i  1 to d
do use a stable sort to sort array A on digit i
Ch8 Sorting in Linear Time

23. Ch8 Sorting in Linear Time

24. Ch8 Sorting in Linear Time
Analysis: Radix sort
Assume that we use counting sort as the intermediate sort.
per pass (digits in range 0, , k)
d passes
total
If , time complexity is
Not in-place (in-place: only constant number of elements of the input array are ever stored outside the array.
Ch8 Sorting in Linear Time

25. Ch8 Sorting in Linear Time

26. Ch8 Sorting in Linear Time

27. Ch8 Sorting in Linear Time
Sample question:
What is the decision tree?
Explain how it can be used to prove the lower bound of the comparison based sorting algorithms.
It is well known that radix sort has the time complexity of O(d n), where d is the maximal number of digits of all the inputs and n is the total number of inputs.
Are these two results contradict to each other? Justify your answer.
Ch8 Sorting in Linear Time

28. Bucket sort (new version)
Divide the interval [0, 1) into n equal-sized subintervals (buckets).
Distribute the n input numbers into the buckets.
Input: n-element array A, with
Needs an auxiliary array B[0..n-1] of linked lists.
Ch8 Sorting in Linear Time

29. Ch8 Sorting in Linear Time
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Ch8 Sorting in Linear Time

30. Ch8 Sorting in Linear Time
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Ch8 Sorting in Linear Time

31. Ch8 Sorting in Linear Time
Analysis
The running time of bucket sort is
where ni is the number of elements placed in the bucket B[i].
Why?
Ch8 Sorting in Linear Time

32. Ch8 Sorting in Linear Time
Analysis (continue)
Taking expectations of both sides and using linearity of expectation, we have
Ch8 Sorting in Linear Time

33. Ch8 Sorting in Linear Time
We claim that
We define indicator random variables
Xij = I {A[j] falls in bucket i}
for i = 0, 1, …, n-1 and j = 1, 2,…,n. thus,
Ch8 Sorting in Linear Time

34. Ch8 Sorting in Linear Time

35. Ch8 Sorting in Linear Time
Indicator random variable Xij is 1 with probability 1/n and 0
otherwise, and therefore
When k  j, the variables Xij and Xik are independent, and hence
Ch8 Sorting in Linear Time

36. Ch8 Sorting in Linear Time
We can conclude that the expected time for bucket sort is
(n)+n·O(2-1/n)= (n).
Ch8 Sorting in Linear Time

37. Ch8 Sorting in Linear Time

38. Ch8 Sorting in Linear Time
Remarks:
Decision tree proves that O(n log n) is required for sorting
Radix sort (or bucket sort) states that O(n) is enough for sorting
Is there anything wrong?
Ch8 Sorting in Linear Time