1. Lecture 5: Master Theorem and Linear Time Sorting
Shang-Hua Teng

2. The Master Theorem Consider where a >= 1 and b>= 1
we will ignore ceilings and floors (all absorbed in the O or Q notation)
if n < c
if n > 1

3. The Master Theorem If for some constant e > 0 then If then
If for some constant e > 0 and if a f(n/b) <= c f(n) for some constant c < 1 and all sufficiently large n, then T(n) =Q(f (n))

4. Divide-and-Conquer Tree

5. How fast can we sort? Can we do better than O(n lg n)?
Well… it depends?
It depends on the model of computation!

6. Comparison sorting
Only comparisons are used to determine the order of elements
Example
Insertion sort
Merge Sort
Quick Sort
Heapsort

7. Decision Trees Example: 3 element sort (a1, a2, a3) a1? a2
“Less than” leads to left branch
“Great than” leads to right branch

8. Decision Trees
Example: 3 element sort (a1, a2, a3)
a1? a2
a2? a3

9. Decision Trees Example: 3 element sort (a1, a2, a3) a1? a2 a2? a3

10. Decision Trees Example: 3 element sort (a1, a2, a3) a1? a2 a2? a3

11. Decision Trees Example: 3 element sort (a1, a2, a3) a1? a2 a2? a3

12. Decision Trees Example: 3 element sort (a1, a2, a3) a1? a2 a2? a3

13. Decision Trees Example: 3 element sort (a1, a2, a3) a1? a2 a2? a3

14. Decision Trees Example: 3 element sort (a1, a2, a3) a1? a2 a2? a3

15. Decision Trees Example: 3 element sort (a1, a2, a3) a1? a2 a2? a3

16. Decision Trees
Each leaf contains a permutation indicating that permutation of order has been established
A decision tree can model a comparison sort
It is the tree of all possible instruction traces
Running time = length of path
Worst case running time = length of the longest path (height of the tree)

17. Lower bound for comparison sorting
Theorem: Any decision tree that sorts n elements has height W(n lg n)
Proof: There are n! leaves
A height h binary tree has no more than 2h nodes

18. Optimal Comparison Sorting Algorithms
Merge sort
Quick sort
Heapsort

19. Sorting in Linear Time Counting sort Radix sort
Suppose we only have k keys {1,2,…,k}
Q ( n+k ) time
Radix sort
Can use counting sort
Place by place (digit by digit ) sorting
IBM Human machine interface
If elements had d places (digits), then we need d calls to counting sort

20. Counting Sort Counts how many occurrences of each key
Allocate proper amount of spaces for each key, from small to large
For each element a[j] in the input, count the number of elements that are less than the element
A[k] is “Less than” A[j] if
A[k] < A[j] or
A[k] = A[j] and k < j

21. Example of Counting Sort
A = [ ], we have four keys
C =[ ]
Prefix sum of C
Prefix_sum( C ) = 0 [ ]
First key goes from 1 to 1
Second key goes from 2 to 1 (empty)
Third key goes from 2 to 3
Fourth key goes from 4 5
Sorted array is = [ ]

22. Counting Sort Counting_sort (A, 1, n) for i = 0 to kdo
for j = 1 to length(A)
for i = 1 to k
for j = length(A) downto 1
Zero out the counting array
C[j] now contains the number
of elements equal to i
C[j] now contains the number
of elements less than or
equal to i

23. Stable sorting It is stable
Stable: after sorting elements with the same values appear in the output array in the same order as they do in the input array
An important property of counting sort
It is stable

24. Complexity of Counting Sort
Q(n+k)

25. Radix Sort
329
457
657
839
436
720
355

26. Radix Sort
329
457
657
839
436
720
355
720
355
436
457
657
329
839

27. Radix Sort
329
457
657
839
436
720
355
720
355
436
457
657
329
839
720
329
436
839
355
457
657

28. Radix Sort
329
457
657
839
436
720
355
720
355
436
457
657
329
839
720
329
436
839
355
457
657
329
355
436
457
657
720
839

29. Correctness Induction on digit position
Assume numbers are sorted by low-order t-1 digits
Sort on digit t
Two numbers that differ in digit t are correctly sorted
2 number = in digit t are put in same order as input, implying correct order
Spreadsheets use stable sort, so you can try by hand if you have spreadsheets

30. Time Complexity of Radix sort
Sort n words with b bits each
b passes of counting sort on 1-bits digits
Or b/2 passes … 2-bits digits
Or b/r passes … r-bits digits
r = log n
Keys from {1,…,nd} needs Q(dn) time