1 Today’s Material Lower Bounds on Comparison-based Sorting Linear-Time Sorting Algorithms –Counting Sort –Radix Sort

2 How Fast Can We Sort? Basic Sorting Algorithms –BubleSort, InsertionSort and SelectionSort all run in O(N^2) Fast Sorting Algorithms –Mergesort, Heapsort and Quicksort all run in O(N log N) best case running time Can we do any better? –Can we come up with a sorting algorithm that will sort in O(N log logN)? What about O(N)?

3 Lower Bound on Comparison-based Sorting Recall our basic assumption: we can only compare two elements at a time –Suppose you are given N elements –How many possible orderings can you get? Example: a, b, c (N = 3) How many distinct sequences exist? Orderings: 1.a b c 2.b c a 3.c a b 4.a c b 5.b a c 6.c b a N = 3: We have 6 orderings = 321 = 3! N choices N-1 choices N-2 choices 1 choice For N elements: –  N! orderings

4 A “Decision Tree” for Sorting N=3 Elements Possible Orderings a < b < c, b < c < a, c < a < b, a < c < b, b < a < c, c < b < a Remaining Orderings a < b < c c < a < b a < c < b b < c < a b < a < c c < b < a a < b a > b b < c < a b < a < c c < b < a b < c < a b < a < c a < c a > c a < b < c a < c < b c < a < b b > c b < c a < b < c a < c < b b < c b > c a > c a < c Leaves contain all possible orderings of a, b, c

5 Decision Trees and Sorting A Decision Tree for Sorting is a Binary Tree such that: –Each node = a set of orderings –Each edge = 1 comparison –Each leaf = 1 unique ordering How many leaves for N distinct elements? –Only 1 leaf has correct sorted ordering for given a, b, c Each sorting algorithm corresponds to a decision tree –Finds correct leaf by following edges (= comparisons) Run time >= maximum no. of comparisons –Depends on: depth of decision tree –What is the depth of a decision tree for N distinct elements?

6 Decision Trees and Sorting Suppose you have a binary tree of depth d. How many leaves can the tree have? –E.g. Depth = 1  at most 2 leaves –Depth = 2  at most 4 leaves, etc. –Depth = d  at most 2 d leaves. Easy to prove. Number of leaves L = log(L) –Decision tree has L = N! leaves –Depth d >= log(N!) –What is log(N!)? log(N!) = log N + log(N-1) + … log(N/2) + … + log 1 >= log N + log(N-1) + … log(N/2) (N/2 terms only) >= (N/2)log(N/2) = (N log N) Result: Any sorting algorithm based on comparisons between elements requires (N log N) comparisons

7 Using the Lower Bound for Sorting Lower bound that we proved for sorting is in fact used to prove lower bound for other problems. Here is how: –Say you have a problem for which you want to prove a lower bound –If we can reduce the sorting problem to the problem to be solved, then we have essentially proved that the lower bound for the new problem is nlogn. Why? Because if we can solve your problem in less time than nlogn, then we can solve sorting in less time than nlogn. And we have just proved that that is not possible!

8 Example  (NlogN) Problems Convex Hull –Given a set of points in the plane, find the closest convex polygon that encloses them Closest Pair –Given a list of n elements, which pair are closest in value Element Uniqueness –Given a list of n numbers, are there any duplicates in the list

9 What about Counting Sort? Problem: Sort integers in the range 1 to B Algorithm: 1.Allocate an array Count having B slots (“buckets”) 2.Initialize: Count[i] = 0 for i = 1 to B 3.Given input integer i, Count[i]++ 4.After reading all inputs, scan Count[i] for i = 1 to B and print i if Count[i] is non-zero Ex: Sort the following integers in the range 1 to 9 –

10 What if A holds records with integer key? // A[1..n]: Holds the initial input. A[I].key is the integer // key value on which to sort on. A[I].data is the other satellite data // B[1..n]: Holds the sorted output // Count[1..B]: Count[I] is the rank of I, that is, the number of // elements of A whose key value is less than or equal to i CountingSort(A, B, n, B) { for I=1 to B do Count[I] = 0; for j=1 to n do Count[A[j].key]++; for I=2 to B do Count[I] += Count[I-1]; for j=n downto 1 do { I = A[j].key; B[Count[I]] = A[j]; Count[I]--; } //end-for } //end-CountingSort

11 Counting Sort Example aers v A v B sv B srv B srv e B asrv e B 0224

12 Counting Sort Running Time What is the running time for sorting N integers using bucket sort? –Running Time: O(B+N) –B to zero/scan the array and N to read the input –If B is O(N), then running time for Bucket sort = O(N) –Doesn’t this violate the (N log N) lower bound result? –No – When we do Count[i]++, we are comparing one element with all B elements, not just two elements Not regular 2-way comparison-based sorting

13 Radix Sort: Stable Counting Sort Problem: What if number of buckets needed is too large? Recall: Stable sort = a sort that does not change order of items with same key Radix sort = stable bucket sort on “slices” of key 1.Divide integers/strings into digits/characters 2.Bucket-sort from least significant to most significant digit/character –Uses linked lists –Stability ensures keys already sorted stay sorted –Takes O(P(B+N)) time where P = number of digits

14 Radix Sort Example [4] 19[4] 95[4] 57[6] 29[6] 17[6] 27[8] 9[5]4 5[7]6 1[7]6 2[7]8 4[9]4 1[9]4 2[9]6 [1]76 [1]94 [2]78 [2]96 [4]94 [5]76 [9]

15 Radix Sort Running Time P = # of digits iterations over the data set Each iteration takes O(B+N) time –Total: O(P(B+N)) time where P = number of digits E.g., N numbers in base 10 in the range (max. 6 digits) –Total: O(6*(10+N)) –Total: O(60 + 6N)

16 Summary of Sorting Sorting choices: –O(N 2 ) – Bubblesort, Selection Sort, Insertion Sort –O(N x ) – Shellsort (x = 3/2, 4/3, 2, etc. depending on incr. seq.) –O(N log N) average case running time: Mergesort: easy to code but uses O(N) extra space Heapsort: needs 2 comparisons to move data (between 2 children and parent) – may not be fast in practice Quicksort: fastest in practice but trickier to code, O(N 2 ) worst case –O(P·N) Radix sort (using Counting sort) for special cases where keys are P digit integers/strings