1 Algorithm Efficiency and Sorting (Walls & Mirrors - Remainder of Chapter 9)

2 Overview Mergesort Quicksort Finding the K th Smallest Item Radix Sort

3 Mergesort: Basic Idea 1) Split an array into two halves. 2) Sort each half by recursively invoking mergeSort. 3) Merge the two sorted halves in the correct order. If we do enough splitting, we will eventually obtain an array with one element, which is trivial to sort — i.e. do nothing (base case). Most of the work goes into merging.

4 Mergesort: Example Problem: Sort the following array from smallest to largest SPLIT mergeSort mergeSort merge

5 Mergesort: Example (Cont’d.) Details behind the two calls to mergeSort (on previous slide): SPLIT SPLIT merge merge 1638 mergeSort 1239 mergeSort 27 mergeSort 27 mergeSort

6 Mergesort: Example (Cont’d.) Details behind the four calls to mergeSort (on previous slide): 1638 SPLIT 1239 SPLIT 27 mergeSort 27 mergeSort mergeSort mergeSort mergeSort mergeSort merge 3912 merge

7 Mergesort: C++ Implementation void mergeSort( DataType a[ ], int first, int last ) { if( first < last ) { // find the midpoint of a[first.. last] int mid = (first + last) / 2; // sort each half mergeSort( a, first, mid ); mergeSort( a, mid + 1, last ); // merge the two halves merge( a, first, mid, last ); }

8 Merge: Basic Idea Problem: Merge sorted sub-arrays, a[ first.. mid ] and a[ mid+1.. last ], so that a[ first.. last ] is put in sorted order. 1) Set first1 = first and first2 = mid ) If a[first1] < a[first2], copy a[first1] into array temp[ ] and increment first1; otherwise, copy a[first2] into temp[ ] and increment first2. 3) Repeat step 2 with new first1 or new first2 until all elements of either a[ first.. mid ] or a[ mid+1.. last ] have been copied into temp[ ]. 4) If all elements of a[ first.. mid ] were copied to temp[ ], then copy remaining elements of a[ mid+1.. last ] into temp[ ]. Otherwise, copy remaining elements of a[ first.. mid ] into temp[ ]. 5) Copy temp[ ] into a[ first.. last ].

9 Merge: Example first1first2 temp[ ]: a[ ]: index

10 Merge: C++ Implementation void merge( DataType a[ ], int first, int mid, int last ) { DataType temp[MaxSize]; // in turn, copy the smallest of a[ first.. mid ] or a[ mid+1.. last ] into // temp[ ] until either sub-array has been copied into temp[ ] int first1 = first, last1 = mid, first2 = mid + 1, last2 = last, index; for( index = first1; first1 <= last1 && first2 <= last2; index++ ) if( a[ first1 ] < a[ first2 ] ) { temp[ index ] = a[ first1 ]; first1++; } else { temp[ index ] = a[ first2 ]; first2++; } // copy the remaining elements of a[ first.. mid ] or // a[ mid+1.. last ] into temp[ ]; copy temp[ ] into a[ ] }

11 Merge: C++ Implementation (Cont’d.) // copy the remaining elements of a[ first.. mid ] into temp[ ] for( ; first1 <= last1; first1++, index++ ) temp[ index ] = a[ first1 ]; // copy the remaining elements of a[ mid+1.. last ] into temp[ ] for( ; first2 <= last2; first2++, index++ ) temp[ index ] = a[ first2 ]; // copy temp[ first.. last ] into a[ first.. last ] for( index = first; index <= last; index++ ) a[ index ] = temp[ index ];

12 Mergesort: Efficiency Each merge step merges a[ first.. mid ] with a[ mid+1.. last ]. If the total number of elements in these two sub- arrays is n, then merging them requires at most n – 1 comparisons. Each merge step involves n moves from a[ ] to temp[ ] and n moves from temp[ ] back to a[ ]. Therefore, each merge step requires at most n – 1 + n + n = 3 * n – 1 data-item operations to merge n elements.

13 Mergesort: Efficiency (Cont’d.) Each call to mergeSort( ) divides the input array in half and then calls mergeSort( ), recursively, on each half. As we saw with binary search, if n = 2 k, for some k, then we can divide an array of n things k times, where k = log 2 n. If n is not a power of 2 then, as we also saw with binary search, k = 1 +  log 2 n  = 1 + log 2 n (rounded down). At level m of the recursion, 2 m calls to merge occur. Each call merges n /2 m elements, and consequently, requires 3 * (n/2 m ) – 1 major operations. The 2 m calls to merge at level m require at most 2 m * [ 3 * (n/2 m ) – 1 ] = 3 * n – 2 m = O( n ) operations

14 Mergesort: Efficiency (Cont’d.) Since there are either log 2 n or 1 + log 2 n levels of recursion (depending on the number of times an array of n things can be divided in half), it follows that, in worst case, mergeSort( ) is O( n * (1 + log 2 n) ) = O( n * log 2 n ) In the average case, Mergesort is also O( n * log 2 n ) Note that, after dividing an array in half, binary search calls itself recursively on only one half of the array. mergeSort( ), on the other hand, calls itself recursively on both halves of the array.

15 Quicksort: Basic Idea 1) Choose a pivot from the array a[ first.. last ] and swap elements, if necessary, to place it in a[ first ]. (Later, we will indicate a strategy for choosing the pivot.) 2) Compare each element in a[ first last ] with the pivot. –If an element is < the pivot, add it to an (initially empty) block of numbers, S1, that are all < the pivot. –If an element is  the pivot, add it to an (initially empty) block of numbers, S2, that are all  the pivot. 3) Arrange a[ first.. last ] to be in the order S1 pivot S2. 4) Invoke quicksort( ) recursively on S1 and on S2. Most of the work goes into forming S1 and S2.

16 Quicksort: Observations Let pivotIndex be the index of the pivot in a[ first.. last ] after a[ first.. last ] is arranged in the order S1 pivot S2. We cannot expect that a[ first.. last ] is now completely sorted. However: The elements in a[ first.. pivotIndex – 1] will remain in a[ first.. pivotIndex – 1] when a[ first.. last ] is completely sorted. (Their positions relative to each other may change.) Likewise, the elements in a[ pivotIndex last ] will remain in a[ pivotIndex last ] when a[ first.. last ] is completely sorted. The pivot will remain at a[ pivotIndex ] when a[ first.. last ] is completely sorted.

17 Quicksort: Example Problem: Sort the following array from smallest to largest. Arbitrarily select a[0] as the pivot. S1 and S2 are empty. We now compare each element in a[1.. 5] with the pivot. Since a[1]  pivot, we add it to S2, yielding a[ ]: a[ ]: pivotS2UNKNOWN

18 Since a[2] < pivot, we add it to S1 as the first element. However, we swap a[2] with a[1] to keep S1 before S2, so that S1 pivot S2 is easy to form. This yields Quicksort: Example (Cont’d.) a[ ]: pivotS2UNKNOWN a[ ]: pivotS1UNKNOWNS2

19 Since a[3]  pivot, we add it to S2, yielding Note that this step required no elements of a[0.. 5] to move. Quicksort: Example (Cont’d.) a[ ]: pivotS1UNKNOWNS a[ ]: pivotS1UNKNOWNS2

20 Since a[4]  pivot, we add it to S2, yielding Note that this step required no elements of a[0.. 5] to move. Quicksort: Example (Cont’d.) a[ ]: pivotS1UNKNOWNS a[ ]: pivotS1UNKNOWNS2

21 Since a[5] < pivot, we add it to S1 by swapping it with the first element of S2, yielding Note that we did not try to insert a[5] as the first element of S1. This would require extra work. Quicksort: Example (Cont’d.) a[ ]: pivotS1UNKNOWNS a[ ]: pivotS1S2

22 Note that every element of a[0.. 5] is in either S1, S2, or pivot. We now arrange a[0.. 5] in the order: S1 pivot S2. This is done by swapping the pivot with the last element of S1, yielding Note that we did not shift all the elements of S1 in order to make room for the pivot after the last element of S1. This would require extra work. Quicksort: Example (Cont’d.) a[ ]: pivotS1S a[ ]: pivot S1S2

23 Note that a[0.. 5] is not yet sorted. However, every element of a[0.. 5] to the left of pivot is < pivot, and every element to the right of pivot is  pivot. (This arrangement is called a partition of a[0.. 5].) When a[0.. 5] is finally sorted, everything that is now to the left of pivot will remain on the left, and everything that is now to the right of pivot will remain on the right. Consequently, pivot will remain in its current position when a[0.. 5] is completely sorted. Quicksort: Example (Cont’d.) a[ ]: pivot S1S2

24 We now invoke quicksort( ) recursively on S1 and on S2. Consider first invoking quicksort( ) on S1: As before, we arbitrarily select a[0] as the pivot. Set new S1 and S2 to empty. Since a[1] < pivot, we add a[1] to S1, yielding Quicksort: Example (Cont’d.) a[ ]: pivot S1S2 1113a[ ]: 10

25 Quicksort: Example (Cont’d.) Now, every element of a[0.. 1] is in either S1, S2, or pivot (S2 is empty). We arrange a[0.. 1] in the order: S1 pivot S2. This is done by swapping the pivot with a[1], yielding 1113a[ ]: 10 pivotS1 1311a[ ]: 10 S1pivot

26 Quicksort: Example (Cont’d.) The next step is to invoke quicksort( ) recursively on S1. Since S1 contains only one element, quicksort( ) immediately returns without doing any work. Now, consider invoking quicksort( ) on the original S2: 1311a[ ]: 10 S1pivot a[ ]:

27 Quicksort: Example (Cont’d.) Arbitrarily select a[3] as the pivot. Set new S1 and S2 to empty. We now compare each element in a[4.. 5] with the pivot. Since a[4] < pivot, we add it to S1, yielding a[ ]: a[ ]: pivotS1UNKNOWN

28 Quicksort: Example (Cont’d.) Since a[5]  pivot, we add it to S2, yielding Since every element of a[3.. 5] is in either S1, S2, or pivot, we now swap pivot with a[4] to arrange a[3.. 5] in the order: S1 pivot S2. This yields a[ ]: pivotS1UNKNOWN a[ ]: pivotS1S2

29 The next step is to invoke quicksort( ) recursively on S1 and on S2. Since S1 and S2 each contain only one element, quicksort( ) returns immediately from each recursive call without doing any work. Array a[0.. 5] now looks like this which is finally in sorted order. Quicksort: Example (Cont’d.) a[ ]: S1pivotS a[ ]:

30 Quicksort: C++ Implementation void quicksort( DataType a[ ], int first, int last ) { int pivotIndex; if( first < last ) { // select pivotIndex, and partition a[first.. last] so that each // element in a[first.. pivotIndex – 1] < a[pivotIndex], and // each element in a[pivotIndex last] >= a[pivotIndex] partition( a, first, last, pivotIndex ); // sort a[first.. pivotIndex – 1] = S1 quicksort( a, first, pivotIndex – 1 ); // sort a[pivotIndex last] = S2 quicksort( a, pivotIndex + 1, last ); }

31 Partition: Basic Idea Select pivotIndex, and partition a[ first.. last ] so that each element in S1 < a[ pivotIndex ], and each element in S2 >= a[ pivotIndex ] a[ ]: pivotUNKNOWN Starting configuration (S1 = S2 = empty) a[ ]: pivotS1UNKNOWNS2 General case

32 Partition: C++ Implementation void partition( DataType a[ ], int first, int last, int &pivotIndex ) { // select a pivot and place it in a[ first ] choosePivot( a, first, last ); DataType pivot = a[ first ]; int lastS1 = first, firstUnknown = first + 1; for( ; firstUnknown <= last; firstUnknown++ ) if( a[ firstUnknown ] < pivot ) { lastS1++; // swap a[ firstUnknown ] with first S2 swap( a[ firstUnknown ], a[ lastS1 ] ); } swap( a[ first ], a[ lastS1 ] ); pivotIndex = lastS1; }

33 ChoosePivot: C++ Implementation // one heuristic for choosing a Pivot and placing it in a[ first ] void choosePivot( DataType a[ ], int first, int last ) { int pivotIndex, mid = (first + last) / 2; // set pivotIndex to the index of the middle value of // a[ first ], a[ mid ], and a[ last ] if( a[ first ] <= a[ mid ] ) { if( a[ mid ] <= a[ last ] ) pivotIndex = mid; else pivotIndex = ( a[ first ] <= a[ last ] ? last : first ); } else if( a[ last ] <= a[ mid ] ) pivotIndex = mid; else pivotIndex = ( a[ first ] <= a[ last ] ? first : last ); swap( a[ first ], a[ pivotIndex ] ); }

34 Quicksort: Observations Note that the heuristic that selects a Pivot by choosing the middle value of a[ first ], a[ mid ], and a[ last ] guarantees that, while first < mid < last, S1 and S2 will each contain at least one element. Consequently, while first < mid < last, S1 and S2 both decrease by at least two elements (the pivot and one of a[ first ], a[ mid ], and a[ last ] ) with each recursive call to quicksort( ). Also, note that this heuristic will always choose the middle value of a[ first.. last ] if a[ first.. last ] is already arranged in either ascending or descending order. Finally, note that in quicksort( ), most of the work is done by partition( ) BEFORE the recursive calls to quicksort( ). On the other hand, in mergeSort( ), most of the work is done by merge( ) AFTER the recursive calls to mergeSort( ).

35 Quicksort: Efficiency In the worst case, the pivot is always chosen as the smallest or largest element in a[ ]. In these cases, each recursive call to quicksort( ) produces a partition with either S1 or S2 empty. In these cases, the array segment processed by quicksort( ) decreases by only one element with each recursive call. Since n – 1 data-item comparisons are required to partition an array of n elements, the total number of data-item comparisons required by quicksort( ) in the worst case is given by (n – 1) + (n – 2) + (n – 3) = n * (n – 1) / 2 If the preceding, choosePivot( ), is used, the worst-case number of comparisons to partition n elements is given by (n – 1) + (n – 3) + (n – 5 ) = n 2 / 4

36 Quicksort: Efficiency (Cont’d.) In the worst case, choosePivot( ) requires 3 data-item comparisons to determine the middle value of a[ first ], a[ mid ], and a[ last ]. Consequently, choosePivot( ) adds at most 3 * (n – 1) data-item comparisons to the preceding. Note that partition( ) requires no data-item moves to place an element in S2. (We simply increment firstUnknown.) However, a swap of data elements, requiring 3 data-item moves, is required to place an element in S1. In the worst case, each data-item comparison needed to partition a[ ] will be followed by swapping an element into S1. Therefore, in the worst case, quicksort( ) will require O( n 2 ) data-item comparisons and O( n 2 ) data-item moves to sort an array of n elements.

37 Quicksort: Summary In the worst case, Quicksort requires O( n 2 ) time to sort an array of n elements, similar to Insertion sort, Selection sort, and Bubble sort. Further analysis shows that Quicksort’s average running time is O( n * log 2 n ), like Mergesort. However, in practice, Quicksort usually runs faster than Mergesort, and Quicksort does not need the extra space for a temporary array as Mergesort does.

38 K th Smallest Item: Basic Idea 1) As with quicksort( ), choose a pivot from the array a[first.. last] and swap elements, if necessary, to place it in a[first]. 2) Compare each element in a[first last] with the pivot. –If an element is < the pivot, add it to an (initially empty) block of numbers, S1, that are all < the pivot. –If an element is  the pivot, add it to an (initially empty) block of numbers, S2, that are all  the pivot. 3) Arrange a[first.. last] to be in the order S1 pivot S2. –If S1 contains  k items, then the k th smallest is in S1. Invoke kthSmallest( ) recursively on S1. –If S1 contains k – 1 items, then k th smallest = pivot. Return pivot. –Otherwise, k th smallest is in S2. Invoke kthSmallest( ) recursively on S2.

39 K th Smallest Item: Example Problem: Find the 4 th smallest item in the following. Arbitrarily select a[0] as the pivot. S1 and S2 are empty. We now compare each element in a[1.. 5] with the pivot. Since a[1]  pivot, we add it to S2, yielding a[ ]: a[ ]: pivotS2UNKNOWN

40 Continue as with quicksort( ) until we obtain As with quicksort( ), everything to the left of pivot is < pivot and everything to the right of pivot is  pivot. In other words, pivot is now in the position it would be in if a[0.. 5] was completely sorted. Since pivot is in the 3 rd position of a[0.. 5], it is the 3 rd smallest item. Therefore, we know that the 4 th smallest must be in S2. K th Smallest Item: Example (Cont’d.) a[ ]: pivot S1S2

41 K th Smallest Item: Example (Cont’d.) Invoke kthSmallest( ) recursively on S2: Arbitrarily select a[3] as the pivot. Set new S1 and S2 to empty. We now compare each element in a[4.. 5] with the pivot. Since a[4] < pivot, we add it to S1, yielding a[ ]: a[ ]: pivotS1UNKNOWN

42 Continue as with quicksort( ) until we obtain From the previous recursive step we know that a[2] contains the 3 rd smallest item and the 1 st and 2 nd smallest items are to the left of a[2] (in some unknown order). Consequently, S1 pivot S2 must contain the 4 th, 5 th, and 6 th smallest items (in some order). Since all items in S1 < pivot  all items in S2, and since S1 contains only a[3], we immediately conclude that a[3] is the desired, 4th smallest element. K th Smallest Item: Example (Cont’d.) a[ ]: S1pivotS2

43 K th Smallest Item: C++ Implementation DataType kthSmallest( int k, DataType a[ ], int first, int last ) { int pivotIndex; if( first < last ) { // select pivotIndex, and partition a[first.. last] so that each // element in a[first.. pivotIndex – 1] < a[pivotIndex], and // each element in a[pivotIndex last] >= a[pivotIndex] partition( a, first, last, pivotIndex ); int sizeS1 = pivotIndex – first; if( k <= sizeS1 ) return kthSmallest( k, a, first, pivotIndex – 1 ); if( k = = sizeS1 + 1) return a[ pivotIndex ]; return kthSmallest( k – (sizeS1 + 1), a, pivotIndex + 1, last ); }

44 K th Smallest Item: Efficiency Recall that quicksort( ) invokes itself recursively on both S1 and S2. However, kthSmallest( ) invokes itself recursively on only one of S1 or S2 or not at all — if the desired item is the pivot. As with quicksort( ), most of the work in kthSmallest( ) occurs in the partitioning step. Analysis similar to that done for quicksort( ) shows that the average and worst-case running time of kthSmallest( ) is the same as quicksort( ), namely Average:O( n * log 2 n ) Worst-case:O( n 2 ) However, in the best case, the first pivot chosen is the desired item, yielding O( n ) running time in the best case.

45 Radix Sort: Basic Idea 1) Arrange a list of (integer) numbers into groups, depending on the value of the last digit in each number: –numbers ending in “0” are placed in the 1st group; –numbers ending in “1” are placed in the 2nd group; … –numbers ending in “9” are placed in the 10th group. 2) Merge the groups in the order, 1st, 2nd, …, 10th, without changing the relative order of the numbers in any group. 3) Arrange the numbers again into groups, now depending on the value of the next-to-last digit in each number, without changing the relative order of any number. 4) Repeat steps 2 and 3 for each successive digit position to the left of the last digit. 5) Sorting is complete when the numbers have been grouped by the value of the first (leading) digit and then merged back into a single list.

46 Radix Sort: What’s a Radix? Radix is another word for the base of a number system. Specifically, it indicates the number of choices for values that are allowed in any given digit’s place. For example, most of the time we work with a base 10 (or radix 10) number system, where any of the 10 integers, 0 - 9, are allowed to represent a digit. However, there are other possibilities, most common of which are base 2 (binary):digits are 0 and 1 base 8 (octal):digits are base 16 (hexadecimal):digits are and A - F Radix sort may be applied to strings of characters, where each character position might contain, for example, one of 97 printable, ASCII characters, including ‘ ’, \t, \n. (Radix = 97)

47 Radix Sort: Example Problem: Sort the following numbers 123, 2154, 222, 4, 283, 1560, 1061, 2150 Pad to the left with zeroes, so that all numbers are of the same length: 0123, 2154, 0222, 0004, 0283, 1560, 1061, 2150 Group by rightmost digit: (1560, 2150), (1061), (0222), (0123, 0283), (2154, 0004) Combine the groups into one, without changing the relative order of any numbers: 1560, 2150, 1061, 0222, 0123, 0283, 2154, 0004

48 Radix Sort: Example (Cont’d.) 1560, 2150, 1061, 0222, 0123, 0283, 2154, 0004 Group by the next-to-the-last digit, without changing the relative order of any numbers within a group: (0004), (0222, 0123), (2150, 2154), (1560, 1061), (0283) Combine the groups into one, without changing the relative order of any numbers: 0004, 0222, 0123, 2150, 2154, 1560, 1061, 0283

49 Radix Sort: Example (Cont’d.) 0004, 0222, 0123, 2150, 2154, 1560, 1061, 0283 Group by the 2nd-to-last digit, without changing the relative order of any numbers within a group: (0004, 1061), (0123, 2150, 2154), (0222, 0283), (1560) Combine the groups into one, without changing the relative order of any numbers: 0004, 1061, 0123, 2150, 2154, 0222, 0283, 1560

50 Radix Sort: Example (Cont’d.) 0004, 1061, 0123, 2150, 2154, 0222, 0283, 1560 Group by the 3rd-to-last digit, without changing the relative order of any numbers within a group: (0004, 0123, 0222, 0283), (1061, 1560), (2150, 2154) Combine the groups into one, without changing the relative order of any numbers: 0004, 0123, 0222, 0283, 1061, 1560, 2150, 2154 Radix sort is now done, with all numbers in sorted order.

51 Radix Sort: C++ Implementation void radixSort( int a[ ], int n, int ndigits ) { Queue queue[RADIX]; // RADIX = 10 for( int j = ndigits – 1; j >= 0; j– – ) { // store a[0..n-1] in queue[0] … queue[RADIX – 1] for( int i = 0; i < n; i++ ) { int k = digit( a[i], j ); // set k to the jth digit of a[i] queue[k].enqueue( a[i] ); // add a[i] to the end of queue[k] } // replace a[0..n-1] with items from queue[0]…queue[RADIX–1] for( int i = 0, k = 0; k < RADIX; k++ ) while( queue[k].dequeue( a[i] ) ) i++; }

52 Radix Sort: Efficiency Note that Radix sort requires no data-item comparisons! For each digit in a data item, Radix sort enqueues and dequeues n elements of a[ ]. The number of data-item moves, therefore, is ndigits * n * 2 For a given application, there is a constant, C, such that ndigits < C. (For example, for 32-bit integers, C = 10.) Therefore, by this analysis, Radix sort requires O( n ) time to sort n items in the best, average, and worst cases.

53 Radix Sort: Efficiency Caveat Although the preceding analysis is technically correct, a more careful analysis shows that the loop for( int i = 0, k = 0; k < RADIX; k++ ) while( queue[k].dequeue( a[i] ) ) i++; executes dequeue( ) RADIX + n times. The running time of Radix sort is, therefore, proportional to ndigits * ( n + RADIX + n ) which is O( ndigits * ( RADIX + n ) ) Although for a given application, there are constants C 1 and C 2, such that ndigits < C 1 and RADIX < C 2, their impact cannot be ignored if either or both of them is large. (RADIX could be ~100 (  256) for character strings, and ndigits could be 40 or more for RSA Security keys.)

54 Radix Sort: Summary Radix sort requires O( ndigits * ( RADIX + n )) = O( n ) time to sort n items in the best, worst, and average cases. Radix sort requires no data-item comparisons. Radix sort requires that the data items all be of the same length. If the data items are integers, they are padded (to the left) with zeroes. If the data items are character strings, they are padded (to the right) with blanks. Radix sort requires extracting each digit from each data item, or n * ndigits extractions. (On a binary computer, this will be most efficient if the RADIX is a power of 2 — e.g. 2, 8 or 16.) Radix sort requires extra storage and software to manage the queues that are required. (Number of queues = RADIX.) For the three previous reasons, Radix sort can be somewhat awkward to use, in spite of its O( n ) running time.

55 Radix Sort: Summary (Cont’d.) In practice, it is sometimes sufficient to use Radix sort on the leading digits only (to form a set of queues), and then use some other method, such as Quicksort or Insertion sort, to sort the individual queues.

56 Growth Rates for Selected Sorting Algorithms † According to Knuth, the average growth rate of Insertion sort is 0.9 times the average growth rate of Selection sort and 0.4 times the average growth rate of Bubble Sort. ‡ This is is really O( ndigits * ( RADIX + n ) ).