1. Chapter 10: Sorting1 Sorting

2. Chapter 10: Sorting2 Chapter Outline How to use standard sorting functions in How to implement these sorting algorithms: Selection sort Bubble sort Insertion sort Shell sort Merge sort Heapsort Quicksort

3. Chapter 10: Sorting3 Chapter Outline (2) Understand the performance of these algorithms Which to use for small arrays Which to use for medium arrays Which to use for large arrays

4. Chapter 10: Sorting4 C++ Standard Library Sort Functions The C++ standard library (in ) provides the following functions: template void sort(RI first, RI last); template void sort(RI first, RI last, Compare comp); The template parameter RI is a random-access iterator. The data to be sorted is in the range first.. last, where last is one past the last data value. In practice you would tend to use these In this class, you will implement some yourself

5. Chapter 10: Sorting5 Using the sort functions Assume that array items contains 16 integer values. sort(items, items + 16); will sort the whole array. sort(items, items + 8); will sort the first half. sort(items, items + 16, greater ()); will sort in descending order. Assume that v is a vector and d is a deque. sort(v.begin(), v.end()); // sorts vector v sort(d.begin(), d.end()); // sorts deque d

6. Chapter 10: Sorting6 Using the sort functions Define comparison for Person class: struct Compare_Person { bool operator()(const Person& p1, const Person& p2) { return (p1.family_name < p2.family_name) || ((p1.family_name == p2.family_name) && (p1.given_name < p2.given_name)); }; To sort the vector people_list sort(people_list.begin(), people_list.end(), Compare_Person());

7. Chapter 10: Sorting7 Conventions of Presentation Write algorithms for a range defined by random- access iterators. For convenience, examples show integers

8. Chapter 10: Sorting8 Selection Sort A relatively easy to understand algorithm Sorts an array in passes Each pass selects the next smallest element At the end of the pass, places it where it belongs Efficiency is O(n 2 ), hence called a quadratic sort Performs: O(n 2 ) comparisons O(n) exchanges (swaps)

9. Chapter 10: Sorting9 Selection Sort Algorithm 1.for fill = 0 to n-2 do // steps 2-6 form a pass 2. set pos_min to fill 3. for next = fill+1 to n-1 do 4. if item at next < item at pos_min 5. set pos_min to next 6. Exchange item at pos_min with one at fill

10. Chapter 10: Sorting10 Selection Sort Example 3565306020 scan 0-4, smallest 20 swap 35 and 20 2065306035scan 1-4, smallest 30 swap 65 and 30 2030656035scan 2-4, smallest 35 swap 65 and 35 2030356065scan 3-4, smallest 60 swap 60 and 60 2030356065done

11. Chapter 10: Sorting11 Selection Sort Code template void selection_sort(RI first, RI last) { for (RI fill = first; fill != last - 1; ++fill) { // Invariant: table[first] through table[fill - 1] is sorted RI pos_min = fill; for (RI next = fill + 1; next != last; ++next) { // Invariant: pos_min references the smallest item in // table[fill] through table[last - 1] if (*next < *pos_min) { pos_min = next; } // Assert pos_min references the minimum value in table[fill] // through table[last - 1]. // If the next smallest item is not a fill, // exchange *fill and *pos_min. if (fill != pos_min) std::iter_swap(pos_min, fill); }

12. Chapter 10: Sorting12 Selection Sort using Compare template void selection_sort(RI first, RI last, Compare comp) { for (RI fill = first; fill != last - 1; ++fill) { // Invariant table[first] through table[fill - 1] is sorted RI pos_min = fill; for (RI next = fill + 1; next != last; ++next) { // Invariant: pos_min references the smallest item in // table[fill] through table[last - 1]. if (comp(*next, *pos_min)) { pos_min = next; } // Assert pos_min references the minimum value in table[fill] // through table[last - 1]. // If the next smallest item is not a fill, // exchange *fill and *pos_min. if (fill != pos_min) std::iter_swap(pos_min, fill); }

13. Chapter 10: Sorting13 Bubble Sort Compares adjacent array elements Exchanges their values if they are out of order Smaller values bubble up to the top of the array Larger values sink to the bottom

14. Chapter 10: Sorting14 Bubble Sort Example

15. Chapter 10: Sorting15 Bubble Sort Algorithm 1.do 2. for each pair of adjacent array elements 3. if values are out of order 4. Exchange the values 5.while the array is not sorted

16. Chapter 10: Sorting16 Bubble Sort Algorithm, Refined 1.do 2. Initialize exchanges to false 3. for each pair of adjacent array elements 4. if values are out of order 5. Exchange the values 6. Set exchanges to true 7.while exchanges

17. Chapter 10: Sorting17 Analysis of Bubble Sort Excellent performance in some cases But very poor performance in others! Works best when array is nearly sorted to begin with Worst case number of comparisons: O(n 2 ) Worst case number of exchanges: O(n 2 ) Best case occurs when the array is already sorted: O(n) comparisons O(1) exchanges (none actually)

18. Chapter 10: Sorting18 Bubble Sort Code template void bubble_sort(RI first, RI last) { int pass = 1; bool exchanges; do { // Invariant: Elements after position last - pass // are in place. exchanges = false; // No exchanges yet. // Compare each pair of adjacent elements. for (RI first_of_pair = first; first_of_pair != last - pass; ++first_of_pair) { RI second_of_pair = first_of_pair + 1; if (*second_of_pair < *first_of_pair) { // Exchange pair. std::iter_swap(first_of_pair, second_of_pair); exchanges = true; // Set flag. } pass++; } while (exchanges); // Assert: Array is sorted. }

19. Chapter 10: Sorting19 Insertion Sort Based on technique of card players to arrange a hand Player keeps cards picked up so far in sorted order When the player picks up a new card Makes room for the new card Then inserts it in its proper place

20. Chapter 10: Sorting20 Insertion Sort Algorithm For each element from 2nd (next_pos = 1) to last: Insert element at next_pos where it belongs Increases sorted subarray size by 1 To make room: Hold next_pos value in a variable Shuffle elements to the right until gap at right place

21. Chapter 10: Sorting21 Insertion Sort Example

22. Chapter 10: Sorting22 Insertion Sort Code template void insertion_sort(RI first, RI last) { for (RI next_pos = first + 1; next_pos != last; ++next_pos) { // Invariant: elements at position first through next_pos - 1 are sorted. // Insert element at position next_pos in the sorted subarray. insert(first, next_pos); } template void insert(RI first, RI next_pos) { while (next_pos != first && *next_pos < *(next_pos - 1)) { std::iter_swap(next_pos, next_pos - 1); // Exchange pair of values --next_pos; // Check next smaller element }

23. Chapter 10: Sorting23 More Efficient Version template void insertion_sort2(RandomAccessIterator first, RandomAccessIterator last) { for (RandomAccessIterator next_pos = first+1; next_pos != last; ++next_pos) { // Invariant: elements at position first thru next_pos - 1 are sorted. // Insert element at position next_pos // in the sorted subarray. insert2(first, next_pos); } template void insert2(RI first, RI next_pos) { typename std::iterator_traits ::value_type next_val = *next_pos; // next_val is element to insert. while (next_pos != first && next_val < *(next_pos - 1)) { *next_pos = *(next_pos - 1); --next_pos; // Check next smaller element. } *next_pos = next_val; // Store next_val where it belongs. }

24. Chapter 10: Sorting24 Analysis of Insertion Sort Maximum number of comparisons: O(n 2 ) In the best case, number of comparisons: O(n) # shifts for an insertion = # comparisons - 1 When new value smallest so far, # comparisons A shift in insertion sort moves only one item Bubble or selection sort exchange: 3 assignments

25. Chapter 10: Sorting25 Comparison of Quadratic Sorts None good for large arrays!

26. Chapter 10: Sorting26 Shell Sort: A Better Insertion Sort Shell sort is a variant of insertion sort It is named after Donald Shell Average performance: O(n 3/2 ) or better Divide and conquer approach to insertion sort Sort many smaller subarrays using insertion sort Sort progressively larger arrays Finally sort the entire array These arrays are elements separated by a gap Start with large gap Decrease the gap on each “pass”

27. Chapter 10: Sorting27 Illustration of Shell Sort

28. Chapter 10: Sorting28 Analysis of Shell Sort Intuition: Reduces work by moving elements farther earlier Its general analysis is an open research problem Performance depends on sequence of gap values For sequence 2 k, performance is O(n 2 ) Hibbard’s sequence (2 k -1), performance is O(n 3/2 ) We start with n/2 and repeatedly divide by 2.2 Empirical results show this is O(n 5/4 ) or O(n 7/6 ) No theoretical basis (proof) that this holds

29. Chapter 10: Sorting29 Shell Sort Algorithm 1.Set gap to n/2 2.while gap > 0 3. for each element from gap to end, by gap 4. Insert element in its gap-separated sub-array 5. if gap is 2, set it to 1 6. otherwise set it to gap / 2.2

30. Chapter 10: Sorting30 Shell Sort Algorithm: Inner Loop 3.1set next_pos to position of element to insert 3.2set next_val to value of that element 3.3while next_pos > gap and element at next_pos-gap is > next_val 3.4Shift element at next_pos-gap to next_pos 3.5Decrement next_pos by gap 3.6Insert next_val at next_pos

31. Chapter 10: Sorting31 Shell Sort Code template void shell_sort(RI first, RI last) { // Set initial gap between adjacent elements. int gap = (last - first) / 2; while (gap > 0) { for (RI next_pos = first + gap; next_pos != last; ++next_pos) { // Insert element at next_pos in its subarray. shell_insert(first, next_pos, gap); } // End for. // Reset gap for next pass. if (gap == 2) { gap = 1; } else { gap = int(gap / 2.2); }

32. Chapter 10: Sorting32 Shell Sort Code (2) template void shell_insert(RI first, RI next_pos, int gap) { typename std::iterator_traits ::value_type next_val = *next_pos; // Shift all values > next_val in subarray down by gap. while ((next_pos > first + gap - 1) // First element not shifted. && (next_val < *(next_pos - gap))) { *next_pos = *(next_pos - gap); // Shift down. next_pos -= gap; // Check next position in subarray. } *next_pos = next_val; }

33. Chapter 10: Sorting33 Merge Sort A merge is a common data processing operation: Performed on two sequences of data Items in both sequences use same compare Both sequences in ordered of this compare Goal: Combine the two sorted sequences in one larger sorted sequence Merge sort merges longer and longer sequences

34. Chapter 10: Sorting34 Merge Algorithm (Two Sequences) Merging two sequences: 1.Access the first item from both sequences 2.While neither sequence is finished 1.Compare the current items of both 2.Copy smaller current item to the output 3.Access next item from that input sequence 3.Copy any remaining from first sequence to output 4.Copy any remaining from second to output

35. Chapter 10: Sorting35 Picture of Merge

36. Chapter 10: Sorting36 Analysis of Merge Two input sequences, total length n elements Must move each element to the output Merge time is O(n) Must store both input and output sequences An array cannot be merged in place Additional space needed: O(n)

37. Chapter 10: Sorting37 Merge Sort Algorithm Overview: Split array into two halves Sort the left half (recursively) Sort the right half (recursively) Merge the two sorted halves

38. Chapter 10: Sorting38 Merge Sort Algorithm (2) Detailed algorithm: if tSize  1, return (no sorting required) set hSize to tSize / 2 Allocate LTab of size hSize Allocate RTab of size tSize – hSize Copy elements 0.. hSize – 1 to LTab Copy elements hSize.. tSize – 1 to RTab Sort LTab recursively Sort RTab recursively Merge LTab and RTab into a

39. Chapter 10: Sorting39 Merge Sort Example

40. Chapter 10: Sorting40 Merge Sort Analysis Splitting/copying n elements to subarrays: O(n) Merging back into original array: O(n) Recursive calls: 2, each of size n/2 Their total non-recursive work: O(n) Next level: 4 calls, each of size n/4 Non-recursive work again O(n) Size sequence: n, n/2, n/4,..., 1 Number of levels = log n Total work: O(n log n)

41. Chapter 10: Sorting41 Merge Sort Code template void merge_sort(RI first, RI last) { if (last - first > 1) { // Split table into two new half tables. typedef typename std::iterator_traits ::value_type value_type; RI middle = first + (last - first) / 2; std::vector left_table(first, middle); std::vector right_table(middle, last); // Sort the halves. merge_sort(left_table.begin(), left_table.end()); merge_sort(right_table.begin(), right_table.end()); // Merge the halves back into the original table. merge(left_table.begin(), left_table.end(), right_table.begin(), right_table.end(), first); }

42. Chapter 10: Sorting42 Merge Sort Code (2) template void merge(RI left, RI end_left, RI right, RI end_right, RI out) { // While there is data in both input sequences while (left != end_left && right != end_right) { // Find the smaller and // insert it into the output sequence. if (*left < *right) { *out++ = *left++; } else { *out++ = *right++; }

43. Chapter 10: Sorting43 Heapsort Merge sort time is O(n log n) But requires (temporarily) n extra storage items Heapsort Works in place: no additional storage Offers same O(n log n) performance Idea (not quite in-place): Insert each element into a priority queue Repeatedly remove from priority queue to array Array slots go from 0 to n-1

44. Chapter 10: Sorting44 Heapsort Picture

45. Chapter 10: Sorting45 Heapsort Picture (2)

46. Chapter 10: Sorting46 Algorithm for In-Place Heapsort Build heap starting from unsorted array While the heap is not empty Remove the first item from the heap: Swap it with the last item Restore the heap property

47. Chapter 10: Sorting47 Heapsort Code template void heap_sort(RI first, RI last) { build_heap(first, last); shrink_heap(first, last); } template void build_heap(RI first, RI last) { int n = 1; // Invariant: table[first] through table[first + n - 1] is a heap. while (n < (last - first)) { ++n; // Add a new item to the heap and reheap. RI child = first + n - 1; RI parent = first + (child - first - 1) / 2; // Find parent. while (parent >= first && *parent < *child) { std::iter_swap(parent, child); // Exchange elements. child = parent; parent = first + (child - 1 - first) / 2; }

48. Chapter 10: Sorting48 Heapsort Code (2) template void shrink_heap(RI first, RI last) { RI n = last; while (n != first) { --n; std::iter_swap(first, n); RI parent = first; while (true) { RI left_child = first + 2 * (parent - first) + 1; if (left_child >= n) break; // No more children. RI right_child = left_child + 1; // Find the larger of the two children. RI max_child = left_child; if (right_child < n // There is a right child. && *left_child < *right_child) max_child = right_child; // If the parent is smaller than the larger child, if (*parent < *max_child) { std::iter_swap(parent, max_child); parent = max_child; } else break; // Heap property is restored. Exit the loop. }

49. Chapter 10: Sorting49 Heapsort Analysis Insertion cost is log i for heap of size i Total insertion cost = log(n)+log(n-1)+...+log(1) This is O(n log n) Removal cost is also log i for heap of size i Total removal cost = O(n log n) Total cost is O(n log n)

50. Chapter 10: Sorting50 Quicksort Developed in 1962 by C. A. R. Hoare Given a pivot value: Rearranges array into two parts: Left part  pivot value Right part > pivot value Average case for Quicksort is O(n log n) Worst case is O(n 2 )

51. Chapter 10: Sorting51 Quicksort Example

52. Chapter 10: Sorting52 Algorithm for Quicksort first and last are end points of region to sort if first < last Partition using pivot, which ends in piv_index Apply Quicksort recursively to left subarray Apply Quicksort recursively to right subarray Performance: O(n log n) provide piv_index not always too close to the end Performance O(n 2 ) when piv_index always near end

53. Chapter 10: Sorting53 Quicksort Code template void quick_sort(RI first, RI last) { if (last - first > 1) { // There is data to be sorted. // Partition the table. RI pivot = partition(first, last); // Sort the left half. quick_sort(first, pivot); // Sort the right half. quick_sort(pivot + 1, last); }

54. Chapter 10: Sorting54 Algorithm for Partitioning 1.Set pivot value to a[fst] 2.Set up to fst and down to lst 3.do 4. Increment up until a[up] > pivot or up = lst 5. Decrement down until a[down] <= pivot or down = fst 6. if up < down, swap a[up] and a[down] 7.while up is to the left of down 8.swap a[fst] and a[down] 9.return down as pivIndex

55. Chapter 10: Sorting55 Trace of Algorithm for Partitioning

56. Chapter 10: Sorting56 Partitioning Code template RI partition(RI first, RI last) { // Start up and down at either end of the sequence. // The first table element is the pivot value. RI up = first + 1; RI down = last - 1; do { while ((up != last - 1) && !(*first < *up)) { ++up; } while (*first < *down) { --down; } if (up < down) { // if up is to the left of down, std::iter_swap(up, down); } } while (up < down); // Repeat while up is left of down. std::iter_swap(first, down); return down; }

57. Chapter 10: Sorting57 Revised Partitioning Algorithm Quicksort is O(n 2 ) when each split gives 1 empty array This happens when the array is already sorted Solution approach: pick better pivot values Use three “marker” elements: first, middle, last Let pivot be one whose value is between the others

58. Chapter 10: Sorting58 Testing Sortiing Algorithms Need to use a variety of test cases Small and large arrays Arrays in random order Arrays that are already sorted (and reverse order) Arrays with duplicate values Compare performance on each type of array

59. Chapter 10: Sorting59 Variety of partitioning algorithms have been published One that partitions an array into three segments was introduced by Edsger W. Dijkstra Problem: partition a disordered three-color flag into three contiguous segments Segments represent the pivot value The Dutch National Flag Problem Dutch National FlagScrambled Dutch National Flag

60. Chapter 10: Sorting60 The Dutch National Flag Problem

61. Chapter 10: Sorting61 Summary of Performance