1. Big-O and Sorting February 6, 2006

2. Administrative Stuff Readings for today: Ch 7.3-7.5 Readings for tomorrow: Ch 8

3. Sorting! Very common to need data in order –Viewing, printing –Faster to search, find min/max, compute median/mode, etc. Lots of different sorting algoritms –From the simple to very complex –Some optimized for certain situations (lots of duplicates, almost sorted, etc.) –Typically sort arrays, but algorithms usually can be adapted for other data structures (e.g. linked lists)

4. Selection sort Sort by "selecting" smallest and putting in front –Search entire array for minimum value –Min is placed in first slot –Could move elements over to make space, but faster to just swap with current first –Repeat for second smallest, third, and so on

5. Selection sort code void SelectionSort(int arr[], int n) { for (int i = 0; i < n-1; i++) { int minIndex = i; for (int j = i+1; j < n; j++) { if (arr[j] < arr[minIndex]) minIndex = j; } Swap(arr[i], arr[minIndex]); }

6. Analyzing selection sort for (int i = 0; i < n-1; i++) { int minIndex = i; for (int j = i+1; j < n; j++) { if (arr[j] < arr[minIndex]) minIndex = j; } Swap(arr[i], arr[minIndex]); } Count statements –First time inner loop N-1 comparisons –N-2 second time, then N-3, … –Last iteration 1 comparison

7. Analyzing selection sort N-1 + N-2 + N-3 + … + 3 + 2 + 1 –"Gaussian sum" Add sum to self Sum =

8. Analyzing selection sort N-1 + N-2 + N-3 + … + 3 + 2 + 1 –"Gaussian sum" Add sum to self N-1 + N-2 + N-3 + … + 3 + 2 + 1 + 1 + 2 + 3 + …. + N-2 + N-1 = N + N + N + …. + N + N = (N-1)N Sum = 1/2 * (N-1)N O(N 2 )

9. Quadratic growth In clock time – 10,000 3 sec – 20,000 13 sec – 50,000 77 sec –100,000 5 min Double input -> 4X time –Feasible for small inputs, quickly unmanagable Halve input -> 1/4 time –Hmm… –If two sorted half-size arrays, how to produce sorted full array?

10. Mergesort "Divide and conquer" algorithm –Divide array in half –Recursively sort each half –Merge two halves together "Easy-split hard-join" –No complex decision about which goes where, just divide in middle –Merge step preserves ordering from each half

11. 6 2 8 4 10 7 1 5 9 3

12. void MergeSort(int array[], int n) { if (n > 1) { int n1 = n/2; int n2 = n - n1; int *arr1 = CopySubArray(array, 0, n1); int *arr2 = CopySubArray(array, n1, n2); MergeSort(arr1, n1); MergeSort(arr2, n2); Merge(array, arr1, n1, arr2, n2); delete[] arr1; delete[] arr2; }

13. CopySubArray // Create a new array in memory void CopyArray(int arr[], int n, int * & copy) { copy = new int[n]; for(int i = 0; i < n; i++) { copy[i] = arr[i]; }

14. Merge code void Merge(int array[], int arr1[], int n1, int arr2[], int n2){ int p = 0, p1 = 0, p2 = 0; while (p1 < n1 && p2 < n2) { if (arr1[p1] < arr2[p2]) array[p++] = arr1[p1++]; else array[p++] = arr2[p2++]; } while (p1 < n1) array[p++] = arr1[p1++]; while (p2 < n2) array[p++] = arr2[p2++]; }

15. void Merge(int array[], int arr1[], int n1, int arr2[], int n2) { int p, p1, p2; p = p1 = p2 = 0; while (p1 < n1 && p2 < n2) { // Merge until hit if (arr1[p1] < arr2[p2]) { // end of one array array[p++] = arr1[p1++]; } else { array[p++] = arr2[p2++]; } while (p1 < n1) { // Merge rest of array[p++] = arr1[p1++]; // remaining array } while (p2 < n2) { array[p++] = arr2[p2++]; } 4 array arr1 arr2 n1 n2 p1 p2 p 78124 5916184

16. Merge sort analysis void MergeSort(int array[], int n) { if (n > 1) { int n1 = n/2; int n2 = n - n1; int *arr1 = CopySubArray(array, 0, n1); int *arr2 = CopySubArray(array, n1, n2); MergeSort(arr1, n1); MergeSort(arr2, n2); Merge(array, arr1, n1, arr2, n2); delete[] arr1; delete[] arr2; }

17. MS(N) Merge sort analysis = N = N/2 + N/2 MS(N/2) + N/4 N/4 = 4*N/4 + N/8 N/8 N/8 N/8 + = 8*N/8 Each level contributes N...

18. MS(N) Merge sort analysis MS(N/2) N/4 N/4 N/8 N/8 N/8 N/8 N/2 K = 1 N = 2 K lg N = K lg N levels * N per level= O(NlgN) K levels … N/2 K

19. In clock time Compare SelectionSort to MergeSort – 10,000 3 sec.05 sec – 20,000 13 sec.15 sec – 50,000 78 sec.38 sec – 100,000 5 min.81 sec – 200,000 20 min 1.7 sec –1,000,000 8 hrs (est) 9 sec O(NlgN) is looking pretty good! But can we do even better?

20. Can we do even better than MergeSort? O(N log N) is fastest sort in the general case –So, theoretically, answer is “no” But, we can come up with a different O(N log N) sort that is practically faster Want to avoid overhead of creating new arrays (as is done in MergeSort) –Bring on the QuickSort!

21. Quicksort 53748621

22. Recursive Insight 53748621

23. 53748621 select “pivot”

24. Partition array so: everything smaller than pivot is on left everything greater than or equal to pivot is on right pivot is in-between Recursive Insight 53748621

25. Partition array so: everything smaller than pivot is on left everything greater than or equal to pivot is on right pivot is in-between Recursive Insight 23145687

26. 23145687 Now recursive sort “red” sub-array

27. Recursive Insight 12345687 Now recursive sort “red” sub-array

28. Recursive Insight 12345687 Now recursive sort “red” sub-array Then, recursive sort “blue” sub-array

29. Recursive Insight 12345678 Now recursive sort “red” sub-array Then, recursive sort “blue” sub-array

30. Recursive Insight 12345678 Everything is sorted!

31. void Quicksort(int arr[], int n) { if (n < 2) return; int boundary = Partition(arr, n); // Sort subarray up to pivot Quicksort(arr, boundary); // Sort subarray after pivot to end Quicksort(arr + boundary + 1, n – boundary - 1); } “boundary” is the index of the pivot This is equal to the number of elements before pivot

32. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; }

33. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53748621

34. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53748621 pivot lhrh

35. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53748621 pivot lhrh

36. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53748621 pivot lhrh

37. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53748621 pivot lhrh

38. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53748621 pivot lhrh

39. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53748621 pivot lhrh

40. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

41. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

42. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

43. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

44. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

45. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

46. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

47. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

48. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53148627 pivot lhrh

49. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

50. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

51. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

52. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

53. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

54. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

55. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

56. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

57. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

58. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 53142687 pivot lhrh

59. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 23145687 pivot lhrh

60. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 23145687 pivot lhrh

61. int Partition(int arr[], int n) { int lh = 1, rh = n - 1; int pivot = arr[0]; while (true) { while (lh = pivot) rh--; while (lh < rh && arr[lh] < pivot) lh++; if (lh == rh) break; Swap(arr[lh], arr[rh]); } if (arr[lh] >= pivot) return 0; Swap(arr[0], arr[lh]); return lh; } 23145687 pivot lhrh Returns 4 (index of pivot)

62. void Quicksort(int arr[], int n) { if (n < 2) return; int boundary = Partition(arr, n); // Sort subarray up to pivot Quicksort(arr, boundary); // Sort subarray after pivot to end Quicksort(arr + boundary + 1, n – boundary - 1); }

63. void Quicksort(int arr[], int n) { if (n < 2) return; int boundary = Partition(arr, n); // Sort subarray up to pivot Quicksort(arr, boundary); // Sort subarray after pivot to end Quicksort(arr + boundary + 1, n – boundary - 1); } O(1) O(n) T(n/2) T(n)= O(1) + O(n) + 2T(n/2) = O(n) + 2T(n/2) Same as MergeSort  O(n log n)

64. The whole recursion 53748621

65. First partition 23145687

66. 23145687

67. Recursive sort {2, 3, 1, 4} 23145687

68. Partition {2, 3, 1, 4} 12345687

69. Recursive sort {1} 12345687

70. 12345687 base case

71. Recursive sort {3, 4} 12345687

72. Partition {3, 4} 12345687

73. Recursive sort {4} 12345687

74. 12345687 base case

75. Recursive sort {6, 8, 7} 12345687

76. Leap of faith! 12345678

77. Empirical comparison of MergeSort vs QuickSort NMerge sort Quicksort 100.54 msec0.10 msec 201.17 msec0.26 msec 402.54 msec0.52 msec 1006.90 msec1.76 msec 20014.84 msec4.04 msec 40031.25 msec8.85 msec 100084.38 msec26.04 msec 2000179.17 msec56.25 msec 4000383.33 msec129.17 msec 10,000997.67 msec341.67 msec