CS 615: Design & Analysis of Algorithms Chapter 4: Sorting Mark Allen Weiss: Chapter 7 Page 219-261

CS 615 Design & Analysis of Algorithms Course Content Introduction, Algorithmic Notation and Flowcharts (Brassard & Bratley Chp: Chapter 3) Efficiency of Algorithms (Brassard & Bratley Chp: Chapter 2) Basic Data Structures (Brassard & Bratley Chp: Chapter 5) Sorting (Weiss Chp: 7) Searching (Brassard & Bratley Chp: Chapter 9) Graph Algorithms (Weiss Chp: 9) Randomized Algorithms (Weiss Chp: 10) String Searching (Sedgewick Chap. 19) NP Completeness (Sedgewick Chap. 40) 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Lecture Content Sorting by Selection Sorting by Insertion Insertion Sort Shellsort Heapsort Mergesort Quicksort 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Sorting by Selection Algorithm: Select the smallest number within the array Bring it to the front of the array Get the next smallest element Insert it into the next location in the array Until all elements are sorted When this algorithm is programmed Required time to sort the array Vary no more than %15 Whatever the initial order of the elements to be sorted select(T) is quadratic procedure select(T[1..n]) for i=1 to n-1 do minj=i minx=T[i] for j=i+1 to n do if T[j]<minx then minj=j minx=T[j] T[minj]=T[i] T[i]=minx 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Sorting by Insertion Algorithm: Set the first element as the minimum For all other elements Check the element if less than current minimum If so compare the item with previous elements in array Exchange the elements until the correct place is found Apply the same operation to all other elements of the array When this algorithm is programmed Variation between best and worst case is very big Required time depends on the number of elements in the array insert(T) is quadratic procedure insert(T[1..n]) for i=2 to n do x=T[i]; j=i-1 while j>0 and x<T[j] do T[j+1]=T[j] j=j-1 T[j+1]=x 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Insertion Sort One of the simplest algorithms Consist of N-1 passes In pass P The element in position P is saved in Tmp All larger elements prior to position P Are moved one spot to the right Then Tmp is placed in correct spot InsertionSort Original 34 8 64 51 32 21 Position Moved After p=1 1 After p=2 After p=3 After p=4 3 After p=5 4 15 May 2019 CS 615 Design & Analysis of Algorithms

Pseudocode/Insertion Sort void InsertionSort( ElementType A[ ], int N ) { int j, P; ElementType Tmp; for( P = 1; P < N; P++ ) Tmp = A[ P ]; for( j = P; j > 0 && A[ j - 1 ] > Tmp; j-- ) A[ j ] = A[ j - 1 ]; A[ j ] = Tmp; } 15 May 2019 CS 615 Design & Analysis of Algorithms

Analysis of Insertion Sort Because of nested loops each step will take N iterations Order of algorithm is O(n2) If the data is in reverse order: If the data is pre-sorted Order of algorithm is O(n) The inner loop always fail immediately 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Shell Sort Invented by Donald Shell Was the first algorithm to break the quadratic time barrier Uses a sequence, h1,h2,...,ht, called “increment sequence” At each sequence: Compare h-distant element Exchange them if the first element is greater than the second one ShellSort Original 81 94 11 96 12 35 17 95 28 58 41 75 15 After 5-sort After 3-sort After 2-sort After 1-sort 15 May 2019 CS 615 Design & Analysis of Algorithms

Pseudocode/Shell Sort /* START: fig7_4.txt */ void Shellsort( ElementType A[ ], int N ) { int i, j, Increment; ElementType Tmp; for( Increment = N / 2; Increment > 0; Increment /= 2 ) for( i = Increment; i < N; i++ ) Tmp = A[ i ]; for( j = i; j >= Increment; j -= Increment ) if( Tmp < A[ j - Increment ] ) A[ j ] = A[ j - Increment ]; else break; A[ j ] = Tmp; } 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Analysis of Shellsort The worst case is O(n2) Hibbard’s increment The worst case is O(n3/2) The average case is O(n5/4) Not proven ! The performance is quite acceptable Widely used in practice A good choice for sorting up large input 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Heapsort 10 Executes in order of O(NlogN) The best of sorting techniques Only Shellsort uses Sadgewick’s increment sequence is better Heap order property (max)Heap: The key values of children are less than the parent The maximum can always be found at the root (min)Heap: The key values of children are greater than the parent The minimum can always be found at the root 7 .... 4 5 4 5 .... 7 10 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Basic Heap Operations 13 Insert Create a hole in the first available location If the new element X does not violate the heap property it is done otherwise slide the hole’s parent to the new hole If the new element can be inserted in the parent’s node insert it and job is finished Else execute the same operations until the correct place is found insert 21 16 24 31 19 68 32 h 65 26 insert 14 13 21 16 24 31 19 68 65 26 32 14 1 2 3 4 5 6 7 8 9 10 11 15 May 2019 CS 615 Design & Analysis of Algorithms

Basic Heap Operations: DeleteMin 13 Create a hole in the root Percolate down: Place the smallest child as the parent Place child’s smallest child in the hole of the child Continue the operation until no placement remains 14 16 19 21 19 68 32 31 65 26 13 14 16 19 21 19 68 65 26 32 31 DeleteMin 1 2 3 4 5 6 7 8 9 10 11 15 May 2019 CS 615 Design & Analysis of Algorithms

After Deletion, The New Heap 14 19 16 14 19 16 26 21 19 68 65 31 32 26 21 19 68 1 2 3 4 5 6 7 8 9 10 65 31 32 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Heap Sorting Build a heap of N elements N succesive insert operations order is Average: O(N) Worst Case: O(NlogN) Perform N DeleteMin operations Each delete will take order of O(logN) Record deleted elments in another array or use the shrinked heap array put the element in emptied location The elements will be sorted in decreasing order Use max(heap) to sort the elements in increasing order The total running time is O(N) + O(NlogN) = O(NlogN) Start Build Heap DeleteMin End 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Example: HeapSort (Max) heap 97 DeleteMax 59 53 59 53 58 26 41 58 31 26 41 31 97 97 53 59 26 41 58 31 59 53 58 26 41 31 97 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Example Continues 58 DeleteMax 53 53 31 41 31 26 41 59 97 26 58 59 97 58 53 31 26 41 59 97 53 41 31 26 58 59 97 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Example Continues 31 41 DeleteMax 26 41 26 31 53 58 59 97 53 58 59 97 31 26 41 53 58 59 97 41 26 31 53 58 59 97 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 15 May 2019 CS 615 Design & Analysis of Algorithms

Example: After HeapSort 26 31 41 53 58 59 97 26 31 41 53 58 59 97 1 2 3 4 5 6 7 8 9 10 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Merge Sort Aptr 1 13 24 26 Runs in O(NlogN) worst-case The fundamental operation: merging two sorted lists Takes two already sorted input arrays: A, B One output array: C Three counters Aptr,Bptr, and Cptr Algorithm Take the smaller of A[Aptr] and B[Bptr] Put it into C[Cptr] Advance the counters When any of input lists exhausted Copy the next array to C This step can be done in one pass O(N) 2 15 27 38 Bptr Cptr Aptr Merge 1 13 24 26 2 15 27 38 Bptr 1 Cptr 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Merge Sort Example Aptr Aptr Aptr 1 13 24 26 1 13 24 26 1 13 24 26 2 15 27 38 2 15 27 38 2 15 27 38 Bptr Bptr Bptr 1 2 1 2 13 1 2 13 15 Cptr Cptr Cptr Aptr Aptr 1 13 24 26 1 13 24 26 2 15 27 38 2 15 27 38 Bptr Bptr 1 2 13 15 24 1 2 13 15 24 26 27 38 Cptr Cptr 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Divide and Conquer Divide the unordered list into two halves Sort the first half Sort the second half Merge two halves Execute the same operations recursively until all of the array is sorted At each step A temporary C array is needed At any point there could be NlogN temporary arrays in memory If memory is low, not a good algorithm Allocate dynamic memory, and use input arrays for temporary arrays MergeSort 15 May 2019 CS 615 Design & Analysis of Algorithms

Example: Full Merge Sort First Half First Half First Half Aptr Aptr Aptr 24 13 26 1 2 27 38 15 24 13 26 1 2 27 38 15 Bptr Second Half Bptr Second Half Bptr Second Half First Half First Half 1 2 13 15 26 Aptr Aptr Cptr 13 24 1 26 2 27 15 38 Bptr Second Half Bptr Second Half First Half Aptr 1 13 24 26 2 15 27 38 1 2 13 15 24 26 27 38 Bptr Second Half 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Analysis of Mergesort The order of total algorithm is O(NlogN) Is hardly used in main memory sorts Needs extra memory to sort two lists Slows down the algorithm due to copying done between the lists Used mostly for external sorting Using disk spaces to sort very large lists Recursion adds extra time Recursive call is sometimes more than 100 times slower than the iteration 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Quicksort Like MergeSort QuickSort is also a divide and conquer and recursive algorithm Average running time O(NLogN) Worst-case O(N2) Good algorithm in theory But in practice not easy to code correctly Algorithm: (For array S) If S has only 0 or 1 elements return Pick any element v in S. This is called the pivot Partition into S1=S-{v} and S2=S-{v} two subsets Quicksort S1 and S2. QuickSort 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms QuickSort Example -5 21 97 61 -31 24 49 18 52 -5 21 -31 18 24 97 61 49 52 Pivot Pivot Pivot -5 -31 21 18 49 52 97 61 18 21 -31 -5 61 97 49 52 61 97 -31 -5 18 21 -31 -5 18 21 24 49 52 61 97 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Selecting the Pivot Do not select the first or the second element as pivot If the array sorted in reverse order Pivot provides a poor partition Alternatives Choose the pivot randomly Random number generation is expensive Not all systems provide real random numbers Choose the median if the array is reverse ordered Eliminates the bad case for Quicksort case Hard to calculate Some other methods are also available Median3 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms Analysis of Quicksort Pivot selection takes a constant time Worst-Case O(N2) Best Case O(NlogN) Average Case 15 May 2019 CS 615 Design & Analysis of Algorithms

CS 615 Design & Analysis of Algorithms End of Chapter 4 Sorting 15 May 2019 CS 615 Design & Analysis of Algorithms