1. Sorting Gordon College 13.1 Some O(n2) Sorting Schemes
13.2 Heaps, Heapsort, and Priority Queues
13.3 Quicksort
13.4 Mergesort
13.5 Radix Sort

2. Sorting Consider a list x1, x2, x3, … xn
We seek to arrange the elements of the list in order
Ascending or descending
Some O(n2) schemes
easy to understand and implement
inefficient for large data sets

3. Categories of Sorting Algorithms
1. Selection sort
Make passes through a list
On each pass reposition correctly
some element
Look for smallest and position that element correctly...
Look for smallest in list and replace 1st element, now start the process over with the remainder of the list
Look for smallest in list and replace 1st element, now start the process over with the remainder of the list

4. Selection Recursive Algorithm If the list has only 1 element ANCHOR
stop – list is sorted
Else do the following:
a. Find the smallest element and place in front
b. Sort the rest of the list
complexity?
First pass – compare n – 1 elements to find smallest
Second pass - compare n – 2 to find smallest
Total comparisons: n-1 + n-2 +… = n(n-1)/2 = (n^2 – n)/2 = n^2
(see example called recursive_selection.cpp)

5. Categories of Sorting Algorithms
2. Exchange sort
Systematically interchange pairs of elements which are out of order
Bubble sort does this
When your reach the end of section without exchanging then the list is in order
you can shorten the section based on the last set to not be exchanged.
Out of order, exchange
In order, do not exchange

6. Bubble Sort Algorithm 1. Initialize numCompares to n - 1
2. While numCompares!= 0, do following
a. Set last = 1 // location of last element in a swap
b. For i = 1 to numPairs if xi > xi Swap xi and xi + 1 and set last = i
c. Set numCompares = last – 1
End while

7. Bubble Sort Algorithm 1. Initialize numCompares to n - 1
2. While numCompares!= 0, do following
a. Set last = 1 // location of last element in a swap
b. For i = 1 to numPairs if xi > xi Swap xi and xi + 1 and set last = i
c. Set numCompares = last – 1
End while …
Allows it to quit if
In order
Try:
Also allows us to
Label the highest as
sorted

8. Categories of Sorting Algorithms
3. Insertion sort
Repeatedly insert a new element into an already sorted list
Note this works well with a linked list implementation

9. Example of Insertion Sort
Given list to be sorted 67, 33, 21, 84, 49, 50, 75
Note sequence of steps carried out
THE CURRENT NUMBER IS PULLED OUT INTO A TEMP LOCATION (POS 0) - SO THAT NUMS CAN BE MOVED OVER IF NECESSARY!
Position 0 is used to store the item that is being insert --- the result is a sorted array from x[1] to x[n]

10. Improved Schemes These 3 sorts - have computing time O(n2)
We seek improved computing times for sorts of large data sets
There are sorting schemes which can be proven to have average computing time O( n log2n )
No universally good sorting scheme
Results may depend on the order of the list

11. Comparisons of Sorts Sort of a randomly generated list of 500 items
Note: times are on 1970s hardware
Algorithm
Type of Sort
Time (sec)
Simple selection
Heapsort
Bubble sort
2 way bubble sort
Quicksort
Linear insertion
Binary insertion
Shell sort
Selection
Exchange
Insertion 69 18
165
141 6 66 37 11

12. Indirect Sorts
What happens if items being sorted are large structures (like objects)?
Data transfer/swapping time unacceptable
Alternative is indirect sort
Uses index table to store positions of the objects
Manipulate the index table for ordering

13. Heaps A heap is a binary tree with properties: It is complete
Each level of tree completely filled
Except possibly bottom level (nodes in left most positions)
It satisfies heap-order property
Data in each node >= data in children
Type of selection sort
Uses a new data structure called a heap to organize the data in such a way to make the sort efficient.

14. Heaps Which of the following are heaps? C A B
Yes – No (not complete) – No (out of order) (ONLY TREE A is a HEAP) A B C

15. Maximum and Minimum Heaps Example

16. Implementing a Heap Use an array or vector
Number the nodes from top to bottom, then on each row – left to right.
Store data in ith node in ith location of array (vector)

17. Implementing a Heap
Note the placement of the nodes in the array 41

18. Implementing a Heap
In an array implementation children of ith node are at myArray[2*i] and myArray[2*i+1]
Parent of the ith node is at mayArray[i/2]
Therefore PARENT at 3 then CHILDREN at 6 and 7

19. Basic Heap Operations Constructor Empty Retrieve max item
Set mySize to 0, allocate array (if dynamic array)
Empty
Check value of mySize
Retrieve max item
Return root of the binary tree, myArray[1]

20. Result called a semiheap
Basic Heap Operations
Delete max item (popHeap)
Max item is the root, replace with last node in tree
Then interchange root with larger of two children
Continue this with the resulting sub-tree(s) – result is a new heap.
Result called a semiheap

21. Exchanging elements when performing a popHeap()

22. Adjusting the heap for popHeap()

23. Percolate Down Algorithm converts semiheap to heap
r = current root node
n = number of nodes
1. Set c = 2 * r //location of left child
2. While r <= n do following // must be child(s) for root
a. If c < n and myArray[c] < myArray[c + 1] Increment c by 1 //find larger child b. If myArray[r] < myArray[c] i. Swap myArray[r] and myArray[c] ii. set r = c iii. set c = 2 * r else Terminate repetition End while
Percolate_down for semiheap: myArray[r]…myArray[n]
-only the value at heap[r] may fail the heap order condition
C = 2 * r (location of left child)
r = current root node
N = number of nodes
Recursive possibilities?

24. Basic Heap Operations Insert an item (pushHeap)
Amounts to a percolate up routine
Place new item at end of array
Interchange with parent so long as it is greater than its parent

25. Example of Heap Before and After Insertion of 50

26. Reordering the tree for the insertion

27. Heapsort Given a list of numbers in an array
Stored in a complete binary tree
Convert to a heap (heapify)
Begin at last node not a leaf
Apply “percolated down” to this subtree
Continue
Step 1 – convert array of numbers into a heap
– heapify the list of numbers
* To get non-leaf node - go to last item and calculate backwards

28. Example of Heapifying a Vector

29. Example of Heapifying a Vector
How are these nodes stored in an array?

30. Example of Heapifying a Vector
Remember: 17 is compared to both children – the larger of the children get brought up

31. Example of Heapifying a Vector

32. Example of Heapifying a Vector

33. Example of Heapifying a Vector

34. Heapsort
Algorithm for converting a complete binary tree to a heap – called "heapify" For r = n/2 down to 1: Apply percolate_down to the subtree in myArray[r] , … myArray[n] End for
Puts largest element at root
n = index for the last node in the tree therefore n/2 is the parent
How do you work the for loop in this case?
n = index for last node in tree
therefore n/2 is parent

35. Heapsort Now swap element 1 (root of tree) with last element
This puts largest element in correct location
Use percolate down on remaining sublist
Converts from semi-heap to heap

36. Heapsort Again swap root with rightmost leaf
Continue this process with shrinking sublist 60 60

37. Heapsort Algorithm
1. Consider x as a complete binary tree, use heapify to convert this tree to a heap
2. for i = n down to 2: a. Interchange x[1] and x[i] (puts largest element at end) b. Apply percolate_down to convert binary tree corresponding to sublist in x[1] .. x[i-1]
Notice how the list shrinks with each loop…

38. Example of Implementing heap sort
int arr[] = {50, 20, 75, 35, 25};
vector<int> v(arr, 5);

39. Example of Implementing heap sort

40. Example of Implementing heap sort

41. Heap Algorithms in STL Found in the <algorithm> library
make_heap() heapify
push_heap() insert
pop_heap() delete
sort_heap() heapsort
See example heapsortSTL in the folder “cs212” and heapsortSTLBugNumber in the folder “BigNumber”

42. Priority Queue A collection of data elements Implementation ?
Items stored in order by priority
Higher priority items removed ahead of lower
Implementation ?

43. Implementation possibilities
list (array, vector, linked list)
insert – O(1)
remove max - O(n)
ordered list
insert - linear insertion sort O(n)
remove max - O(1)
Heap (Best)
Basic operations have O(log2n) time

44. Priority Queue Basic Operations Constructor Insert
Find, remove smallest/largest (priority) element
Replace
Change priority
Delete an item
Join two priority queues into a larger one

45. Priority Queue STL priority queue adapter uses heap
priority_queue<BigNumber, vector<BigNumber> > v;
cout << "BIG NUMBER DEMONSTRATION" << endl;
for(int k=0;k<6;k++) {
cout << "Enter BigNumber: "; cin >> a;
v.push(a); }
cout<<"POP IN ORDER"<<endl;
while(!v.empty())
cout<<v.top()<<endl;
v.pop();
Basic operations - page 751

46. Quicksort More efficient exchange sorting scheme
(bubble sort is an exchange sort)
Typical exchange: involves elements that are far apart
Fewer interchanges are required to correctly position an element.
Quicksort uses a divide-and-conquer strategy
A recursive approach:
The original problem partitioned into simpler sub problems
Each sub problem considered independently.
Subdivision continues until sub problems obtained are simple enough to be solved directly
More efficient exchange sorting scheme than bubble sort

47. Quicksort Basic Algorithm Choose an element - pivot
Perform sequence of exchanges so that
<elements less than P> <P> <elements greater than P>
All elements that are less than this pivot are to its left and
All elements that are greater than the pivot are to its right.
Divides the (sub)list into two smaller sub lists,
Each of which may then be sorted independently in the same way.

48. Quicksort recursive If the list has 0 or 1 elements, ANCHOR
return. // the list is sorted
Else do:
Pick an element in the list to use as the pivot.
  Split the remaining elements into two disjoint groups:
SmallerThanPivot = {all elements < pivot}
LargerThanPivot = {all elements > pivot}
 Return the list rearranged as:
Quicksort(SmallerThanPivot),
pivot,
Quicksort(LargerThanPivot).

49. Quicksort Example
Given to sort: 75, 70, 65, , 98, 78, 100, 93, 55, 61, 81,
Select arbitrarily pivot: the first element 75
Search from right for elements <= 75, stop at first match
Search from left for elements > 75, stop at first match
Swap these two elements, and then repeat this process. When can you stop? 84 68

50. Quicksort Example When done, swap with pivot
75, 70, 65, 68, 61, 55, 100, 93, 78, 98, 81, 84
When done, swap with pivot
This SPLIT operation places pivot 75 so that all elements to the left are <= 75 and all elements to the right are >75.
75 is in place.
Need to sort sublists on either side of 75
Swap with pivot – swap the highest of the low numbers…must keep our finger on this one.

51. Quicksort Example Need to sort (independently): 55, 70, 65, 68, 61 and
100, 93, 78, 98, 81, 84
Let pivot be 55, look from each end for values larger/smaller than 55, swap
Same for 2nd list, pivot is 100
Sort the resulting sublists in the same manner until sublist is trivial (size 0 or 1)

52. QuickSort Recursive Function
template <typename ElementType>
void quicksort (ElementType x[], int first int last) {
int pos; // pivot's final position
if (first < last) // list size is > 1
split(x, first, last, pos); // Split into 2 sublists
quicksort(x, first, pos - 1); // Sort left sublist
quicksort(x,pos + 1, last); // Sort right sublist }

53. template <typename ElementType>
void split (ElementType x[], int first, int last, int & pos) {
ElementType pivot = x[first]; // pivot element
int left = first, // index for left search
right = last; // index for right search
while (left < right)
while (pivot < x[right]) // Search from right for
right--; // element <= pivot
// Search from left for
while (left < right && // element > pivot
x[left] <= pivot)
left++;
if (left < right) // If searches haven't met
swap (x[left], x[right]); // interchange elements }
// End of searches; place pivot in correct position
pos = right;
x[first] = x[pos];
x[pos] = pivot; } 
Code of the split function

54. Quicksort
Visual example of a quicksort on an array
etc. …

55. QuickSort Example
v = {800, 150, 300, 650, 550, 500, 400, 350, 450, 900}
Note: use of v[0] – item indexed by 0 holds the pivot.
Step 1 – pivot is selected (random)
Pivot selected at random

56. QuickSort Example

57. QuickSort Example

58. QuickSort Example

59. QuickSort Example
quicksort(x, 0, 4);
quicksort(x, 6, 9);

60. QuickSort Example
quicksort(x, 0, 0);
quicksort(x, 2, 4);

61. QuickSort Example
quicksort(x, 6, 6);
quicksort(x, 8, 9);

62. QuickSort Example

63. Quicksort Performance
O(n log2n) is the average case computing time
If the pivot results in sublists of approximately the same size.
O(n2) worst-case
List already ordered or elements in reverse.
When Split() repeatedly creates a sublist with one element. (when pivot is always smallest or largest value)
What 2 pivots would result in empty sublist?

64. Improvements to Quicksort
An arbitrary pivot gives a poor partition for nearly sorted lists (or lists in reverse)
Virtually all the elements go into either SmallerThanPivot or LargerThanPivot
all through the recursive calls.
Quicksort takes quadratic time to do essentially nothing at all.
Stopped 3/31

65. Improvements to Quicksort
Better method for selecting the pivot is the
median-of-three rule,
Select the median (middle value) of the first, middle, and last elements in each sublist as the pivot.
(4 10 6) - median is 6
Often the list to be sorted is already partially ordered
Median-of-three rule will select a pivot closer to the middle of the sublist than will the “first-element” rule.
Median – select the middle value select 34 as the pivot.

66. Improvements to Quicksort
Quicksort is a recursive function
stack of activation records must be maintained by system to manage recursion.
The deeper the recursion is, the larger this stack will become. (major overhead)
The depth of the recursion and the corresponding overhead can be reduced
sort the smaller sublist at each stage

67. Improvements to Quicksort
Another improvement aimed at reducing the overhead of recursion is to use an iterative version of Quicksort()
Implementation: use a stack to store the first and last positions of the sublists sorted "recursively". In other words – create your own low-overhead execution stack.

68. Improvements to Quicksort
For small files (n <= 20), quicksort is worse than insertion sort;
small files occur often because of recursion.
Use an efficient sort (e.g., insertion sort) for small files.
Better yet, use Quicksort() until sublists are of a small size and then apply an efficient sort like insertion sort.

69. Mergesort Sorting schemes are either …
internal -- designed for data items stored in main memory
external -- designed for data items stored in secondary memory.
Previous sorting schemes were all internal sorting algorithms:
required direct access to list elements
not possible for sequential files
made many passes through the list
not practical for files

70. Mergesort
Mergesort can be used both as an internal and an external sort.
Basic operation in mergesort is merging,
combining two lists that have previously been sorted
resulting list is also sorted.

71. Merge Algorithm 1. Open File1 and File2 for input, File3 for output
2. Read first element x from File1 and first element y from File2
3. While neither eof File1 or eof File2 If x < y then a. Write x to File3 b. Read a new x value from File1 Otherwise a. Write y to File3 b. Read a new y from File2 End while
4. If eof File1 encountered copy rest of of File2 into File3. If eof File2 encountered, copy rest of File1 into File3

72. Binary Merge Sort Given a single file
Split into two files (alternatively into each file)
Split the two files – alternatively taking a value from F and placing them into F1 and F2.

73. Binary Merge Sort
Merge first one-element "subfile" of F1 with first one-element subfile of F2
Gives a sorted two-element subfile of F
Continue with rest of one-element subfiles
If either F1 or F2 contains a remaining sublist – simply copy that to F

74. Binary Merge Sort Split again Merge again as before
Each time, the size of the sorted subgroups doubles

75. Note we always are limited to subfiles of some power of 2
Binary Merge Sort
Last splitting gives two files each in order
Last merging yields a single file, entirely in order
Note we always are limited to subfiles of some power of 2
CRITICISM: restricts itself to subfile of size 1, 2, 4, 8, …. 2^k where 2^k >= size of F must go through a series of k split-merge phases.
ALLOW OTHER SIZES – SO THAT # OF PHASES IS REDUCED IF FILES CONTAIN LONGER “RUNS”

76. Natural Merge Sort Allows sorted subfiles of other sizes
Number of phases can be reduced when file contains longer "runs" of ordered elements
Consider file to be sorted, note in order groups
Takes advantage of natural longer “runs”

77. Natural Merge Sort Copy alternate groupings into two files
Use the sub-groupings, not a power of 2
Look for possible larger groupings

78. EOF for F2, Copy remaining groups from F1
Natural Merge Sort
Merge the corresponding sub files
EOF for F2, Copy remaining groups from F1

79. Natural Merge Sort Split again, alternating groups
Merge again, now two subgroups
One more split, one more merge gives sort

80. Natural Merge Sort Split algorithm for natural merge sort
Open F for input and F1 and F2 for output
While the end of F has not been reached:
Copy a sorted subfile of F into F1 as follows: repeatedly read an element of F and write it to F1 until the next element in F is smaller than this copied item or the end of F is reached.
If the end of F has not been reached, copy the next sorted subfile of F into F2 using the method above.
End while.

81. Natural Merge Sort Merge algorithm for natural merge sort
Open F1 and F2 for input, F for output.
Initialize numSubfiles to 0
While not eof F1 or not eof F2
While no end of subfile in F1 or F2 has been reached:
If the next element in F1 is less than the next element in F2
Copy the next element from F1 into F.
Else
Copy the next element from F2 into F.
End While
If the eof F1 has been reached
Copy the rest of subfile F2 to F.
Copy the rest of subfile F1 to F.
Increment numSubfiles by 1.
Copy any remaining subfiles to F, incrementing numSubfiles by 1 for each.

82. Natural Merge Sort Mergesort algorithm
Repeat the following until numSubfiles is equal to 1:
Call the Split algorithm to split F into files F1 and F2.
Call the Merge algorithm to merge corresponding subfiles in F1 and F2 back into F.
Worst case for natural merge sort O(n log2n)
Worst case – items in reverse order.
This algorithm can easily be modified to become an internal sort – using arrays or vectors.

83. Natural MergeSort Example
The merge algorithm takes a sequence of elements in a vector v having index range [first, last). The sequence consists of two ordered sublists separated by an intermediate index, called mid.

84. Natural MergeSort Review

85. Natural MergeSort Review
So forth and so on…

86. Natural MergeSort Review
Finish with the few remaining nums from the sublist B and then copy all of the numbers from the tempVector to the original vector.

87. Recursive Natural MergeSort
Recursively work this sort down to the lowest level (anchor condition: only one element left) - work our way back to the top merging as we go.

88. Recursive Natural MergeSort
Show them the simulation for mergesort recursive… ( or ( )
ANY ALGORITHM WHICH PERFORMS SORTING USING COMPARISONS CANNOT HAVE A WORST-CASE PERFORMANCE BETTER THAN O(n log n)
A SORTING ALGORITHM BASED ON COMPARISIONS CANNOT BE O(n) - even its average runtime.
TO OBTAIN O(n) – YOU MUST USE A SORT WITHOUT COMPARISONS LIKE Radix Sort

89. Sorting Fact
any algorithm which performs sorting using comparisons cannot have a worst-case performance better than O(n log n)
a sorting algorithm based on comparisons cannot be O(n) - even for its average runtime.

90. Radix Sort
Based on examining digits in some base-b numeric representation of items
Least significant digit radix sort
Processes digits from right to left
Create groupings of items with same value in specified digit
Collect in order and create grouping with next significant digit
Base-b - 10-base 16-base 2-base
Used in early punched-card sorting machines (drop your card and then need to sort them using hoppers)

91. Radix Sort
Order ten 2 digit numbers in 10 bins from smallest number to largest number. Requires 2 calls to the sort Algorithm.
Initial Sequence:
Pass 0: Distribute the cards into bins according to the 1's digit (100).
TIME
Each pass through the array takes O(n) time. If the maximum magnitude of a number in the array is `v', and we are treating entries as base `b' numbers, then 1+floor(logb(v)) passes are needed.
If `v' is a constant, radix sort takes linear time, O(n). Note however that if all of the numbers in the array are different then v is at least O(n), so O(log(n)) passes are needed, O(n.log(n))-time overall..
EXTRA SPACE
If a temporary array is used, the extra work-space used is O(n). It is possible do the sorting on each digit-position in-situ and then only O(log(n)) space is needed to keep track of the array sections yet to be processed, either recursively or on an explicit stack.

92. Radix Sort Final Sequence: 91 6 85 15 92 35 30 22 39
Pass 1: Take the new sequence and distribute the cards into bins determined by the 's digit (101).
Show the program for this one (radix sort demo)
The inner loop (the instructions that distribute values into the containers are encountered most often) n times each time the loop is encountered
This process is executed in NUM_DIGITS * n times – O(n)

93. Sort Algorithm Analysis
Selection Sort (uses a swap)
Worst and average case O(n^2)
can be used with linked-list (doesn’t require random-access data)
Can be done in place
not at all fast for nearly sorted data

94. Sort Algorithm Analysis
Bubble Sort (uses an exchange)
Worst and average case O(n^2)
Since it is using localized exchanges - can be used with linked-list
Can be done in place
O(n^2) - even if only one item is out of place

95. Sort Algorithm Analysis sorts actually used
Insertion Sort (uses an insert)
Worst and average case O(n^2)
Does not require random-access data
Can be done in place
It is fast (linear time) for nearly sorted data
It is fast for small lists
Most good sorting methods call Insertion Sort for small lists

96. Sort Algorithm Analysis sorts actually used
Merge Sort
Worst and average case O(n log n)
Does not require random-access data
For linked-list - can be done in place
For an array - need to use a buffer
It is not significantly faster on nearly sorted data (but it is still log-linear time)

97. Sort Algorithm Analysis sorts actually used
QuickSort
Worst O(n^2)
Average case O(n log n) [good time]
can be done in place
Additional space for recursion O(log n)
Can be slow O(n^2) for nearly sorted or reverse data.
Sort used for STL sort()

98. End of sorting