Sorting Algorithms

Motivation Example: Phone Book Searching Example: Phone Book Searching If the phone book was in random order, we would probably never use the phone! If the phone book was in random order, we would probably never use the phone! Let’s say ½ second per entry Let’s say ½ second per entry There are 70,000 households in Ilam There are 70,000 households in Ilam 35,000 seconds = 10hrs to find a phone number! 35,000 seconds = 10hrs to find a phone number! Best time: ½ second Best time: ½ second average time is about 5 hrs average time is about 5 hrs

Motivation The phone book is sorted: The phone book is sorted: Jump directly to the letter of the alphabet we are interested in using Jump directly to the letter of the alphabet we are interested in using Scan quickly to find the first two letters that are really close to the name we are interested in Scan quickly to find the first two letters that are really close to the name we are interested in Flip whole pages at a time if not close enough Flip whole pages at a time if not close enough

The Big Idea Take a set of N randomly ordered pieces of data a j and rearrange data such that for all j (j >= 0 and j = 0 and j < N), R holds, for relational operator R: a 0 R a 1 R a 2 R … a j … R a N-1 R a N a 0 R a 1 R a 2 R … a j … R a N-1 R a N If R is <=, we are doing an ascending sort – Each consecutive item in the list is going to be larger than the previous If R is <=, we are doing an ascending sort – Each consecutive item in the list is going to be larger than the previous If R is >=, we are doing a descending sort – Items get smaller as move down the list If R is >=, we are doing a descending sort – Items get smaller as move down the list

Queue Example: Radix Sort Also called bin sort: Repeatedly shuffle data into small bins Collect data from bins into new deck Repeat until sorted Appropriate method of shuffling and collecting? For integers, key is to shuffle data into bins on a per digit basis, starting with the rightmost (ones digit) Collect in order, from bin 0 to bin 9, and left to right within a bin

Radix Sort: Ones Digit Data: Bin 0 Bin 1 Bin Bin 3 Bin Bin 5 Bin 6 Bin Bin 8 Bin After Call: After Call:

Radix Sort: Tens Digit Data: Bin 0 Bin 1 Bin 2 Bin Bin Bin Bin 6 Bin Bin 8 Bin 9 After Call:

Radix Sort: Hundreds Digit Data: Bin 0 Bin 1 Bin Bin 3 Bin Bin Bin Bin 7 Bin 8 Bin 9 Final Sorted Data:

Radix Sort Algorithm Begin with current digit as one’s digit While there is still a digit on which to classify { For each number in the master list, Add that number to the appropriate sublist keyed on the current digit For each sublist from 0 to 9 For each number in the sublist Remove the number from the sublist and append to a new master list Advance the current digit one place to the left. }

Radix Sort and Queues Each list (the master list (all items) and bins (per digit)) needs to be first in, first out ordered – perfect for a queue.

A Quick Tangent How fast have the sorts you’ve seen before worked? How fast have the sorts you’ve seen before worked? Bubble, Insertion, Selection: O(n^2) Bubble, Insertion, Selection: O(n^2) We will see sorts that are better, and in fact optimal for general sorting algorithms: We will see sorts that are better, and in fact optimal for general sorting algorithms: Merge/Quicksort: O(n log n) Merge/Quicksort: O(n log n) How fast is radix sort? How fast is radix sort?

Analysis of Radix Sort Let n be the number of items to sort Let n be the number of items to sort Outer loop control is on maximum length of input numbers in digits (Let this be d) Outer loop control is on maximum length of input numbers in digits (Let this be d) For every digit, For every digit, Assign each number to sort to a group (n operations) Assign each number to sort to a group (n operations) Pull each number back into the master list (n operations) Pull each number back into the master list (n operations) Overall running time: 2 * n * d => O(n) Overall running time: 2 * n * d => O(n)

Analysis of Radix Sort O(n log n) is optimal for general sorting algorithms O(n log n) is optimal for general sorting algorithms Radix sort is O(n)? How does that work? Radix sort is O(n)? How does that work? Radix sort is not a general sorting algorithm – It can’t sort arbitrary information – Rectangles objects, Automobiles objects, etc are no good. Radix sort is not a general sorting algorithm – It can’t sort arbitrary information – Rectangles objects, Automobiles objects, etc are no good. Can sort items that can be broken into constituent pieces and whose pieces can be ordered Can sort items that can be broken into constituent pieces and whose pieces can be ordered Integers (digits), Strings (characters) Integers (digits), Strings (characters)

Sorting Algorithms What does sorting really require? What does sorting really require? Compare pieces of data at different positions Compare pieces of data at different positions Swap the data at those positions until order is correct Swap the data at those positions until order is correct

Selection Sort void selectionSort(int* a, int size) { for (int k = 0; k < size-1; k++) { int index = mininumIndex(a, k, size); swap(a[k],a[index]);}} int minimumIndex(int* a, int first, int last) { int minIndex = first; for (int j = first + 1; j < last; j++) { if (a[j] < a[minIndex]) minIndex = j; } return minIndex; }

Selection Sort What is selection sort doing? What is selection sort doing? Repeatedly Repeatedly Finding smallest element by searching through list Finding smallest element by searching through list Inserting at front of list Inserting at front of list Moving “front of list” forward by 1 Moving “front of list” forward by 1

Selection Sort Step Through minIndex(a, 0, 5) ? =1 swap (a[0],a[1])

Order From Previous Find minIndex (a, 1, 5) =4 Find minIndex (a, 2, 5) =

Find minIndex (a, 3, 5) = 3 K = 4 = size-1 Done!

Cost of Selection Sort void selectionSort(int* a, int size) { for (int k = 0; k < size-1; k++) { int index = mininumIndex(a, k, size); swap(a[k],a[index]);}} int minimumIndex(int* a, int first, int last) { int minIndex = first; for (int j = first + 1; j < last; j++) { if (a[j] < a[minIndex]) minIndex = j; } return minIndex; }

Cost of Selection Sort How many times through outer loop? How many times through outer loop? Iteration is for k = 0 to N-1 times Iteration is for k = 0 to N-1 times How many comparisons in minIndex? How many comparisons in minIndex? Depends on outer loop – Consider 5 elements: Depends on outer loop – Consider 5 elements: K = 0 j = 1,2,3,4 K = 0 j = 1,2,3,4 K = 1 j = 2, 3, 4 K = 1 j = 2, 3, 4 K = 2 j = 3, 4 K = 2 j = 3, 4 K = 3 j = 4 K = 3 j = 4 Total comparisons is equal to , which is N-1 + N-2 + N-3 … + 1 Total comparisons is equal to , which is N-1 + N-2 + N-3 … + 1 What is that sum? What is that sum?

Cost of Selection Sort (N-1) + (N-2) + (N-3) + … (N-1) (N-2) (N-3) + 3 … N + N + N … => repeated addition of N How many repeated additions? There were n-1 total starting objects to add, we grouped every 2 together – approximately N/2 repeated additions => Approximately N * N/2 = O(N^2) comparisons

Insertion Sort void insertionSort(int* a, int size) { for (int k = 1; k < size; k++) { int temp = a[k]; int position = k; while (position > 0 && a[position-1] > temp) { a[position] = a[position-1]; position--;} a[position] = temp; }}

Insertion Sort List of size 1 (first element) is already sorted List of size 1 (first element) is already sorted Repeatedly Repeatedly Chooses new item to place in list (a[k]) Chooses new item to place in list (a[k]) Starting at back of the list, if new item is less than item at current position, shift current data right by 1. Starting at back of the list, if new item is less than item at current position, shift current data right by 1. Repeat shifting until new item is not less than thing in front of it. Repeat shifting until new item is not less than thing in front of it. Insert the new item Insert the new item

318 Insertion Sort Step Through Single card list already sorted A[0] A[1]A[2]A[3]A[4] A[0]A[1] A[2]A[3]A[4] Move 3 left until hits something smaller

A[0]A[1] A[2]A[3]A[4] Move 3 left until hits something smaller Now two sorted A[0]A[1]A[2] A[3]A[4] Move 18 left until hits something smaller

A[0]A[1]A[2] A[3]A[4] Move 18 left until hits something smaller Now three sorted A[0]A[1]A[2]A[3] A[4] Move 9 left until hits something smaller

A[0]A[1]A[2]A[3] A[4] Move 9 left until hits something smaller Now four sorted A[0]A[1]A[2]A[3]A[4] Move 5 left until hits something smaller

A[0]A[1]A[2]A[3]A[4] Move 5 left until hits something smaller Now all five sorted Done

Cost of Insertion Sort void insertionSort(int* a, int size) { for (int k = 1; k < size; k++) { int temp = a[k]; int position = k; while (position > 0 && a[position-1] > temp) { a[position] = a[position-1]; position--;} a[position] = temp; }}

Cost of Insertion Sort Outer loop Outer loop K = 1 to N-1 K = 1 to N-1 Inner loop Inner loop Worst case: Compare against all items in list Worst case: Compare against all items in list Inserting new smallest thing Inserting new smallest thing K = 1, 1 step (position = k = 1, while position > 0) K = 1, 1 step (position = k = 1, while position > 0) K = 2, 2 steps [position = 2,1] K = 2, 2 steps [position = 2,1] K = 3, 3 steps [position = 3,2,1] K = 3, 3 steps [position = 3,2,1] K = 4, 4 steps [position = 4,3,2,1] K = 4, 4 steps [position = 4,3,2,1] Again, worst case total comparisons is equal to sum of I from 1 to N-1, which is O(N 2 ) Again, worst case total comparisons is equal to sum of I from 1 to N-1, which is O(N 2 )

Cost of Swaps Selection Sort: void selectionSort(int* a, int size) { for (int k = 0; k < size-1; k++) { int index = mininumIndex(a, k, size); swap(a[k],a[index]);}} One swap each time, for O(N) swaps One swap each time, for O(N) swaps

Cost of Swaps Insertion Sort void insertionSort(int* a, int size) { for (int k = 1; k < size; k++) { int temp = a[k]; int position = k; while (position > 0 && a[position-1] > temp) { a[position] = a[position-1]; position--;} a[position] = temp; }} Do a shift almost every time do compare, so O(n 2 ) shifts Do a shift almost every time do compare, so O(n 2 ) shifts Shifts are faster than swaps (1 step vs 3 steps) Shifts are faster than swaps (1 step vs 3 steps) Are we doing few enough of them to make up the difference? Are we doing few enough of them to make up the difference?

Another Issue - Memory Space requirements for each sort? Space requirements for each sort? All of these sorts require the space to hold the array - O(N) All of these sorts require the space to hold the array - O(N) Require temp variable for swaps Require temp variable for swaps Require a handful of counters Require a handful of counters Can all be done “in place”, so equivalent in terms of memory costs Can all be done “in place”, so equivalent in terms of memory costs Not all sorts can be done in place though! Not all sorts can be done in place though!

Which O(n 2 ) Sort to Use? Insertion sort is the winner: Insertion sort is the winner: Worst case requires all comparisons Worst case requires all comparisons Most cases don’t (jump out of while loop early) Most cases don’t (jump out of while loop early) Selection use for loops, go all the way through each time Selection use for loops, go all the way through each time

Tradeoffs Given random data, when is it more efficient to: Given random data, when is it more efficient to: Just search versus Just search versus Insertion Sort and search Insertion Sort and search Assume Z searches Assume Z searches Search on random data: Z * O(n) Sort and binary search: O(n 2 ) + Z *log 2 n

Tradeoffs Z * n <= n 2 + (Z * log 2 n) Z * n – Z * log 2 n <= n 2 Z * (n-log 2 n) <= n 2 Z <= n 2 /(n-log 2 n) For large n, log 2 n is dwarfed by n in (n-log 2 n) For large n, log 2 n is dwarfed by n in (n-log 2 n) Z <= n 2 /n Z <= n (approximately)

Improving Sorts Better sorting algorithms rely on divide and conquer (recursion) Better sorting algorithms rely on divide and conquer (recursion) Find an efficient technique for splitting data Find an efficient technique for splitting data Sort the splits separately Sort the splits separately Find an efficient technique for merging the data Find an efficient technique for merging the data We’ll see two examples We’ll see two examples One does most of its work splitting One does most of its work splitting One does most of its work merging One does most of its work merging