Sorting Algorithms. Motivation Example: Phone Book Searching Example: Phone Book Searching If the phone book was in random order, we would probably never.

Sorting Algorithms

Motivation Example: Phone Book Searching Example: Phone Book Searching If the phone book was in random order, we would probably never use the phone! If the phone book was in random order, we would probably never use the phone! Potentially, we would need to look at every entry in the book Potentially, we would need to look at every entry in the book Let’s say ½ second per entry Let’s say ½ second per entry There are 72,000 households in Ilam There are 72,000 households in Ilam 35,000 seconds = 10hrs to find a phone number! 35,000 seconds = 10hrs to find a phone number! Next time make a call, could potentially take another 10+ hours Next time make a call, could potentially take another 10+ hours We might get lucky and the number we are looking for is the 1 st number – ½ second total We might get lucky and the number we are looking for is the 1 st number – ½ second total On average though, would be much more time expensive than the phone call itself (average time is about 5 hrs) On average though, would be much more time expensive than the phone call itself (average time is about 5 hrs)

Motivation Because we know the phone book is sorted: Because we know the phone book is sorted: Can use heuristics to facilitate our search Can use heuristics to facilitate our search Jump directly to the letter of the alphabet we are interested in using Jump directly to the letter of the alphabet we are interested in using Scan quickly to find the first two letters that are really close to the name we are interested in Scan quickly to find the first two letters that are really close to the name we are interested in Flip whole pages at a time if not close enough Flip whole pages at a time if not close enough Every time we pick up the phone book, know it will take a short time to find what we are looking for Every time we pick up the phone book, know it will take a short time to find what we are looking for

The Big Idea Take a set of N randomly ordered pieces of data a j and rearrange data such that for all j (j >= 0 and j = 0 and j < N), R holds, for relational operator R: a 0 R a 1 R a 2 R … a j … R a N-1 R a N a 0 R a 1 R a 2 R … a j … R a N-1 R a N If R is <=, we are doing an ascending sort – Each consecutive item in the list is going to be larger than the previous If R is <=, we are doing an ascending sort – Each consecutive item in the list is going to be larger than the previous If R is >=, we are doing a descending sort – Items get smaller as move down the list If R is >=, we are doing a descending sort – Items get smaller as move down the list

Sorting Algorithms What does sorting really require? What does sorting really require? Really a repeated two step process Really a repeated two step process Compare pieces of data at different positions Compare pieces of data at different positions Swap the data at those positions until order is correct Swap the data at those positions until order is correct For large groups of data, For large groups of data, Implementation by a computer program would be very useful Implementation by a computer program would be very useful Computers can do the data management (compare and swap) faster than people. Computers can do the data management (compare and swap) faster than people.

Example 20318952035918

Selection Sort void selectionSort(int* a, int size) { for (int k = 0; k < size-1; k++) { int index = mininumIndex(a, k, size); swap(a[k],a[index]);}} int minimumIndex(int* a, int first, int last) { int minIndex = first; for (int j = first + 1; j < last; j++) { if (a[j] < a[minIndex]) minIndex = j; } return minIndex; }

Selection Sort What is selection sort doing? What is selection sort doing? Repeatedly Repeatedly Finding smallest element by searching through list Finding smallest element by searching through list Inserting at front of list Inserting at front of list Moving “front of list” forward by 1 Moving “front of list” forward by 1

Selection Sort Step Through minIndex(a, 0, 5) ? =1 swap (a[0],a[1]) 2031895 2031895

Order From Previous Find minIndex (a, 1, 5) =4 Find minIndex (a, 2, 5) = 3 2031895 5318920 5318920 5391820

Find minIndex (a, 3, 5) = 3 K = 4 = size-1 Done! 5391820 5391820 5391820

Cost of Selection Sort void selectionSort(int* a, int size) { for (int k = 0; k < size-1; k++) { int index = mininumIndex(a, k, size); swap(a[k],a[index]);}} int minimumIndex(int* a, int first, int last) { int minIndex = first; for (int j = first + 1; j < last; j++) { if (a[j] < a[minIndex]) minIndex = j; } return minIndex; }

Cost of Selection Sort How many times through outer loop? How many times through outer loop? Iteration is for k = 0 to N-1 times Iteration is for k = 0 to N-1 times How many comparisons in minIndex? How many comparisons in minIndex? Depends on outer loop – Consider 5 elements: Depends on outer loop – Consider 5 elements: K = 0 j = 1,2,3,4 K = 0 j = 1,2,3,4 K = 1 j = 2, 3, 4 K = 1 j = 2, 3, 4 K = 2 j = 3, 4 K = 2 j = 3, 4 K = 3 j = 4 K = 3 j = 4 Total comparisons is equal to 4 + 3 + 2 + 1, which is N-1 + N-2 + N-3 … + 1 Total comparisons is equal to 4 + 3 + 2 + 1, which is N-1 + N-2 + N-3 … + 1 What is that sum? What is that sum?

Cost of Selection Sort (N-1) + (N-2) + (N-3) + … + 3 + 2 + 1 (N-1) + 1 + (N-2) + 2 + (N-3) + 3 … N + N + N … => repeated addition of N How many repeated additions? There were n-1 total starting objects to add, we grouped every 2 together – approximately N/2 repeated additions => Approximately N * N/2 = O(N^2) comparisons

Insertion Sort void insertionSort(int* a, int size) { for (int k = 1; k < size; k++) { int temp = a[k]; int position = k; while (position > 0 && a[position-1] > temp) { a[position] = a[position-1]; position--;} a[position] = temp; }}

Insertion Sort List of size 1 (first element) is already sorted List of size 1 (first element) is already sorted Repeatedly Repeatedly Chooses new item to place in list (a[k]) Chooses new item to place in list (a[k]) Starting at back of the list, if new item is less than item at current position, shift current data right by 1. Starting at back of the list, if new item is less than item at current position, shift current data right by 1. Repeat shifting until new item is not less than thing in front of it. Repeat shifting until new item is not less than thing in front of it. Insert the new item Insert the new item

Insertion Sort Insertion Sort is the classic card playing sort. Insertion Sort is the classic card playing sort. Pick up cards one at a time. Pick up cards one at a time. Single card already sorted. Single card already sorted. When select new card, slide it in at the appropriate point by shifting everything right. When select new card, slide it in at the appropriate point by shifting everything right. Easy for humans because shifts are “free”, where for computer have to do data movement. Easy for humans because shifts are “free”, where for computer have to do data movement.

318 Insertion Sort Step Through Single card list already sorted A[0] A[1]A[2]A[3]A[4] A[0]A[1] A[2]A[3]A[4] Move 3 left until hits something smaller 20 95 2031895

A[0]A[1] A[2]A[3]A[4] Move 3 left until hits something smaller Now two sorted A[0]A[1]A[2] A[3]A[4] Move 18 left until hits something smaller 1895 320 2018 95 3

A[0]A[1]A[2] A[3]A[4] Move 18 left until hits something smaller Now three sorted A[0]A[1]A[2]A[3] A[4] Move 9 left until hits something smaller 31820 95 3 18 209 5

A[0]A[1]A[2]A[3] A[4] Move 9 left until hits something smaller Now four sorted A[0]A[1]A[2]A[3]A[4] Move 5 left until hits something smaller 391820 3918205 5

A[0]A[1]A[2]A[3]A[4] Move 5 left until hits something smaller Now all five sorted Done 3918205

Cost of Insertion Sort void insertionSort(int* a, int size) { for (int k = 1; k < size; k++) { int temp = a[k]; int position = k; while (position > 0 && a[position-1] > temp) { a[position] = a[position-1]; position--;} a[position] = temp; }}

Cost of Insertion Sort Outer loop Outer loop K = 1 to N-1 K = 1 to N-1 Inner loop Inner loop Worst case: Compare against all items in list Worst case: Compare against all items in list Inserting new smallest thing Inserting new smallest thing K = 1, 1 step (position = k = 1, while position > 0) K = 1, 1 step (position = k = 1, while position > 0) K = 2, 2 steps [position = 2,1] K = 2, 2 steps [position = 2,1] K = 3, 3 steps [position = 3,2,1] K = 3, 3 steps [position = 3,2,1] K = 4, 4 steps [position = 4,3,2,1] K = 4, 4 steps [position = 4,3,2,1] Again, worst case total comparisons is equal to sum of I from 1 to N-1, which is O(N 2 ) Again, worst case total comparisons is equal to sum of I from 1 to N-1, which is O(N 2 )

Cost of Swaps Selection Sort: void selectionSort(int* a, int size) { for (int k = 0; k < size-1; k++) { int index = mininumIndex(a, k, size); swap(a[k],a[index]);}} One swap each time, for O(N) swaps One swap each time, for O(N) swaps

Cost of Swaps Insertion Sort void insertionSort(int* a, int size) { for (int k = 1; k < size; k++) { int temp = a[k]; int position = k; while (position > 0 && a[position-1] > temp) { a[position] = a[position-1]; position--;} a[position] = temp; }} Do a shift almost every time do compare, so O(n 2 ) shifts Do a shift almost every time do compare, so O(n 2 ) shifts Shifts are faster than swaps (1 step vs 3 steps) Shifts are faster than swaps (1 step vs 3 steps) Are we doing few enough of them to make up the difference? Are we doing few enough of them to make up the difference?

Another Issue - Memory Space requirements for each sort? Space requirements for each sort? All of these sorts require the space to hold the array - O(N) All of these sorts require the space to hold the array - O(N) Require temp variable for swaps Require temp variable for swaps Require a handful of counters Require a handful of counters Can all be done “in place”, so equivalent in terms of memory costs Can all be done “in place”, so equivalent in terms of memory costs Not all sorts can be done in place though! Not all sorts can be done in place though!

Which O(n 2 ) Sort to Use? Insertion sort is the winner: Insertion sort is the winner: Worst case requires all comparisons Worst case requires all comparisons Most cases don’t (jump out of while loop early) Most cases don’t (jump out of while loop early) Selection use for loops, go all the way through each time Selection use for loops, go all the way through each time

Tradeoffs Given random data, when is it more efficient to: Given random data, when is it more efficient to: Just search versus Just search versus Insertion Sort and search Insertion Sort and search Assume Z searches Assume Z searches Search on random data: Z * O(n) Sort and binary search: O(n 2 ) + Z *log 2 n

Tradeoffs Z * n <= n 2 + (Z * log 2 n) Z * n – Z * log 2 n <= n 2 Z * (n-log 2 n) <= n 2 Z <= n 2 /(n-log 2 n) For large n, log 2 n is dwarfed by n in (n-log 2 n) For large n, log 2 n is dwarfed by n in (n-log 2 n) Z <= n 2 /n Z <= n (approximately)

Improving Sorts Better sorting algorithms rely on divide and conquer (recursion) Better sorting algorithms rely on divide and conquer (recursion) Find an efficient technique for splitting data Find an efficient technique for splitting data Sort the splits separately Sort the splits separately Find an efficient technique for merging the data Find an efficient technique for merging the data We’ll see two examples We’ll see two examples One does most of its work splitting One does most of its work splitting One does most of its work merging One does most of its work merging

Quicksort General Quicksort Algorithm: Select an element from the array to be the pivot Select an element from the array to be the pivot Rearrange the elements of the array into a left and right subarray Rearrange the elements of the array into a left and right subarray All values in the left subarray are < pivot All values in the left subarray are < pivot All values in the right subarray are > pivot All values in the right subarray are > pivot Independently sort the subarrays Independently sort the subarrays No merging required, as left and right are independent problems [ Parallelism?!? ] No merging required, as left and right are independent problems [ Parallelism?!? ]

Quicksort void quicksort(int* arrayOfInts, int first, int last) { int pivot; int pivot; if (first < last) if (first < last) { pivot = partition(arrayOfInts, first, last); pivot = partition(arrayOfInts, first, last); quicksort(arrayOfInts,first,pivot -1); quicksort(arrayOfInts,first,pivot -1); quicksort(arrayOfInts,pivot+1,las t); quicksort(arrayOfInts,pivot+1,las t); }}

Quicksort int partition(int* arrayOfInts, int first, int last) { int temp; int temp; int p = first; // set pivot = first index int p = first; // set pivot = first index for (int k = first+1; k <= last; k++) // for every other indx for (int k = first+1; k <= last; k++) // for every other indx { if (arrayOfInts[k] <= arrayOfInts[first]) // if data is smaller if (arrayOfInts[k] <= arrayOfInts[first]) // if data is smaller { p = p + 1; // update final pivot location p = p + 1; // update final pivot location swap(arrayOfInts[k], arrayOfInts[p]); swap(arrayOfInts[k], arrayOfInts[p]); } } swap(arrayOfInts[p], arrayOfInts[first]); swap(arrayOfInts[p], arrayOfInts[first]); return p; return p;}

9532018 9183205 9318205 Partition Step Through partition(cards, 0, 4) P = 0 K = 1P = 1 K = 3 cards[1] < cards[0] ? Nocards[3] < cards[0]? Yes P = 2 P = 0 K = 2temp = cards[3] cards[2] < cards[0] ? Yescards[3] = cards[2] P = 1cards[2] = cards[3] temp = cards[2]P = 2 K = 4 cards[2] = cards[1]cards[4] < cards[0]? No cards[1] = temp temp = cards[2], cards[2] = cards[first] cards[first] = temp, return p = 2; 3918205

Complexity of Quicksort Worst case is O(n 2 ) Worst case is O(n 2 ) What does worst case correspond to? What does worst case correspond to? Already sorted or near sorted Already sorted or near sorted Partitioning leaves heavily unbalanced subarrays Partitioning leaves heavily unbalanced subarrays On average is O(n log 2 n), and it is average a lot of the time. On average is O(n log 2 n), and it is average a lot of the time.

Complexity of Quicksort Recurrence Relation: [Average Case] 2 sub problems ½ size (if good pivot) Partition is O(n) a = 2 b = 2 k = 1 2 = 2 1 Master Theorem: O(nlog 2 n)

Complexity of Quicksort Recurrence Relation: [Worst Case] Partition separates into (n-1) and (1) Partition separates into (n-1) and (1) Can’t use master theorem: Can’t use master theorem: b (subproblem size) changes n-1/n n-2/n-1 n-3/n-2 Note that sum of partition work: Note that sum of partition work: n + (n-1) + (n-2) + (n-3) … Sum(1,N) = N(N+1)/2= O(N 2 )

Complexity of Quicksort Requires stack space to implement recursion Requires stack space to implement recursion Worst case: O(n) stack space Worst case: O(n) stack space If pivot breaks into 1 element and n-1 element subarrays If pivot breaks into 1 element and n-1 element subarrays Average case: O(log n) Average case: O(log n) Pivot splits evenly Pivot splits evenly

MergeSort General Mergesort Algorithm: General Mergesort Algorithm: Recursively split subarrays in half Recursively split subarrays in half Merge sorted subarrays Merge sorted subarrays Splitting is first in recursive call, so continues until have one item subarrays Splitting is first in recursive call, so continues until have one item subarrays One item subarrays are by definition sorted One item subarrays are by definition sorted Merge recombines subarrays so result is sorted Merge recombines subarrays so result is sorted 1+1 item subarrays => 2 item subarrays 1+1 item subarrays => 2 item subarrays 2+2 item subarrays => 4 item subarrays 2+2 item subarrays => 4 item subarrays Use fact that subarrays are sorted to simplify merge algorithm Use fact that subarrays are sorted to simplify merge algorithm

MergeSort void mergesort(int* array, int* tempArray, int low, int high, int size) { if (low < high) { int middle = (low + high) / 2; int middle = (low + high) / 2; mergesort(array,tempArray,low,middle, size); mergesort(array,tempArray,low,middle, size); mergesort(array,tempArray,middle+1, high, size); mergesort(array,tempArray,middle+1, high, size); merge(array,tempArray,low,middle,high, size); merge(array,tempArray,low,middle,high, size); }}

MergeSort void merge(int* array, int* tempArray, int low, int middle, int high, int size) { int i, j, k; int i, j, k; for (i = low; i <= high; i++) { tempArray[i] = array[i]; } // copy into temp array for (i = low; i <= high; i++) { tempArray[i] = array[i]; } // copy into temp array i = low; j = middle+1; k = low; i = low; j = middle+1; k = low; while ((i <= middle) && (j <= high)) {// merge while ((i <= middle) && (j <= high)) {// merge if (tempArray[i] <= tempArray[j])// if lhs item is smaller if (tempArray[i] <= tempArray[j])// if lhs item is smaller array[k++] = tempArray[i++];// put in final array, increment else// final array position, lhs index else// final array position, lhs index array[k++] = tempArray[j++];// else put rhs item in final array }// increment final array position }// increment final array position // rhs index while (i <= middle)// one of the two will run out while (i <= middle)// one of the two will run out array[k++] = tempArray[i++];// copy the rest of the data array[k++] = tempArray[i++];// copy the rest of the data }// only need to copy if in lhs array // rhs array already in right place

MergeSort Example Recursively Split 2031895 2031895

MergeSort Example Recursively Split 2031895 203 18 9 5

MergeSort Example Merge 2031895

Merge Sort Example Temp Array i j Array Temp[i] < Temp[j] Yes k 2 cards Not very interesting Think of as swap 203 18 203 203 3

MergeSort Example Temp Array i j Array Temp[i] < Temp[j] No k Update J, K by 1 => Hit Limit of Internal While Loop, as J > High Now Copy until I > Middle Array k 18203 318 20183

MergeSort Example 2 Card Swap Final after merging above sets 9559 2018359 3 i=1,j=3 5 i=1,j=4 i=0,j=3 9 i=1,j=5 18 i=2,j=5 20 i=3,j=5

Complexity of MergeSort Recurrence relation: 2 subproblems ½ size Merging is O(n) for any subproblem Always moving forwards in the array a = 2 b = 2 k = 1 2 = 2 1 Master Theorem: O(n log 2 n) Always O(n log 2 n) in both average and worst case Doesn’t rely on quality of pivot choice

Space Complexity of Mergesort Need an additional O(n) temporary array Need an additional O(n) temporary array Number of recursive calls: Number of recursive calls: Always O(log 2 n) Always O(log 2 n)

Tradeoffs When it is more useful to: When it is more useful to: Just search Just search Quicksort or Mergesort and search Quicksort or Mergesort and search Assume Z searches Assume Z searches Search on random data: Z * O(n) Fast Sort and binary search: O(nlog 2 n) + Z *log 2 n

Tradeoffs Z * n <= nlog 2 n + Zlog 2 n Z(n - log 2 n) <= n log 2 n Z <= (n log 2 n) / (n-log 2 n) Z <= (n log 2 n) / n [Approximation] Z <= log 2 n [Approximation] Where as before, had to do N searches to make up for cost of sorting, now only do log 2 N 1,000,000 items = 19 searches, instead of 1,000,000

How Fast? Without specific details of what sorting, O(n log 2 n) is the maximum speed sort possible. Without specific details of what sorting, O(n log 2 n) is the maximum speed sort possible. Only available operations: Compare, Swap Only available operations: Compare, Swap Proof: Decision Tree – describes how sort operates Proof: Decision Tree – describes how sort operates Every vertex represents a comparison, every branch a result Every vertex represents a comparison, every branch a result Moving down tree – Tracing a possible run through the algorithm Moving down tree – Tracing a possible run through the algorithm

How Fast? K1 <= K2 [1,2,3] K2 <= K3K1 <= K3 [1,2,3] [2,1,3] Yes No stopK1 <= K3 Yes No K2 <= K3stop Yes No [1,2,3] [1,3,2] [2,1,3] [1,3,2][3,1,2] [2,3,1] [3,2,1]

How Fast? There are n! possible “stop” nodes – effectively all permutations of the n numbers in the array. There are n! possible “stop” nodes – effectively all permutations of the n numbers in the array. Thus any decision tree representing a sorting algorithm must have n! leaves Thus any decision tree representing a sorting algorithm must have n! leaves The height of a this type of tree (a binary tree) is correlated with number of leaves: The height of a this type of tree (a binary tree) is correlated with number of leaves: Height k = 2^(k-1) leaves Height k = 2^(k-1) leaves Must be at least log 2 n! + 1 height Must be at least log 2 n! + 1 height

How Fast? Path from top to bottom of tree – trace of a run of the algorithm Path from top to bottom of tree – trace of a run of the algorithm Need to prove that (log 2 n!) is lower bounded by (n log 2 n) Need to prove that (log 2 n!) is lower bounded by (n log 2 n) n! = (n)(n-1)(n-2)(n-3) … (3)(2)(1) > (n)(n-1)(n-2)(n-3) … ceil(n/2) // doing fewer multiplies > (n)(n-1)(n-2)(n-3) … ceil(n/2) // doing fewer multiplies > ceil(n/2) (ciel(n/2)) // doing multiplies of bigger things > ceil(n/2) (ciel(n/2)) // doing multiplies of bigger things > approximately (n/2) (n/2) > approximately (n/2) (n/2) log 2 n! > log 2 (n/2) (n/2) log 2 n! > (n/2) log 2 (n/2)//exponentiation in logs = multiplication out front log 2 n! > (n/2)(log 2 n – log 2 2) // division in logs = subtraction log 2 n! > (n/2)(log 2 n – 1) log 2 n! > (n/2)(log 2 n) – (n/2) log 2 n! > (1/2) [nlog 2 n – n] log 2 n! ~ O(n log 2 n)

How Fast? Lower bound of n 2 ? No Lower bound of n 2 ? No Would Need to prove log 2 n! >= n 2 Would Need to prove log 2 n! >= n 2 Meaning we would need to prove n! > 2^(n 2 ) Meaning we would need to prove n! > 2^(n 2 )

Stable Sorts Stable sort: A sort that preserves the input ordering of elements that are equal Why important? Maintenance of pre-specified ordering / properties An example: Preservation of a previous sort on another key

Stable Sorts Which sorts have we seen that are stable? Which sorts have we seen that are stable? Insertion Sort Insertion Sort Picking up in order from “extras” pile Picking up in order from “extras” pile Only move past if you are greater than item in front of you Only move past if you are greater than item in front of you Mergesort Mergesort No swapping No swapping Fill in result array from temp array left to right Fill in result array from temp array left to right Merging – Two equal things – one on the left stays on the left Merging – Two equal things – one on the left stays on the left Unstable: Unstable: Selection Sort- Why? Selection Sort- Why? Quick Sort- Why? Quick Sort- Why?

Selection Sort – Unstable void selectionSort(int* a, int size) { for (int k = 0; k < size-1; k++) { int index = mininumIndex(a, k, size); swap(a[k],a[index]);}} int minimumIndex(int* a, int first, int last) { int minIndex = first; for (int j = first + 1; j < last; j++) { if (a[j] < a[minIndex]) minIndex = j; } return minIndex; } Unstable because swapping may rearrange order of original items (Sort doesn’t care what it picks up and moves)

Selection Sort - Unstable Do Swap to Put 2 of Spades in Place for Value Sort – Ruins 3’s Type Sort 2a3a3b2b 2a 3a3b2b

Quicksort int partition(int* arrayOfInts, int first, int last) { int temp; int temp; int p = first; // set pivot = first index int p = first; // set pivot = first index for (int k = first+1; k <= last; k++) { // for every other index for (int k = first+1; k <= last; k++) { // for every other index if (arrayOfInts[k] <= arrayOfInts[first]) { // if data on is smaller if (arrayOfInts[k] <= arrayOfInts[first]) { // if data on is smaller p = p + 1; // update final pivot location p = p + 1; // update final pivot location temp = arrayOfInts[k];// rearrange data temp = arrayOfInts[k];// rearrange data arrayOfInts[k] = arrayOfInts[p]; arrayOfInts[k] = arrayOfInts[p]; arrayOfInts[p] = temp; } } arrayOfInts[p] = temp; } } temp = arrayOfInts[p];// put pivot in right spot temp = arrayOfInts[p];// put pivot in right spot arrayOfInts[p] = arrayOfInts[first]; arrayOfInts[p] = arrayOfInts[first]; arrayOfInts[first] = temp; arrayOfInts[first] = temp; return p; return p;} Possibility to put pivot/item where pivot belongs out of order w/ respect to previous sort

Sorting Algorithms. Motivation Example: Phone Book Searching Example: Phone Book Searching If the phone book was in random order, we would probably never.

Similar presentations

Presentation on theme: "Sorting Algorithms. Motivation Example: Phone Book Searching Example: Phone Book Searching If the phone book was in random order, we would probably never."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Sorting Algorithms. Motivation Example: Phone Book Searching Example: Phone Book Searching If the phone book was in random order, we would probably never.

Similar presentations

Presentation on theme: "Sorting Algorithms. Motivation Example: Phone Book Searching Example: Phone Book Searching If the phone book was in random order, we would probably never."— Presentation transcript:

Similar presentations

About project

Feedback