Chapter 5 Efficiency and Analysis

Algorithm selection Algorithms are ordered lists of steps for solving a problem Algorithms are also abstractions of the programming process Often there are many different algorithms for solving the same problem

Algorithms: main considerations How fast will it run How much memory will it use How easy is it to implement The choice depends on the software –It may mean your algorithm is not the most efficient

Program efficiency Computer scientists analyze algorithms for their efficiency –In a machine independent manner –By focussing on the critical operations performed –Like the number of comparisons (for sorting routines)

Rate of growth The most important aspect of algorithm efficiency Two algorithms may be just as efficient for small data sets but differ greatly for large ones.

Asymptotic Efficiency The asymptotic efficiency of an algorithm –describes its relative efficiency as n gets very large. Often called the “order of complexity” “Big O” - O() Example: Sequential search is O(n)

Exercise –Rank the asymptotic orders of the following functions, from highest to lowest. (You may wish to graph some or all of them.)

Analyzing the Linear Search Assumes a list of n elements Assumes a search key A linear search looks for the target key value by proceeding in sequential fashion through the list

Sample record format keydata

Linear Search algorithm For each item in the list if the item’s key matches the target, stop and report “success” Report “failure”

Linear Search int linearSearch(int a[], int n, int target) { int i; for (i = 0; i < n; i++) if (a[i] == target) // key comparison return i; return -1; // use -1 to indicate failure }

Analysis Speed of an algorithm is measured by counting key comparisons Best case is 1 comparison Worst case is n comparisons Average case is n/2 comparisons –if the target is in the list

Analysis (continued) What if the target we are looking for has only a 50% chance of being in the list. The complexity must account for both targets that can be found and targets that cannot. For targets that can be found it is n/2 For targets that cannot be found it is n For targets that can be found only 50% of the time it is: 1/2*n/2 (found) + 1/2*n (not found) The order of complexity therefore is: n/4 + n/2 = 3n/4

Polynomial expressions All polynomial equations are made up of a set of terms. Usually with coefficients and exponents The fastest growing term is called the term of the “highest order” A linear search, like the last example, has only one term to describe it: 3n/4 This can be written as (3/4)n 3/4 is the coefficient, n is the exponent n is also the highest order term

From polynomials to Big-O A linear function is one whose growth is tied to n In other words, n is the highest order polynomial The growth of the expression is tied to the highest order term and is called the “order of magnitude” Order of magnitude is expressed as Big-O For linear functions, Big-O is n We express this as O(n) To find out what the order of an expression is, look for the highest order term.

Graphing Big-O The x axis corresponds to the number of elements in the list The y axis is the number of critical operations required by an algorithm The order of complexity is the highest order term in the analysis because it is the one that increases the fastest Constant multipliers are ignored Example: n+100 is graphed as O(n) For a linear search n/2 is also O(n) Similarly, from last example, 3n/4 is O(n)

A graph of the average case performance of Linear Search

Exercise –For the following formulas, identify the high-order term and indicate the order of each using big-O notation.

Binary Search Repeatedly divides the list in half and in half again The result is a great improvement on linear searching What is the asymptotic efficiency? –Best case is 1 comparison –Worst case is log(base 2)n Logarithmic complexity is much faster than linear –Even if we use the worst case

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] Searching for 26

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] Searching for 26

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] Searching for 26

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] Searching for 26

Equation 5-2

The growth of the base-2 log function

Graph illustrating relative growth of linear and logarithmic functions

Defective Binary Search int binarySearch(int a[], int n, int target) { // Precondition: array a is sorted in ascending order from a[0] to a[n-1] int first(0), last(n - 1), int mid; while (first <= last) { mid = (first + last)/2; if (target == a[mid]) return mid; else if (target < a[mid]) last = mid; else // must be that target > a[mid] first = mid; } return -1; // use -1 to indicate item not found }

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 8 Searching for 26 First 0 Last 17

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 4 Searching for 26 First 0 Last 8

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 6 Searching for 26 First 4 Last 8

Problem: bad loop invariant This code works fine for finding an item that is contained in the list. However, it does not work so well if the item is not in the list. We get an infinite loop in that case. The reason for this is the bad loop invariant

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 8 First 0 Last 17 Searching for 24

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 4 First 0 Last 8 Searching for 24

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 6 First 4 Last 8 Searching for 24

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 5 Searching for 24 First 4 Last 6

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 4 First 4 Last 5 Searching for 24 WE ARE NOW STUCK HERE FOREVER!!

Illustrated Invariant for Binary Search

Binary Search Invariant a[first] <= target <= a[last] This assumes target is in the list The correct invariant should be if target in a, then a[first] <= target <= a[last]

Importance of loop invariant Proving program correctness –Argument values (does the loop have what it needs to work correctly) –Termination (regardless of it’s correctness, will it ever stop) We violated the termination requirement of the loop invariant by not proving that it could be able to stop under all conditions

Proof of correctness We can show that the original loop invariant is incorrect using a simple trace. Tracing skills are an important part of your programmers skill set. The next slide contains an example of a trace for the binary search program that demonstrates the problem of the infinite loop.

Trace of Code Example 5-2

Verified Binary Search int binarySearch(int a[], int n, int target) { // Precondition: array a is sorted in ascending order // from a[0] to a[n-1] int first(0); int last(n - 1); int mid; while (first <= last) { // Invariant: // if target in a, then a[first] <= target <= a[last] mid = (first + last)/2;

Verified Binary Search if (target == a[mid]) return mid; else if (target < a[mid]) last = mid - 1; else // must be that target > a[mid] first = mid + 1; } return -1; //use -1 to indicate item not found }

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 8 First 0 Last 17 Searching for 24

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 4 First 0 Last 7 Searching for 24

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 6 First 5 Last 7 Searching for 24

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 5 Searching for 24 First 5 Last 5

Binary search strategy A[0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11][12] [13][14] [15][16] [17] A[mid] 5 Searching for 24 PROGRAM TERMINATES WHEN first > last First 5 Last 4

Analysis of simple sorting algorithms Selection sort Bubble sort

Analysis of selection sort Assume n element list Makes n-1 passes through the list Each pass has a sequential search of some portion of the n elements Analysis: n * n = O(n 2 )

int maxSelect(int a[], int n) { int maxPos(0), currentPos(1); while (currentPos < n) { // Invariant: a[maxPos] >= a[0]... a[currentPos-1] if (a[currentPos] > a[maxPos]) maxPos = currentPos; currentPos++; } return maxPos; } The maxSelect function

The operation of Selection Sort

Selection Sort void selectionSort(int a[], int n) { int last(n-1); int maxPos; while (last > 0) { // invariant: a[last+1]... a[n-1] is sorted && // everything in a[0]... a[last] <= everything in a[last+1]... a[n-1] maxPos = maxSelect(a, last+1); // last+1 is length from 0 to last swapElements(a, maxPos, last); last--; }

Selection Sort example maxPos currPos n A[0] A[1] A[2] A[3] A[4] last4

Selection Sort example maxPos currPos n A[0] A[1] A[2] A[3] A[4] last4

Selection Sort example maxPos currPos n A[0] A[1] A[2] A[3] A[4] last4

Selection Sort example maxPos currPos n A[0] A[1] A[2] A[3] A[4] last4

Selection Sort example maxPos currPos n A[0] A[1] A[2] A[3] A[4] last4

Swap, after one pass maxPos currPos n A[0] A[1] A[2] A[3] A[4] last4

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last3

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last3

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last3

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last3

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last3

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last2

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last2

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last2

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last2

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last1

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last1

Selection Sort maxPos currPos n A[0] A[1] A[2] A[3] A[4] last0

Result after each pass A[0] A[1] A[2] A[3] A[4]

Result after each pass A[0] A[1] A[2] A[3] A[4]

Result after each pass A[0] A[1] A[2] A[3] A[4]

Result after each pass A[0] A[1] A[2] A[3] A[4]

Result after each pass A[0] A[1] A[2] A[3] A[4]

Result after each pass A[0] A[1] A[2] A[3] A[4]

Result after each pass A[0] A[1] A[2] A[3] A[4]

Result after each pass A[0] A[1] A[2] A[3] A[4]

Analysis Number of passes: n-1 –the first pass guaranteed to place 1 item –the second guarantees a second –the third a third, etc. –After n-1 passes we have them all in place

Analysis: comparisons In the first pass we compared n-1 pairs In the second, n-2 In the third, n-3, etc. Actual number of comparisons made across all passes, for this example was: = 10

Order of complexity The number of passes is O(n) The number of comparisons in each pass is also O(n) Therefore, the order of complexity of the sort is O(n*n) = O(n 2 )

Analysis of selection sort (con’t) What if the list is sorted to begin with? This selection sort is a mindless one –it would not know that it should stop. Best case and worst case are the same (except for the swaps) The best, worst and average cases are all quadratic algorithms with O(n 2 )

Bubble Sort Also n-1 passes Also n-pass comparisons in each pass Order of complexity is therefore n-squared The same as the selection sort The main difference is that swapping may occur as many as n-1 times on a single pass, with the selection sort it only occurs once, at the end of the pass.

Example of one phase of Bubble Sort

bubbleSortPhase // void swapElements(int a[], int maxPos, int last); void bubbleSortPhase(int a[], int last) { // Precondition: a is an array indexed from a[0] to a[last] // Move the largest element between a[0] and a[last] into a[last], // by swapping out of order pairs int pos;

bubbleSortPhase for (pos = 0; pos < last - 1; pos++) if (a[pos] > a[pos+1]) { swapElements(a, pos, pos+1); } // Postconditions: a[0]... a[last] // contain the same elements, // possibly reordered; a[last] >= a[0]... a[last-1] }

Bubble Sort void bubbleSortPhase(int a[], int last); void bubbleSort(int a[], int n) { // Precondition: a is an array indexed from a[0] to a[n-1] int i; for (i = n - 1; i > 0; i--) bubbleSortPhase(a, i); // Postcondition: a is sorted }

Equation 5-7

Version 1 of bubble sort This version uses n-1 passes During each pass, n-1 pairs are compared Every time a pair needs to be swapped this is done before the pass can continue

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Bubble Sort: difficult example Posn A[0] A[1] A[2] A[3] A[4] last4 Pos+1

Result after each pass A[0] A[1] A[2] A[3] A[4]

Analysis For this example, we made n-1 passes through the array Each time we looked at n-1 pairs –(we should have only looked at n-pass pairs. Why?) Either way, the number of passes is O(n) The number of pairs processed in each pass is O(n) So, overall we get O(n*n) = O(n 2 )

Selection and Bubblesort comparison Both are O(n 2 ) sorts If we count up the number of critical operations for both sorts, handling n-1, n- 2, etc. pairs for each pass, using the data in the last example, we get Selection sort: 10 comparisons Bubble sort: 10 comparisons

What about swaps? The same swap function can be used for both programs. Let’s say it takes 3 operations, then the actual number of operations is Selection sort: (n-1) = 22 Bubble sort: (10) = 40 The selection sort uses roughly half the number of operations as the bubble sort

Moral Two sorts of the same order O(n 2 ) may not be the same speed. It depends on the data sets they are sorting. It also depends on the way they are implemented.

Data sets The data set we chose for the bubble sort was the worst one possible. What happens if we run it on the data set we originally used for the selection sort?

Result after pass A[0] A[1] A[2] A[3] A[4]

Result after pass A[0] A[1] A[2] A[3] A[4]

Improvement needed Neither the selection sort, nor the bubble sort know enough to stop if the list is sorted early! We should be able to come up with a smart version of the bubble sort that can do this. Instead of the outer for loop, let’s use a while loop that runs until the array is sorted.

Further improvements We will also make sure the inner loop runs the minimum number of times (n-pass) and that we keep track of whether a swap was needed for the pass we are on. If a swap was needed then we cannot assume the array is sorted. If a swap was not needed, then the array is sorted and we should send that signal to the outer loop.

Bubble sort improvements What improvements can you suggest for the bubble sort that we have seen?

Improvements Do not look at portions of the array that are already in sorted order Leave when the array is sorted –this is something the insertion and selection sorts we have seen could not do.

Improvements How much of an operational difference does the smart bubble sort have over it’s dumb form? How much of an operational difference does the smart bubble sort have over the selection sort? Do these improvements make a difference in Big-O? Which form of sort is best?

Bubble Sort void bubbleSort(int a[], int n) { // Precondition: a is an array indexed from a[0] to a[n-1] bool sorted(false); int pass(0); while (!sorted) { sorted = true; for (pos = 0; pos < last - 1; pos++) if (a[pos] > a[pos+1]) { swapElements(pos, pos+1); sorted = false; } pos++; } // Postcondition: a is sorted }

Result after pass A[0] A[1] A[2] A[3] A[4]

Result after pass A[0] A[1] A[2] A[3] A[4]

Analysis Only two passes are made 4 pairs are checked on pass 1 swap is called 3 times on pass 1 3 pairs are checked on pass 2, no swaps Total operations: 4*3+3 = 15 Better than the selection sort. The savings would be even bigger for larger lists that were sorted early.

Bubble sort: final analysis Not counting swapping, the best case is only n-1 operations! Worst case is still n-squared Order of complexity same as selection sort no matter what.