Java Methods Big-O Analysis of Algorithms Object-Oriented Programming and Data Structures 2nd AP edition  with GridWorld Maria Litvin ● Gary Litvin Theoretical details in this chapter are for mathematically-inclined students. This is not AP CS material. Big-O Analysis of Algorithms Copyright © 2011 by Maria Litvin, Gary Litvin, and Skylight Publishing. All rights reserved.

Objectives: Learn the big-O definition and notation Review big-O for some common tasks and algorithms studied earlier Review several popular sorting algorithms and their average, best, and worst case big-O This chapter also provides the necessary background for discussing data structures in the subsequent chapters.

Evaluation of Algorithms Practice: benchmarks t n Theory: asymptotic analysis, big-O The first impression may be that benchmarks leave no room for theory. This is not so.

Analysis of Algorithms — Assumptions Abstract computer model with “unlimited” RAM Unlimited range for integers Basic operations on integers (addition, multiplication, etc.) take fixed time regardless of the values of the operands This model of an abstract machine is not realistic, but it is useful. There is another model based on operations on individual bits, which is more realistic, but harder to use.

Big-O Analysis Studies space and time requirements in terms of the “size” of the task The concept of “size” is somewhat informal here: The size of a list or another collection The dimensions of a 2-D array The argument of a function (for example, factorial(n) or fibonacci(n)) The number of objects involved (for example, n disks in the Tower of Hanoi puzzle) In the model based on bits, the size of the task is simply the number of bits necessary to represent the input data.

Big-O Assumptions Here our main concern is time Ignores a constant factor measures the time in terms of the number of some abstract steps Applies to large n, studies asymptotic behavior So the time is measured in abstract units or steps, not seconds, minutes, or hours.

Example 1: Sequential Search for (int i = 0; i < n; i++) if (a[ i ] == target) break; n/2 iterations on average Small when n   t A is a constant. t(n) = An The average time is approximately A·n — linear time (for large n) n

Example 2: Binary Search n The Number of Comparisons 1 3 2 7 3 ... ... n log2(n) t tinit and titer here are different from tinit and titer of Sequential Search. The average time is approximately C·log2 n — logarithmic time (for large n) t(n) = C log n n

Sequential vs. Binary Search The logarithmic curve increases without bound when n increases But... log grows much more slowly than any linear function. No matter what the constants are, the linear “curve” eventually overtakes the logarithmic curve. t For large enough n, 0.0000001 * n > 1000000 * log n. n

Order of Growth For large enough n, a binary search on the slowest computer will finish earlier than a sequential search on the fastest computer. It makes sense to talk about the order of growth of a function. Constant factors are ignored. The theory is somewhat abstract. In the real world, you have to look at the details. For example, if the values that are at the beginning of the list are much more frequent search targets than the values father from the beginning, then Sequential Search will be better than Binary Search. For example, the order of growth of any linear function exceeds the order of growth of any logarithmic function

Big-O Given two positive functions t(n) and g(n), we say that t(n) = O(g(n)) if there exist a positive constant A and some number N such that t(n)  A g(n) for all n > N. This is a formal definition. In practice we use the tightest possible upper bound. In other words, we are especially interested in the case where f(n)/g(n) approaches a constant when n increases; then we say f(n) = O(g(n)).

Big-O (cont’d) Big-O means that asymptotically t(n) grows not faster than g(n). In analysis of algorithms, it is common practice to use big-O for the tightest possible upper bound. Example: Sequential Search is O(n) Binary Search is O(log n) like “t(n)  g(n)” like “t(n) = g(n)” Order of growth is often stated in terms of big-O for the following functions: 1, n, n2, n3, log n, n log n, an.

Big-O (cont’d) For a log, the base is not important: from the Change of Base Theorem where If b > 1, then K > 0.

Big-O (cont’d) For a polynomial the big-O is determined by the degree of the polynomial: In general, big-O for a sum of functions is determined by the function with the highest big-O.

Big-O (cont’d) For “triangular” nested loops The total number of iterations is: for (int i = 0; i < n; i++) for (int j = i; j < n; j++) ... n(n-1)/2 is a quadratic polynomial of n.

Big-O Examples O(1) — constant time Finding a median value in a sorted array Calculating 1 + 2 + ... + n using the formula for the sum of an arithmetic sequence Push and pop operations in an efficiently implemented stack; add and remove operations in a queue (Chapter 22) Finding a key in a lookup table or a sparsely populated hash table (Chapter 25) Theoretically, O(1), constant time, is the best, but in practice the “constant” can exceed O(log n) time when n is not very large.

Big-O Examples O(log n) — logarithmic time Binary search in a sorted list of n elements Finding a target value in a binary search tree with n nodes (Chapter 24) add and remove operations in a priority queue, implemented as a heap, with n nodes (Chapter 26) In practice, logarithmic time can be really close to constant time.

Big-O Examples O(n) — linear time Traversing a list with n elements, (for example, finding max or min) Calculating n factorial or the n-th Fibonacci number iteratively Traversing a binary tree with n nodes (Chapter 24) A single loop with a constant increment runs in linear time.

Big-O Examples O(n log n) — “n log n” time Mergesort and Quicksort of n elements Heapsort (Chapter 26) Discovering an n log n algorithm as opposed to an n2 algorithm is considered a breakthrough.

Big-O Examples O(n2) — quadratic time More simplistic sorting algorithms, such as Selection Sort of n elements Traversing an n by n 2-D array Finding duplicates in an unsorted list of n elements (implemented with a nested loop) A loop nested inside another loop, each running in linear time.

Big-O Examples O(an) (a > 1) — exponential time Recursive Fibonacci implementation (a  3/2; see Chapter 23) Solving the Tower of Hanoi puzzle (a = 2; see Section 23.5) Generating all possible permutations of n symbols Exponential time quickly overwhelms even the fastest computer.

Big-O Summary O(1)  <  O(log n)  <  O(n)  <  O(n log n)  <  O(n2)  <  O(n3)  <  O(an) Polynomial time is considered manageable. But algorithms that take exponential time are not very practical for larger n. Tower of Hanoi, for example, takes a few milliseconds for a small number of disks (n = 10 or n = 20), but it takes forever (billions of years) for 64 disks.

Sorting O(n2) O(n log n) By counting Selection Sort Insertion Sort Mergesort Quicksort Heapsort If you are limited to “honest” comparison sorting, any sorting algorithm, in its worst case scenario, takes at least O(nlog n) steps These sorting algorithms are based on comparing values. Other sorting algorithms may be based on analyzing properties of values. For example, if you know that your list only holds values from 0 to 9, you can just count how many of each in one sweep (O(n)) and construct the sorted list. The Radix Sort algorithm can sort words in n*d time where d is the maximum number of letters in the word. It first sorts the words by the last letter, into separate “buckets,” then collects the buckets together. The same is repeated for the next to last letter, and so on., ending up with a sorted list.

Sorting by Counting — O(n2) For each value in the list, count how many values are smaller (or equal with a smaller index). Place that value into the position indicated by the count. Always n2 comparisons for (int i = 0; i < n; i++) { count = 0; for (int j = 0; j < n; j++) if (a [ j ] < a [ i ] | | (a [ j ] == a [ i ] && j < i ) ) count++; b [count ] = a [ i ]; } This method is mentioned just for completeness. Even Selection Sort is better.

Selection Sort — O(n2) Find the largest value and swap it with the last element. Decrement the “logical” size of the list and repeat while the size exceeds 1. while (n > 1) { iMax = 0; for (int i = 1; i < n; i++) if (a [ i ] > a [ iMax ]) iMax = i; swap (a, iMax, n-1); n--; } Always n(n-1)/2 comparisons Selection Sort always takes the same time regardless of the values.

Insertion Sort — O(n2) Keep the beginning of the list sorted. Insert the next value in order into the sorted. beginning segment. O(n2) comparisons on average, but only O(n) when the list is already sorted for (int i = 1; i < n; i++) { temp = a [ i ]; for (int j = i; j > 0 && a [ j-1 ] > temp; j--) a [ j ] = a [ j-1 ]; a [ j ] = temp; } Properly implemented Insertion Sort runs in O(n) time when the array is already sorted.

Mergesort — O(n log n) Split the list down the middle into two lists of approximately equal length. Sort (recursively) each half. Merge the two sorted halves into one sorted list. private void sort (double a [ ], int from, int to) { if (to == from) return; int middle = (from + to) / 2; sort(a, from, middle); sort(a, middle + 1, to); merge (a, from, middle, middle + 1, to); } You can add an additional check: check whether the merge is necessary. Then Mergesort will take only O(n) time for an array that is already sorted. Each merge takes O(n) comparisons

Quicksort — O(n log n) Choose a “pivot” element. Partition the array so that all the values to the left of the pivot are smaller than or equal to it, and all the elements to the right of the pivot are greater than or equal to it. Sort (recursively) the left-of-the-pivot and right-of-the-pivot pieces. Proceed from both ends of the array as far as possible; when both values are on the wrong side of the pivot, swap them Partitioning is a rather elegant algorithm, which runs in O(n) time. O(n log n) comparisons, on average, if partitioning splits the array evenly most of the time

Heapsort — O(n log n) An elegant algorithm based on heaps (Chapter 26) Proposed in 1964 by J. Williams.

Sorting Summary For most algorithms the running time depends on data, so it makes sense to talk about the best case, average, and worst case.

Review: What assumptions are made for big-O analysis? What O(...) expressions are commonly used to measure the big-O performance of algorithms? Is it true that n(n-1)/2 = O(n2)? Give an example of an algorithm that runs in O(log n) time. Give an example of an algorithm (apart from sorting) that runs in O(n2) time. What assumptions are made for big-O analysis? A constant factor is ignored; the size of the task n is large. What O(...) expressions are commonly used to measure the big-O performance of algorithms? 1, n, n2, log n, n log n, an. Is it true that n(n-1)/2 = O(n2)? Yes. Give an example of an algorithm that runs in O(log n) time. Binary Search Give an example of an algorithm (apart from sorting) that runs in O(n2) time. Finding duplicates in an unsorted list.

Review (cont’d): Name three O(n log n) sorting algorithms. How many comparisons are needed in sorting by counting? In Selection Sort? What is the main idea of Insertion Sort? What is the best case for Insertion Sort? What is the main idea of Quicksort? What is the worst case for Quicksort? Name three O(n log n) sorting algorithms. Mergesort, Quicksort, Heapsort. How many comparisons are needed in sorting by counting? In Selection Sort? Counting — n2; Selection Sort — n(n - 1)/2. What is the main idea of Insertion Sort? Keep the beginning of the list sorted, insert the next value in order. What is the best case for Insertion Sort? If properly implemented, when the array is already sorted. What is the main idea of Quicksort? Partition the array into two parts, smaller than the pivot and larger than the pivot; then sort (recursively) each part. What is the worst case for Quicksort? It can degenerate into an O(n2) performance if selection of the pivot is consistently bad (for example, the array is already sorted and we always choose the first element as the pivot).