Algorithm Analysis CS 201 Fundamental Structures of Computer Science

Introduction Once an algorithm is given for a problem and decided to be correct, an important step is to determine how much in the way of resources (such as time or space) the algorithm will require. In this chapter, we shall discuss –How to estimate the time required for a program

Mathematical background The idea of the following definitions is to establish a relative order among functions. Given two functions, there are usually points where one function is smaller than the other function. –It does not make sense to claim f(N) < g(N) –Let us compare f(N) = 1000N and g(N) = N 2 –Thus, we will compare their relative rates of growth

Mathematical background Definition 1:

Mathematical background Definition 2:

Mathematical background Definition 3:

Examples f(N)=N 3 grows faster than g(N)=N 2, so g(N) = O(f(N)) or f(N) =  (g(N)) f(N)=N 2 and g(N)=2N 2 grow at the same rate, so f(N) = O(g(N)) and f(N) =  (g(N)) If g(N)=2N 2, g(N)=O(N 4 ), g(N)=O(N 3 ), g(N)=O(N 2 ) are all technically correct, but the last one is the best answer. Do not say T(N)=O(2N 2 ) or T(N)=O(N 2 +N). The correct form is T(N)=O(N 2 ).

Mathematical background Rule 1: Rule 2: Rule 3:

Examples Determine which of f(N)=N logN and g(N)=N 1.5 grows faster.  Determine which of logN and N 0.5 grows faster.  Determine which of log 2 N and N grows faster.  Since N grows faster than any power of a log, g(N) grows faster than f(N).

Examples Consider the problem of downloading a file over the Internet. –Setting up the connection: 3 seconds –Download speed: 1.5 Kbytes/second If a file is N kilobytes, the time to download is T(N) = N/ , i.e., T(N) = O(N). –1500K file takes 1003 seconds –750K file takes 503 seconds If the connection speed doubles, both times decrease, but downloading 1500K still takes approximately twice the time downloading 750K.

Typical growth rates FunctionName cConstant log NLogarithmic log 2 NLog-squared NLinear N log N N2N2 Quadratic N3N3 Cubic 2N2N Exponential

Plots of various algorithms

Algorithm analysis The most important resource to analyze is generally the running time –We assume that simple instructions (such as addition, comparison, and assignment) take exactly one unit time Unlike the case with real computers –For example, I/O operations take more time compared to comparison and arithmetic operators –Obviously, we do not have this assumption for fancy operations such as matrix inversion, list insertion, and sorting. –We assume infinite memory –We do not include the time required to read the input

Algorithm analysis Typically, the size of the input is the main consideration –Worst-case performance represents a guarantee for performance on any possible input –Average-case performance often reflects typical behavior –Best-case performance is often of little interest Generally, it is focused on the worst-case analysis –It provides a bound for all inputs –Average-case bounds are much more difficult to compute

Algorithm analysis Although using Big-Theta would be more precise, Big-Oh answers are typically given. Big-Oh answer is not affected by the programming language –If a program is running much more slowly than the algorithm analysis suggests, there may be an implementation inefficiency e.g., This can occur in C++ when arrays are inadvertently copied in their entirety instead of passed with references.

Algorithm analysis If two programs are expected to take similar times, probably the best way to decide which is faster to code them both up and run them! On the other hand, we would like to eliminate the bad algorithmic ideas early by algorithm analysis –Although different instructions can take different amounts of time (these would correspond to constants in Big-Oh notation), we ignore this difference in our analysis and try to find the upper bound of the algorithm

A simple example Estimate the running time of the following function int sum(int n){ int partialSum = 0; for (int i = 0; i <= n; i++) partialSum += i * i * i; return partialSum; }

General rules Rule 1 – for loops: The running time of a for loop is at most the running time of the statements inside the for loop (including tests) times the number of iterations Rule 2 – nested loops: Analyze these inside out. The total running time of a statement inside a group of nested loops is the running time of the statement multiplied by the product of the sizes of all the loops for (i = 0; i < n; i++) for (j = i; j < n; j++) k++;

General rules Rule 3 – consecutive statements: just add. for (i = 0; i < n; i++) a[i] = 0; for (i = 0; i < n; i++) for (j = 0; j < n; j++) a[i] += a[j] + i + j; Rule 4 – if/else: For the following, the running time is never more than that of the test plus the larger of that of S1 and S2 if (condition) S1 else S2

General rules If there are function calls, these must be analyzed first If there are recursive functions, be careful about their analyses. For some recursions, the analysis is trivial long factorial(int n){ if (n <= 1) return 1; return n * factorial(n-1); }

General rules Let us analyze the following recursion long fib(int n){ if (n <= 1) return 1; return fib(n-1) + fib(n-2); } By induction, it is possible to show that So the running time grows exponentially. By using a for loop, the running time can be reduced substantially

Sequential search index = -1; for (i = 0; i < N; i++) if (A[i] == value){ index = i; break; } Worst-case: It is the last element or it is not found –N iterations  O(N) Best-case: It is the first element –1 iteration  O(1) Average case: We may have N different cases –(N + 3) / 2 iterations  O(N)

Searching a value in a sorted array using binary search int binarySearch(int *A, int N, int value){ int low = 0, high = N; while (low <= high){ int mid = (low + high) / 2; if (a[mid] < value) low = mid + 1; else if (a[mid] > value) high = mid - 1; else return mid; } return -1; }

Iterative solution of raising an integer to a power long pow1(long x, long n){ long result = 1; for (int i = 1; i <= n; i++) result = result * x; return result; }

Recursive solution of raising an integer to a power long pow2(long x, long n){ if (n == 0) return 1; if (n == 1) return x; if (n % 2 == 0) return pow2(x*x,n/2); else return pow2(x*x,n/2) * x; }