Estimating Algorithm Performance How to determine how much time an algorithm A uses to solve problem X? Depends on input; use input length as parameter. Determine function f(n) representing cost. Code it and use timer running on many inputs Analyze steps of algorithm, estimating amount of work each step takes (Algorithm Analysis). Coding often impractical at early design stages. Algorithm Analysis: determining how long an algorithm will take to solve a problem USING A MODEL. TCSS 342 Intro Alg. Anal. v 1.0

RAM Model for Algorithm Analysis Typically use a simple model for basic operation costs. RAM (Random Access Machine) model RAM model has all the basic operations: +, -, *, / , =, comparisons. fixed sized integers (e.g., 32-bit) infinite memory All (basic) operations take exactly one time unit to execute. analysis= determining run-time efficiency. model = estimate, not meant to represent everything in real-world TCSS 342 Intro Alg. Anal. v 1.0

Critique of the model Weaknesses: Strengths: not all operations take the same amount of time in a real machine does not account for page faults, disk accesses, etc. Strengths: simple easier to prove things about the model than the real machine can estimate algorithm behavior on any hardware/software model = approximation of real world. can predict run-time of algorithm on machine. TCSS 342 Intro Alg. Anal. v 1.0

Why models? model real world Idea: (Useful) statements using the model translate into useful statement about real computers TCSS 342 Intro Alg. Anal. v 1.0

Relative rates of growth Distinguish between fast- and slow-growing functions. 100n2 + 1000 n3 + 2n2 Motivation: we usually care only about algorithm performance when there are large number of inputs. We usually don’t care about small changes in run-time performance. (inaccuracy of estimates make small changes less relevant). Consider algorithms with slow growth rate better than those with fast growth rates. TCSS 342 Intro Alg. Anal. v 1.0

Big-Oh notation Defn: T(N) = O(f(N)) if there are positive constants c and n0 such that T(N)  c f(N) for all N  n0 . Idea: We are concerned with how the function grows when N is large. We are not picky about constant factors: coarse distinctions among functions. Lingo: “T(N) grows no faster than f(N).” Important (not in Lewis & Chase book). Write on board. (for next slide). TCSS 342 Intro Alg. Anal. v 1.0

Examples n = O(2n) ? 2n = O(n) ? n = O(n2) ? n2 = O(n) ? n = O(1) ? 10 log n = O(n) ? 214n + 34 = O(2n2 + 8n) ? TCSS 342 Intro Alg. Anal. v 1.0

Preferred big-Oh usage If f(N) = 5N, then f(N) = O(N5) f(N) = O(N3) f(N) = O(N)  preferred f(N) = O(N log N) Ignore constant factors and low order terms. T(N) = O(N), not T(N) = O(5N). T(N) = O(N3), not T(N) = O(N3 + N2 + N log N). Bad style: f(N)  O(g(N)). Wrong: f(N)  O(g(N)). TCSS 342 Intro Alg. Anal. v 1.0

figure 1.11 asymptotic complexity of selected functions TCSS 342 Intro Alg. Anal. v 1.0

figure 1.12 Increase in problem size with a ten-fold increase in processor speed Book has errata, including on this figure. 2^n = exponential complexity. 2n TCSS 342 Intro Alg. Anal. v 1.0

Big Omega, Theta Defn: T(N) = (g(N)) if there are positive constants c and n0 such that T(N)  c g(N) for all N  n0 . Lingo: “T(N) grows no slower than g(N).” Defn: T(N) = (h(N)) if and only if T(N) = O(h(N)) and T(N) = (h(N)). Big-Oh, Omega, and Theta establish a relative order among all functions of N. TCSS 342 Intro Alg. Anal. v 1.0

Intuition, little-Oh Defn: T(N) = o(p(N)) if T(N) = O(p(N)) and T(N)  (p(N)). notation intuition O (Big-Oh)   (Big-Omega)   (Theta) = o (little-Oh) < TCSS 342 Intro Alg. Anal. v 1.0

More about Asymptotics Fact: If f(N) = O(g(N)), then g(N) = (f(N)). Proof: Suppose f(N) = O(g(N)). Then there exist constants c and n0 such that f(N)  c g(N) for all N  n0. Then g(N)  f(N) for all N  n0, and so g(N) = (f(N)). TCSS 342 Intro Alg. Anal. v 1.0

More terminology Suppose T(N) = O(f(N)). f(N) is an upper bound on T(N). T(N) grows no faster than f(N). Suppose T(N) = (g(N)). g(N) is a lower bound on T(N). T(N) grows at least as fast as g(N). If T(N) = o(h(N)), then we say that T(N) grows strictly slower than h(N). TCSS 342 Intro Alg. Anal. v 1.0

Facts about Big-Oh If T1(N) = O(f(N)) and T2(N) = O(g(N)), then T1(N) + T2(N) = O(f(N) + g(N)). T1(N) * T2(N) = O(f(N) * g(N)). If T(N) is a polynomial of degree k, then T(N) = (Nk). log k N = O(N), for any constant k. TCSS 342 Intro Alg. Anal. v 1.0

Techniques Algebra ex. f(N) = N / log N, g(N) = log N. same as asking which grows faster, N or log 2 N. Evaluate . limit is Big-Oh relation f(N) = o(g(N)) c  0 f(N) = (g(N))  g(N) = o(f(N)) no limit no relation TCSS 342 Intro Alg. Anal. v 1.0

Techniques, cont’d L’Hôpital’s rule: If and , then . example: f(N) = N, g(N) = log N. Use L’Hôpital’s rule. f’(N) = 1, g’(N) = 1/N.  g(N) = o(f(N)). TCSS 342 Intro Alg. Anal. v 1.0

Kinds of Algorithm Analysis Express the running time as f(N), where N is the size of the input. Worst case: your enemy gets to pick the input. Average case: need to assume a probability distribution on the inputs. Amortized: your enemy gets to pick the inputs/operations, but you only have to guarantee speed over a large number of operations. However, even with input size N, cost of an algorithm could vary on different input. TCSS 342 Intro Alg. Anal. v 1.0

Search avg. case, if found: n/2 accesses // return index if x is found // in array a; // or NOT_FOUND if not found public int SequentialSearch( int[] a, int x) { int i; for (i = 0; i < a.length; i++) { if (a[i] == x) return i; } return NOT_FOUND; worst case: O(n) avg. case, if found: n/2 accesses avg. case, if not found: n accesses  constant work  in loop TCSS 342 Intro Alg. Anal. v 1.0

Binary Search  constant work  in loop // Given a sorted array a and // a key x, return index if x is found // in array a // or NOT_FOUND if not found public int BinarySearch(int[] a, int x){ int low = 0; int high = a.length – 1; while (low <= high) { int mid = (low + high) / 2; if (a[mid] < x) { low = mid + 1; } else if (a[mid] > x) { high = mid – 1; } else { return mid; } return NOT_FOUND;  constant work  in loop TCSS 342 Intro Alg. Anal. v 1.0

Analysis Intuitive argument about the running time: number of elements to consider (high – low + 1) is halved each iteration high – low + 1 = n at start The algorithm iterates at most log2 n.  O(log n) steps to execute. … mid n-1 TCSS 342 Intro Alg. Anal. v 1.0