CSC 430: Basics of Algorithm Runtime Analysis 1

What is Runtime Analysis? Fundamentally, it is the process of counting the number of “steps” it takes for an algorithm to terminate “Step” = primitive computer operation – Arithmetic – Read/Write memory – Make a comparison – … The number of steps needed depends on: – The fundamental algorithmic structure – The data structures you use 2

3 Historical Reasons to Care about Runtime Analysis Charles Babbage (1864) As soon as an Analytic Engine exists, it will necessarily guide the future course of the science. Whenever any result is sought by its aid, the question will arise - By what course of calculation can these results be arrived at by the machine in the shortest time? - Charles Babbage Analytic Engine (schematic) Earliest computational problems were physical: What’s the fastest way to reconfigure a train using two turntables? When first invented, computers were rare (and quite slow) Had to schedule use-time This held true until about 1990 Apple rounded-rectangle story

Why YOU Should Care It’s no longer about speed A prediction: – By 2020, all new personal computers will contain more than 32 micro- processors What is it about? – Google – Wikipedia – Facebook – YouTube Licensed under Creative Commons: Moore%27s_Law_-_2008.svg 4

Other Types of Algorithm Analysis Algorithm space analysis Approximate-solution analysis Parallel algorithm analysis A PROBLEM’S run-time analysis Complexity class analysis 5

6 Runtime Motivates Design Easiest Algorithm to implement for any problem: Brute force. – check every possible solution. – Typically takes 2 N time or worse for inputs of size N. – Unacceptable in practice. Usually, through insight, we can do much better What do we mean by “better”?

7 Worst-Case Analysis Best case running time: – Rarely used; often superfluous Average case running time: – Obtain bound on running time of algorithm on random input as a function of input size N. – Hard (or impossible) to accurately model real instances by random distributions. – Acceptable in certain, easily-justified scenarios Worst case running time: – Obtain bound on largest possible running time – Generally captures efficiency in practice. – Pessimistic view, but hard to find effective alternative.

Polynomial Time Desirable scaling property: When the input size increases (e.g., if it doubles), the algorithm should only slow down by some constant factor C. Poly-time algorithms satisfy the above scaling property. A function is polynomial if there exist constants c > 0 and d > 0 such that on every input of size N, its running time is bounded by c N d steps. 8

9 Defining Efficiency: Worst-Case Polynomial-Time Def. An algorithm is efficient if its worst-case running time is polynomial or better. Not always true, but works in practice: – 6.02   N 20 is technically poly-time – Breaking through the exponential barrier of brute force typically exposes some crucial structure of the problem. – Most poly-time algorithms that people develop have low constants and low exponents (e.g., N 1 N 2 N 3 ). Exceptions. – Sometimes the problem size demands greater efficiency than poly-time – Some poly-time algorithms do have high constants and/or exponents, and are useless in practice. – Some exponential-time (or worse) algorithms are widely used because the worst-case instances seem to be rare. simplex method Unix grep

10 Ignorance is NOT Bliss Lesson: Know the running time of your algorithms!

2.2 Asymptotic Order of Growth 11

What is Asymptotic Analysis Instead of giving precise runtime, we’ll show that algorithms are bounded within a constant factor of common functions Three types of bounds – Upper Bounds (never greater) – Lower Bounds (never less) – Tight Bounds (Upper and Lower are same) 12

13 Growth Notation Upper bounds (Big-Oh): – T(n)  O(f(n)) if there exist constants c > 0 and n 0  0 such that for all n  n 0 we have T(n)  c · f(n). Lower bounds (Omega): – T(n)   (f(n)) if there exist constants c > 0 and n 0  0 such that for all n  n 0 we have T(n)  c · f(n). Tight bounds (Theta): – T(n)   (f(n)) if T(n) is both O(f(n)) and  (f(n)). Frequently abused notation: T(n) = O(f(n)). – Asymmetric: f(n) = 5n 3 ; g(n) = 3n 2 f(n) = O(n 3 ) = g(n) but f(n)  g(n). Better notation: T(n)  O(f(n)). Frequently abused notation: T(n) = O(f(n)). – Asymmetric: f(n) = 5n 3 ; g(n) = 3n 2 f(n) = O(n 3 ) = g(n) but f(n)  g(n). Better notation: T(n)  O(f(n)).

Use-Cases Big-Oh: – “The algorithm’s runtime is O(N lgN), a potential improvement over prior approaches, which were O(N 2 ).” Omega: – “Their algorithm’s runtime is  (N 2 ), which is worse than our algorithm’s proven O(N 1.5 ) runtime.” Theta: – “Our original algorithm was  (N lgN) and O(N 2 ). With modifications, we were able to improve the runtime bounds to  (N 1.5 )” 14

15 Properties Transitivity. – If f  O(g) and g  O(h) then f  O(h). – If f   (g) and g   (h) then f   (h). – If f   (g) and g   (h) then f   (h). Additivity. – If f  O(h) and g  O(h) then f + g  O(h). – If f   (h) and g   (h) then f + g   (h). – If f   (h) and g  O(h) then f + g   (h). – These generalize to any number of terms “Symmetry”: – If f   (h) then h   (f) – If f  O(h) then h   (f) – If f   (h) then h  O(f)

16 Asymptotic Bounds for Some Common Functions Polynomials. a 0 + a 1 n + … + a d n d is  (n d ) if a d > 0. Polynomial time. Running time is O(n d ) for some constant d independent of the input size n. Logarithms. O(log a n) = O(log b n) for any constants a, b > 0. Logarithms. For every x > 0, log n = O(n x ). Exponentials. For every r > 1 and every d > 0, n d = O(r n ). every exponential grows faster than every polynomial can avoid specifying the base log grows slower than every polynomial

Proving Bounds Use Transitivity: – If an algorithm is at least as fast as another, then it has the same upper bound Use Additivity: – f(n) = n 3 + n 2 + n – f(n)   (n 3 ) Be direct: – Start with the definition and provide the parts it requires Take the logarithm of both functions Use limits: Keep in mind that sometimes functions can’t be bounded. (But I won’t assign you any like that!) 17

Practice 18

2.4 A Survey of Common Running Times 19

20 Linear Time: O(n) Linear time. Running time is at most a constant factor times the size of the input. Computing the maximum. Compute maximum of n numbers a 1, …, a n. def max(L): max = L[0] for i in L: if i > max: max = i return max

21 Linear Time: O(n) Claim. Merging two lists of size n takes O(n) time. Pf. After each comparison, the length of output list increases by 1. #the merge algorithm combines two #sorted lists into one larger sorted #list def merge(A,B): i = j = 0 M = [] while i < len(A) or j < len(B): if A[i]  B[j]: M.append(A[i]) i+=1 else: M.append(B[j]) j+=1 if i == len(A): return M + B[j:] return M + A[i:]

22 O(n log n) Time O(n log n) time. Arises in divide-and-conquer algorithms. Sorting. Mergesort and heapsort are sorting algorithms that perform O(n log n) comparisons. Largest empty interval. Given n time-stamps x 1, …, x n on which copies of a file arrive at a server, what is largest interval of time when no copies of the file arrive? O(n log n) solution. Sort the time-stamps. Scan the sorted list in order, identifying the maximum gap between successive time-stamps.

23 Quadratic Time: O(n 2 ) Quadratic time. Enumerate all pairs of elements. Closest pair of points. Given a list of n points in the plane (x 1, y 1 ), …, (x n, y n ), find the pair that is closest. Remark.  (n 2 ) seems inevitable, but this is just an illusion. def closest_pair(L): x1 = L[0] x2 = L[1] #no need to compute the sqrt for the distance min = (x1[0] – x2[0])**2 + (x1[1] – x2[1])**2 for i in range(len(L)): for j in range(i+1,len(L)) : x1,x2 = L[i],L[j] d = (x1[0] – x2[0])**2 + (x1[1] – x2[1])**2 if d < min : min  d return min see chapter 5

24 Cubic Time: O(n 3 ) Cubic time. Enumerate all triples of elements. Set disjointness. Given n sets S 1, …, S n each of which is a subset of 1, 2, …, n, is there some pair of these which are disjoint? def find_disjoint_sets(S): for i in range(len(S)): for j in range(i+1,len(S)): for p in S[i]: if p in S[j]: break; else: #no p in S[i] belongs to S[j] return S[i],S[j] #return tuple if found return None #return nothing if all sets overlap

25 Polynomial Time: O(n k ) Time Independent set of size k. Given a graph, are there k nodes such that no two are joined by an edge? O(n k ) solution. Enumerate all subsets of k nodes. – Check whether S is an independent set = O(k 2 ). – Number of k element subsets = – O(k 2 n k / k!) = O(n k ). def independent_k(V,E,k): #subset_k,independent omitted for space for sub in subset_k(V,k): if independent(sub,E): return sub return None poly-time for k=17, but not practical k is a constant

26 Exponential Time Independent set. Given a graph, what is maximum size of an independent set? O(2 n ) solution. Enumerate all subsets. def max_independent(V,E): for k in len(V): if not independent_k(V,E,k): return k-1