Dale Roberts Department of Computer and Information Science, School of Science, IUPUI Dale Roberts, Lecturer Computer Science, IUPUI CSCI 240 Analysis of Algorithms

Dale Roberts Characteristics of Algorithms Algorithms are precise. Each step has a clearly defined meaning; “Deterministic” Algorithms are effective. The task is always done as required; “Correct” Algorithms have a finite number of steps; Algorithms must terminate. How do you know?

Dale Roberts Example: sorting numbered cards Given some numbered cards. Our aim is to put them into nondecreasing order.

Dale Roberts Example: sorting numbered cards

Dale Roberts Example: sorting numbered cards

Dale Roberts Example: sorting numbered cards

Dale Roberts Example: sorting numbered cards

Dale Roberts Example: sorting numbered cards

Dale Roberts Example: sorting numbered cards

Dale Roberts Example: sorting numbered cards

Dale Roberts Expressing computer algorithms It should be expressed in a language more precise, less ambiguous, and more compact than a “natural language” such as English; Algorithms are usually written in a pseudocode and later translated to a real programming language. Sometimes algorithms are “flowcharted” using HIPO (Hierarchical Input, Processing, and Output) symbols.

Dale Roberts Insertion Sort in Pseudocode B[1] = A[1] B[1] = A[1] for j = 2 to n for j = 2 to n { i = j - 1 i = j - 1 while 0 < i and A[j] < B[i] while 0 < i and A[j] < B[i] i = i - 1 i = i - 1 for k = j downto i + 2 for k = j downto i + 2 B[k] = B[k-1] B[k] = B[k-1] B[i+1] = A[j] B[i+1] = A[j] } Insertion of jth card Finding the place to insert A[j] Shifting a part of array B Inserting A[j] A is an array of numbers of length n, B is an empty array

Dale Roberts Choosing an Analysis Metric Estimate the running time of algorithms; = F(Problem Size) = F(Input Size) = number of primitive operations used (add, multiply, compare etc)

Dale Roberts Analysis for Insertion Sort Insertion-Sort(A)CostTimes (Iterations) 1 B[1] = A[1] c1 2 for j = 2 to n { c2 3 i = j - 1 c3 4 while 0 < i and A[j] < B[i] c4 5 i = i - 1 c5 6 for k = j downto i + 2 c6 7 B[k] = B[k-1] c7 8 B[i+1] = A[j] } c8 1 n - 1 n

Dale Roberts Insertion Sort Analysis (cont.) Best Case: Array already sorted, Array already sorted, for all j. (Linear in n)

Dale Roberts Insertion Sort Analysis (cont.) Worst Case: Array in reverse order, Array in reverse order, Note that We are usually interested in the worst-case running time We are usually interested in the worst-case running time for all j.

Dale Roberts Average Case and Worse Case Defined Most important resource to analyze is running time; other factors are algorithm used and input to the algorithm Parameter N, usually referring to number of data items processed, affects running time most significantly. N may be degree of polynomial, size of file to be sorted or searched, number of nodes in a graph, etc. N may be degree of polynomial, size of file to be sorted or searched, number of nodes in a graph, etc. Worst case is amount of time program would take with worst possible input configuration worst case is bound for input and easier to find; usually the metric chosen Average case is amount of time a program is expected to take using "typical" input data definition of "average" can affect results average case is much harder to compute

Dale Roberts Average Case and Worse Case An algorithm may run faster on certain data sets than on others. Finding the average case can be very difficult, so typically algorithms are measured by the worst-case time complexity. Also, in certain application domains (e.g., air traffic control, surgery, IP lookup) knowing the worst-case time complexity is of crucial importance.

Dale Roberts Comparison of Growth Rates Functions in Increasing Order

Dale Roberts Comparison of Growth Rates (cont)

Dale Roberts Comparison of Growth Rates (cont)

Dale Roberts Comparison of Growth Rates (cont)

Dale Roberts Comparison of Growth Rates (cont)

Dale Roberts Comparison of Growth Rates (cont)

Dale Roberts Asymptotic Analysis Ignoring constants in T(n) Analyzing T(n) as n "gets large" Example: The big-oh ( O ) Notation

Dale Roberts 3 major notations O(g(n)), Big-Oh of g of n, the Asymptotic Upper Bound;  (g(n)), Theta of g of n, the Asymptotic Tight Bound; and  (g(n)), Omega of g of n, the Asymptotic Lower Bound.

Dale Roberts Big-Oh Defined The O symbol was introduced in 1927 to indicate relative growth of two functions based on asymptotic behavior of the functions now used to classify functions and families of functions T(n) = O(f(n)) if there are constants c and n0 such that T(n) < c*f(n) when n  n0 c*f(n) T(n) n0n0 n c*f(n) is an upper bound for T(n)

Dale Roberts Big-O Describes an upper bound for the running time of an algorithm Upper bounds for Insertion Sort running times: worst case: O(n 2 ) T(n) = c 1 *n 2 + c 2 *n + c 3 best case: O(n) T(n) = c 1 *n + c 2 Time Complexity

Dale Roberts Big-O Notation We say Insertion Sort’s run time is O(n 2 ) Properly we should say run time is in O(n 2 ) Read O as “Big-O” (you’ll also hear it as “order”) In general a function f(n) is O(g(n)) if there exist positive constants c and n 0 such that f(n)  c  g(n) for all n  n 0 e.g. if f(n)=1000n and g(n)=n 2, n 0 > 1000 and c = 1 then f(n 0 ) 1000 and c = 1 then f(n 0 ) < 1.g(n 0 ) and we say that f(n) = O(g(n)) The O notation indicates 'bounded above by a constant multiple of.'

Dale Roberts Big-Oh Properties Fastest growing function dominates a sum O(f(n)+g(n)) is O(max{f(n), g(n)}) Product of upper bounds is upper bound for the product If f is O(g) and h is O(r)  then fh is O(gr) f is O(g) is transitive If f is O(g) and g is O(h) then f is O(h) Hierarchy of functions O(1), O(logn), O(n 1/2 ), O(nlogn), O(n 2 ), O(2 n ), O(n!)

Dale Roberts Some Big-Oh’s are not reasonable Polynomial Time algorithms An algorithm is said to be polynomial if it is O( n c ), c >1 Polynomial algorithms are said to be reasonable They solve problems in reasonable times! Constants or low-order terms are ignored e.g. if f(n) = 2n 2 then f(n) = O(n 2 ) Exponential Time algorithms An algorithm is said to be exponential if it is O( r n ), r > 1 Exponential algorithms are said to be unreasonable

Dale Roberts Can we justify Big O notation? Big O notation is a huge simplification; can we justify it? It only makes sense for large problem sizes For sufficiently large problem sizes, the highest-order term swamps all the rest! Consider R = x 2 + 3x + 5 as x varies: x = 0 x2 = 0 3x = 10 5 = 5 R = 5 x = 10 x2 = 100 3x = 30 5 = 5 R = 135 x = 100 x2 = x = = 5 R = 10,305 x = 1000 x2 = x = = 5 R = 1,003,005 x = 10,000 R = 100,030,005 x = 100,000 R = 10,000,300,005

Dale Roberts Classifying Algorithms based on Big-Oh A function f(n) is said to be of at most logarithmic growth if f(n) = O(log n) A function f(n) is said to be of at most quadratic growth if f(n) = O(n 2 ) A function f(n) is said to be of at most polynomial growth if f(n) = O(n k ), for some natural number k > 1 A function f(n) is said to be of at most exponential growth if there is a constant c, such that f(n) = O(c n ), and c > 1 A function f(n) is said to be of at most factorial growth if f(n) = O(n!). A function f(n) is said to have constant running time if the size of the input n has no effect on the running time of the algorithm (e.g., assignment of a value to a variable). The equation for this algorithm is f(n) = c Other logarithmic classifications: f(n) = O(n log n) f(n) = O(log log n) f(n) = O(log log n)

Dale Roberts Rules for Calculating Big-Oh Base of Logs ignored log a n = O(log b n) Power inside logs ignored log(n 2 ) = O(log n) Base and powers in exponents not ignored 3 n is not O(2 n ) 2 a (n ) is not O(a n ) If T(x) is a polynomial of degree n, then T(x) = O(x n )

Dale Roberts Big-Oh Examples 1. 2n 3 + 3n 2 + n = 2n 3 + 3n 2 + O(n) = 2n 3 + O( n 2 + n) = 2n 3 + O( n 2 ) = 2n 3 + O( n 2 + n) = 2n 3 + O( n 2 ) = O(n 3 ) = O(n 4 ) = O(n 3 ) = O(n 4 ) 2. 2n 3 + 3n 2 + n = 2n 3 + 3n 2 + O(n) = 2n 3 + O  (n 2 + n) = 2n 3 + O  (n 2 + n) = 2n 3 + O  (n 2 ) = O  (n 3 ) = 2n 3 + O  (n 2 ) = O  (n 3 )

Dale Roberts Big-Oh Examples (cont.) 3. Suppose a program P is O(n 3 ), and a program Q is O(3 n ), and that currently both can solve problems of size 50 in 1 hour. If the programs are run on another system that executes exactly 729 times as fast as the original system, what size problems will they be able to solve?

Dale Roberts Big-Oh Examples (cont) n 3 = 50 3  n = 3 50  729 n = n = log 3 (729  3 50 ) n = n = log 3 (729) + log n = 50  9 n = 6 + log n = 50  9 = 450 n = = 56 Improvement: problem size increased by 9 times for n 3 algorithm but only a slight improvement in problem size (+6) for exponential algorithm.

Dale Roberts Big-Theta and Omega Defined If f =  (g), then f is at least as big as g (or g is a lower bound for f) e.g. f(n) = n 3 and g(n) = n 2 e.g. f(n) = n 3 and g(n) = n 2 If f =  (g), f=O(g) and f =  (g) (or g is both an upper and lower bound. It is a “tight” fit) e.g. f(n) = n 3 + n 2 and g(n) = n 3 e.g. f(n) = n 3 + n 2 and g(n) = n 3

Dale Roberts Statement Analysis – Critical Section When analyzing an algorithm, we do not care about the behavior of each statement We focus our analysis on the part of the algorithm where the greatest amount of its time is spent A critical section has the following characteristics: It is an operation central to the functioning of the algorithm, and its behavior typifies the overall behavior of the algorithm It is contained inside the most deeply nested loops of the algorithm and is executed as often as any other section of the algorithm.

Dale Roberts Statement Analysis – Critical Section (cont) The critical section can be said to be at the "heart" of the algorithm We can characterize the overall efficiency of an algorithm by counting how many times this critical section is executed as a function of the problem size The critical section dominates the completion time If an algorithm is divided into two parts, the first taking O(f(n)) followed by a second that takes O(g(n)), then the overall complexity of the algorithm is O(max[f(n), g(n)]). The slowest and most time-consuming section determines the overall efficiency of the algorithm.

Dale Roberts Statement Analysis - Sequence Consecutive statements Maximum statement is the one counted e.g. a fragment with single for-loop followed by double for- loop is O(n 2 ). Block #1 Block #2 t1t1 t2t2 t 1 +t 2 = max(t 1,t 2 )

Dale Roberts Statement Analysis - If If/Else: if cond then S1else S2; Block #1Block #2 t1t1 t2t2 Max(t 1,t 2 )

Dale Roberts Statement Analysis - For For Loops Running time of a for-loop is at most the running time of the statements inside the for- loop times number of iterations for (i = sum = 0; i < n; i++) for (i = sum = 0; i < n; i++) sum += a[i]; sum += a[i]; for loop iterates n times, executes 2 assignment statements each iteration ==> asymptotic complexity of O(n)

Dale Roberts Statement Analysis – Nested For Nested For-Loops Analyze inside-out. Total running time is running time of the statement multiplied by product of the sizes of all the for-loops e.g. for (i =0; i < n; i++) for (j = 0, sum = a[0]; j <= i ; j++) for (j = 0, sum = a[0]; j <= i ; j++) sum += a[j]; sum += a[j]; printf("sum for subarray - through %d is %d\n", i, sum); printf("sum for subarray - through %d is %d\n", i, sum);

Dale Roberts Statement Analysis – Nested For (cont) )( )( )()1( nO iO iO i n i n i n i i j n i          O  O      

Dale Roberts Statement Analysis – General Rules Strategy for analysis analyze from inside out analyze function calls first if recursion behaves like a for-loop, analysis is trivial; otherwise, use recurrence relation to solve

Dale Roberts Big-Omega Example Example: n 1/2 =  ( lg n). Use the definition with c = 1 and n 0 = 16. Checks OK. Let n > 16: n 1/2 > (1) lg n if and only if n > ( lg n ) 2 Use the definition with c = 1 and n 0 = 16. Checks OK. Let n > 16: n 1/2 > (1) lg n if and only if n > ( lg n ) 2 by squaring both sides. This is an example of polynomial vs. log. n(log n)^2Diff

Dale Roberts Theta Asymptotic Tight Bound Theta means that f is bounded above and below by g; Big Theta implies the "best fit". f(n) does not have to be linear itself in order to be of linear growth; it just has to be between two linear functions. We will use Theta whenever we have enough information to show that the f(n) is both an upper and lower bound. Theta is a “stronger” statement than Big-Oh or Big-Omega.

Dale Roberts Theta Example Example: f(n) = n 2 - 5n The constant 13 doesn't change as n grows, so it is not crucial. The low order term, -5n, doesn't have much effect on f compared to the quadratic term, n 2. Q: What does it mean to say f(n) =  (g(n)) ? A: Intuitively, it means that function f is the same order of magnitude as g.

Dale Roberts Theta Example (cont.) Q: What does it mean to say f 1 (n) =  (1)? A: f 1 (n) =  (1) means after a few n, f 1 is bounded above & below by a constant. Q: What does it mean to say f 2 (n) =  (n lg n)? A: f 2 (n) =  (n lg n) means that after a few n, f 2 is bounded above and below by a constant times nlg n. In other words, f 2 is the same order of magnitude as nlg n. More generally, f(n) =  (g(n)) means that f(n) is a member of  (g(n)) where  (g(n)) is a set of functions of the same order of magnitude.

Dale Roberts Acknowledgements Philadephia University, Jordan Nilagupta, Pradondet