Lecture 2 Computational Complexity Time Complexity Space Complexity Algoirthms use less space than time. e.g., NP is included in PSPACE.

In computer science, the time complexity of an algorithm quantifies the amount of time taken by an algorithm to run as a function of the length of the string representing the input[1]:226.

The Importance of Analyzing the Running Time of an Algorithm An example to illustrate the importance of analyzing the running time of an algorithm Problem: Given an array of n elements A[1..n], sort the entries in A in non-decreasing order. Assumption: Each element comparison takes 10-6 seconds on some computing machine Conclusion: Time is undoubtedly an extremely precious resource to be investigated in the analysis of algorithms. Question: How to Analysis Running Time? algorithm # of element comparisons n=128 =27 n=1,048,567 =220 Selection Sort Merge Sort

Running Time How to measure the efficiency of an algorithm from the point of view of time? The running time usually increases with the growing of input size. So, the running time of an algorithm is usually described as a function of input size. What is the measure of time? How to define the value of the function for input size n ? If the running time is only dependent on input size? If not only dependent on input size, but also on input? Is actual (exact) running time a good measure? The answer is No. Why? Actual time is determined by not only the algorithm, but also many other factors; The measure should be machine or technology independent; Our estimates of times are relative as opposed to absolute;

Time complexity In computer science, the time complexity of an algorithm quantifies the amount of time taken by an algorithm to run as a function of the length of the string representing the input[1]:226. Time complexity is commonly estimated by counting the number of elementary operations performed by the algorithm, where an elementary operation takes a fixed amount of time to perform.

Running Time (continued) Notice! Different models assume different elementary steps! Examples of usually used elementary operations Arithmetic operations: addition, subtraction, multiplication and division. Comparisons and logical operations. Assignments, including assignments of pointers.

Example ! For some algorithm, selection sort for example, its running time is only dependent on input size. For any input with input size n, the running time of selection sort algorithm is always cn(n-1)/2 + bn + a for some constants a, b, c. Our main concern is not in small input instances but the behavior of the algorithms under investigation on large input instances, especially with the size of the input in the limit. In this case, we are interested in the order of growth of the running time function.

Example But for many algorithms, running time is dependent also on input structure!

Example (cont.) Then It is tedious! And tj depends on different input! e.g., if A is initially in sorted order, then for each j=2, …, n, the while loop test in line 5 is only executed once, thus, tj=1 for j=2, …,n. And we have: Which is a linear function, and the order of growth of the function is n.

Example (cont.) If the array is initially in decreasing order. Then for each j=2, …,n, the while loop test is executed j times, thus tj=j for j=2, …,n. Thus, we have Which is a quadratic function, the order of growth of which is n2

Asymptotic notions How to define running time in function of input size when running time is dependent also on input? For simplicity, we need asymptotic notions! Our main concern is not in small input instances but the behavior of the algorithms under investigation on large input instances, especially with the size of the input in the limit. In this case, we are interested in the order of growth of the running time function.

In this course, we use three notations: Asymptotic Notations e.g., the running time of INSERTION-SORT falls between a1n+b1 and a2n2+b2n+c2, and neither of them can represent the whole case. By using the asymptotic notations, the running time of INSERTION-SORT can be simplified. In this course, we use three notations: O(.) : “Big-Oh” – the most used Ω(.) : “Big omega” Θ(.) : “Big theta”

O-notation Informal definition of O(.): If an algorithm’s running time t(n) is bounded above by a function g(n), to within a constant multiple c, for n  n0, we say that the running time of the algorithm is O(g(n)), and the algorithm is an O(g(n))-algorithm. Obviously, O-notation is used to bound the worst-case running time of an algorithm

O-notation Formal definition of O(.): A function t(n) is said to be in O(g(n)), if there exist some c > 0, n0 > 0, such that 0 ≤ t(n) ≤ cg(n), for all n ≥ n0. Sometimes, also denote as t(n) = O(g(n)).

Ω-notation Informal definition of Ω(.) : If an algorithm’s running time t(n) is bounded below by a function g(n), to within a constant multiple c, for n  n0, we say that the running time of the algorithm is Ω(g(n)), and the algorithm is called an Ω(g(n))-algorithm.

Ω-notation Formal definition of Ω(.) : A function t(n) is said to be in Ω(g(n)), if there exist some c > 0, n0 > 0, such that t(n) ≥ cg(n) ≥0, for all n ≥ n0. Sometimes, also denote as t(n) = Ω(g(n)). Obviously, Ω -notation is used to bound the best-case running time of an algorithm

Θ-notation Informal definition of Θ(.) : If an algorithm’s running time t(n) is bounded above and below by a function g(n), to within a constant multiple c1 and c2, for n  n0, we say that the running time of the algorithm is Θ(g(n)), and the algorithm is called a Θ(g(n))-algorithm, or of order Θ(g(n)).

Θ-notation Formal definition of Θ(.) : A function t(n) is said to be in Θ(g(n)) if there exist c1 > 0, c2 > 0, and n0 > 0, such that 0 ≤ c2g(n) ≤ t(n) ≤ c1g(n), for all n ≥ n0. Sometimes, also denote as t(n) = Θ(g(n)). g(n) is an asymptotically tight bound for t(n)

Example The running time of INSERTION-SORT is O(n2), and (n), which means the running time of every input of size n for n  n0 is up-bounded by a constant times n2, and lower-bounded by a constant times n when n is sufficiently large. Notice that for some algorithm, INSERTION-SORT for example, there does not exist a function g(n) such that the running time of the algorithm is (g(n)).

Illustration of some typical asymptotic running time functions We can see:The linear algorithm is obviously slower than the quadratic one and faster than logarithmic one, etc..

Three Types of Analysis Given input size, the running time may vary on different input instance, e.g., the running time of INSERTION-SORT falls between linear and quadratic functions. Can we give an exact order? To have an overall grading on the performance of an algorithm, we consider: Best-Case Analysis: Too optimistic Average-Case Analysis: Too difficult, e.g. the difficulty to define “average case”, the difficulty related with mathematics. And most time average-case running time is in the same order as worst-case running time. Worst-Case Analysis: Very useful and practical. We will adopt this approach.

Example of Three Types of Analysis

Example of analysis (average-case) You should first figure out the distribution space of the instances of size n. Then compute an expected running time. Usually, we assume all possibilities are equally likely. Thus, in this example, in the while loop, on average, half of the elements in A[1, ..,j-1] are less than A[j], and half are greater than A[j]. Then on average, tj=j/2, and we can see the average-case time complexity is also quadratic.

Time complexity Since an algorithm's performance time may vary with different inputs of the same size, one commonly uses the worst-case time complexity of an algorithm, denoted as T(n), which is defined as the maximum amount of time taken on any input of size n.   Worst-case time complexity of an algorithm is the longest running time (or maximum number of elementary operations ) taken on any input of size n. E.g., the worst-case time complexity of INSERTION-SORT (or the running time of INSERTION-SORT in the worst case) is (n2), which is an asymptotic tight bound. The running time (time complexity) of INSERTION-SORT is O(n2), (n). But the running time (time complexity) of INSERTION-SORT is NOT (n2), NOT (n2), and is NOT O(n). We cannot give a tight bound on the running time for all inputs. We usually consider one algorithm to be more efficient than another if its worst-case running time has a lower order of growth.

Worst-case time complexity It is NOT true that each algorithm has a tight bound on its worst-case time complexity. E.g., suppose A is a sorted array of numbers, x is a number. n is the length of A. The following algorithm decides whether x is in A. 1. if n is odd k BINARYSEARCH (A, x) 2. else k LINEARSEARCH (A, x) Obviously, the running time of this algorithm is O(n) for each input, and thus in the worst case. And for each constant n0, there are infinitely many inputs whose size is larger than n0 and whose cost is lower-bounded by a constant times n, but, the running time in the worst case is NOT (n), and NOT (n), since, for each n0, there exists some nn0 such that no input of that size costs no less than a constant times n.

Input size Input size is the length of the string representing the input. Under TURING model, this is easy to define, that is the number of nonblank cells the input occupies on the input tape. In real world, this is impossible. The input size is not a precise measure of the input, and its interpretation is subject to the problem for which the algorithm is designed. Commonly used measures of input size are the following: 1.sorting and searching# of entries in the array or list 2.Graph algorithms # of vertices or edges, or both 3.Computational geometry # of points, vertices, edges, line segments, polygons, etc. 4.Matrix operations dimensions of the input matrices 5.Number theory and cryptography # of bits in the input.

Input size Inconsistencies brought about: An algorithm for adding two n  n matrices which performs n2 additions sounds quadratic, while it is indeed linear in the input size. The following two algorithms both compute the sum , and the # of elementary operations performed is the same, but they have different time complexity. Algorithm 1 input: A positive integer n and an array A[1..n]with A[j]=j, 1j n Output: 1.sum0 2.for j1 to n 3. sum sum + A[j] 4. end for 5.return sum Algorithm 2 input: A positive integer n Output: 1.sum0 2.for j1 to n 3. sum sum + j 4. end for 5.return sum Both algorithms run in time (n). But Algorithm 1 is a linear algorithm since its time complexity is (n), where n is the real length of its input, while Algorithm 2 is an exponential algorithm since its input is a number whose length is k=log(n+1) , and (n)= (2k).