2. Measuring complexity Section 7.1 Giorgi Japaridze Theory of Computability

3. Measuring complexity 7.1.a Giorgi Japaridze Theory of Computability Definition 7.1 Let M be a deterministic TM that halts for every input. The running time or time complexity of M is the function f: N  N, where f(n) is the maximum number of steps that M uses on any input of length n. If f(n) is the time complexity of M, we say that M runs in time f(n), or that M is an f(n) time machine. Customarily we use n to represent the length of the input. If the time complexity of M is f(n) = n 2 +2, at most how many steps would M take to accept or reject the following strings?  0 10 01001

4. What are the time complexities f(n) of: 7.1.b Giorgi Japaridze Theory of Computability 1. The fastest machine that decides {w | w starts with a 0}? 2. The fastest machine that decides {w | w ends with a 0}? 3. The following machine M1, deciding the language {0 k 1 k | k  0}: M1 = “On input string w: 1. Scan across the tape and reject if a 0 is found to the right of a 1. 2. Repeat if both 0s and 1s remain on the tape: 3. Scan across the tape, crossing off a single 0 and a single 1. 4. If 0s still remain after all the 1s have been crossed off, or if 1s still remain after all the 0s have been crossed off, reject. Otherwise, if neither 0s nor 1s remain on the tape, accept.” Here we are lazy to try to figure out the exact values of the constants a, b, c (though, apparently b=1). But are those constants really all that important once we know that n 2 is involved?

5. Asymptotic analysis 7.1.c Giorgi Japaridze Theory of Computability The exact running time of an algorithm often is a complex expression and depends on implementation and model details such as the number of states, the number of tape symbols, whether the “stay put” option is allowed, etc. Therefore, we usually just estimate time complexity, considering only “very large” inputs and disregarding constant factors. This sort of estimation is called asymptotic analysis. It is insensitive with respect to the above minor technical variations. E.g., if the running time is f(n) = 6n 3 +2n 2 +20n+45 (n --- the length of input), on large n’s the first term dominates all other terms, in the sense that 2n 2 +20n+45 is less than n 3, so that 6n 3 < f(n) < 7n 3. And, as we do not care about constant factors (which is something between 6 and 7), after disregarding it we are left with just n 3, i.e. the highest of the orders of the terms. We express the above by using the asymptotic notation or big-O notation, writing f(n) = O(n 3 ). Intuitively, O can be seen as a suppressed constant, and the expression f(n) = O(n 3 ) as saying that, on large inputs, f(n) does not exceed n 3 by more than some constant factor. The small-o notation: f(n) = o(n 4 ) intuitively means that, on large inputs, f(n) gets smaller (and smaller) compared with n 4 --- smaller by more than any given constant factor.

6. The definitions of big-O and small-o 7.1.d Giorgi Japaridze Theory of Computability Definition 7.2 Let f and g be functions f,g: N  R +. Say that f(n) = O(g(n)) iff positive integers c and n 0 exists such that for every integer n  n 0, f(n)  cg(n). When f(n) = O(g(n)), we say that g(n) is an asymptotic upper bound for f(n). N means natural numbers, R + means positive real numbers Definition 7.5 Let f and g be functions f,g: N  R +. Say that f(n) = o(g(n)) iff for every positive real number c, a number n 0 exists such that for every integer n  n 0, f(n) < cg(n). In other words, lim = 0. n   f(n) g(n) We always have f(n)=O(f(n)) while never f(n)=o(f(n))! Our focus will mainly be on O. Intuition: “The complexity f(n) is  the complexity g(n)”. Intuition: “The complexity f(n) is < the complexity g(n)”.

7. How Big-O interacts with polynomial functions 7.1.e Giorgi Japaridze Theory of Computability f(n) = O(g(n)) iff there are c and n 0 such that, for every n  n 0, f(n)  cg(n). f(n) = O(f(n)): pick c = n 0 = 3f(n) = O(f(n)): pick c = n 0 = 5n+10 = O(n): pick c = n 0 = 3n 2 +4n+2 = O(n 2 ): pick c = n 0 = Generally, b d n d + b d-1 n d-1 + … + b 2 n 2 + b 1 n + b 0 = O(n d )

8. How big-O interacts with logarithms 7.1.f Giorgi Japaridze Theory of Computability f(n) = O(g(n)) iff there are c and n 0 such that, for every n  n 0, f(n)  cg(n). log 8 n = O(log 2 n): pick c = n 0 = Remember that log b n = log 2 n / log 2 b = (1/log 2 b) * log 2 n log 2 n = O(log 8 n): pick c = n 0 = Hence, in asymptotic analysis, we can simply write log n without specifying the base. Remember that log n c = c log n Hence log n c = Since log c n = n log c and c is a constant, we have log c n = Generally, log c f(n) =

9. More on big-O 7.1.g Giorgi Japaridze Theory of Computability Big-O notation also appears in expressions such as f(n)=O(n 2 )+O(n). Here each occurrence of O represents a different suppressed constant. Because the O(n 2 ) term dominates the O(n) term, we have O(n 2 )+O(n) = O(n 2 ). Such bounds n c (c  0) are called polynomial bounds. When O occurs in an exponent, as in f(n) = 2 O(n), the same idea applies. This expression represents an upper bound of 2 cn for some (suppressed) constant c. In other words, this is the bound d n for some (suppressed) constant d (d=2 c ). Such bounds d n, or more generally 2 ( n  ) where  is a positive real number, are called exponential bounds. Sometimes you may see the expression f(n) = 2 O(log n). Using the identity n=2 log n and thus n c =2 c log n, we see that 2 O(log n) represents an upper bound of n c for some c. The expression n O(1) represents the same bound in a different way, because O(1) represents a value that is never more than a fixed constant.

10. Small-o revisited 7.1.h Giorgi Japaridze Theory of Computability Definition 7.5 Let f and g be functions f,g: N  R +. Say that f(n) = o(g(n)) iff for every positive real number c, a number n 0 exists such that for every integer n  n 0, f(n) < cg(n). In other words, lim = 0. n   f(n) g(n) Equivalently, Definition 7.5* Let f and g be functions f,g: N  R +. Say that f(n) = o(g(n)) if for every c, an n 0 exists such that for every n  n 0, cf(n) < g(n). In other words, lim = . n   g(n) f(n)

11. Examples on small-o 7.1.i Giorgi Japaridze Theory of Computability 1.  n = o(n). Indeed, given any c, consider an arbitrary n with c c 2 ). Then c  n <  n  n = (  n) 2 = n, as desired. f(n) = o(g(n)) iff for every c there is an n 0 s.t., for all n  n 0, cf(n) < g(n). 3. n = o(n log n). Given c, consider an arbitrary n with c 2 c ). Then cn < (log n)n = n log n, as desired. 4. log n = o(n). Given c, consider an arbitrary n with c log n < n. 2. n 2 = o(n 3 ). Given c, consider any n with c<n. Then cn 2 < nn 2 = n 3. Generally: we always have f(n)=O(f(n)) while never f(n)=o(f(n)) f(n)=o(g(n)) implies f(n)=O(g(n)) but not vice versa if f(n)=o(g(n)), then g(n)≠o(f(n)) and g(n)≠O(f(n)) 5. n log n = o(n 2 ). Can be seen to follow from the previous statement.

12. Time complexity classes 7.1.j Giorgi Japaridze Theory of Computability Definition 7.7 Let t: N  R + be a function. We define the t-time complexity class, TIME(t(n)), to be the collection of all languages that are decidable by an O(t(n)) timeTuring machine. {w | w starts with a 0}  TIME(n) ? {w | w starts with a 0}  TIME(1) ? {w | w ends with a 0}  TIME(n) ? {w | w ends with a 0}  TIME(1) ? {0 k 1 k | k  0}  TIME(n) ? {0 k 1 k | k  0}  TIME(n 2 ) ? {0 k 1 k | k  0}  TIME(n log n) ? Every regular language A  TIME(n) ? Linear time Constant time Square time

13. The O(n 2 ) time machine for {0 k 1 k | k  0} 7.1.k Giorgi Japaridze Theory of Computability M1 = “On input string w: 1. Scan across the tape and reject if a 0 is found to the right of a 1. 2. Repeat if both 0s and 1s remain on the tape: 3. Scan across the tape, crossing off a single 0 and a single 1. 4. If 0s still remain after all the 1s have been crossed off, or if 1s still remain after all the 0s have been crossed off, reject. Otherwise, if neither 0s nor 1s remain on the tape, accept.” Stage 1 takes 2n (plus-minus a constant) steps, so it uses O(n) steps. Note that moving back to the beginning is not explicitly mentioned there. The beauty of big-O is that it allows us to suppress these details (ones that affect the number of steps only by a constant factor). Each scan in Stage 3 takes O(n) steps, and Stage 3 is repeated at most n/2 times. So, Stage 2 (together with all repetitions of Stage 3) takes (n/2)O(n) = O(n 2 ) steps. Stage 4 takes (at most) O(n) steps. Thus, the complexity is O(n)+O(n 2 )+O(n) = O(n 2 ) Asymptotic analysis of the time complexity of M1: What if we allowed the machine to cross out two 0s and two 1s one each pass?

14. An O(n log n) time machine for {0 k 1 k | k  0} 7.1.l Giorgi Japaridze Theory of Computability M2 = “On input string w: 1. Scan across the tape and reject if a 0 is found to the right of a 1. 2. Repeat if both 0s and 1s remain on the tape: 3. Scan across the tape, checking whether the total number of 0s and 1s remaining is even or odd. If it is odd, reject. 4. Scan again across the tape, crossing off every other 0 starting with the first 0, and then crossing off every other 1 starting with the first 1. 5. If no 0s and no 1s remain on the tape, accept. Otherwise reject.” How many steps do each of the Stages 1, 3, 4 and 5 take? How many times are Stages 3 and 4 are repeated? What is the overall time complexity? Smart algorithms can often be much more efficient than brute force ones!

15. An O(n) time two-tape machine for {0 k 1 k | k  0} 7.1.m Giorgi Japaridze Theory of Computability M3 = “On input string w: 1. Scan across the tape and reject if a 0 is found to the right of a 1. 2. Scan across the 0s on tape 1 until the first 1. At the same time, copy the 0s onto tape 2. 3. Scan across the 1s on tape until the end of the input. For each 1 read on tape 1, cross off a 0 on tape 2 (moving right-to-left there). If all 0s are crossed off before all the 1s are read, reject. 4. If all the 0s have now been crossed off, accept. If any 0s remain, reject.” How many steps do each of the Stages 1, 2, 3 and 4 take? How many times are Stages 3 and 4 are repeated? What is the overall time complexity? An important difference between computability theory and complexity theory: The former is insensitive with respect to “reasonable” variations of the underlying Computation models (variants of Turing machines), while the latter is: to what complexity class a given language belongs may depend on the choice of the model! Fortunately, however, time requirements do not differ greatly for typical deterministic models. So, if our classification system isn’t very sensitive to relatively small (such as linear vs. square) differences in complexity, the choice of deterministic model is not crucial.

16. Single-tape vs. multitape machines 7.1.n Giorgi Japaridze Theory of Computability Theorem 7.8 Let t(n) be a function, where t(n)  n. Then every t(n) time multitape Turing machine has an equivalent O(t 2 (n)) time single-tape Turing machine. Proof Idea. Remember the proof of Theorem 3.13. It shows how to convert a multitape TM M into an equivalent single-tape TM S that simulates M. We need to analyze the time complexity of S. The simulation of each step of M takes O(k) steps in S, where k is the length of the active content of the tape of S (specifically, S makes two passes through its tape; a pass may require shifting, which still takes O(k) steps). How big can k be? Not bigger than the number of steps M takes, multiplied by the (constant) number c of tapes. That is, k  ct(n). Thus, S makes O(t(n)) passes through the active part of its tape, and each pass takes (at most) O(t(n)) steps. Hence the complexity of S is O(t(n))  O(t(n)) = O(t 2 (n)).

17. Definition of time complexity for nondeterministic machines 7.1.o Giorgi Japaridze Theory of Computability Definition 7.9 Let M be a nondeterministic TM that is a decider (meaning that, on every input, each branch of computation halts). The running time or time complexity of M is the function f: N  N, where f(n) is the maximum number of steps that M uses on any branch of its computation on any input of length n, as shown below, with standing for a halting (accept or reject) state. Deterministic Nondeterministic … … f(n)

18. Deterministic vs. nondeterministic machines 7.1.p Giorgi Japaridze Theory of Computability Theorem 7.11 Let t(n) be a function, where t(n)  n. Then every t(n) time nondeterministic TM has an equivalent 2 O(t(n)) time deterministic TM. Proof Idea. One should remember the proof of Theorem 3.16. It shows how to convert a nondeterministic TM N into an equivalent deterministic TM D that simulates N by searching N’s nondeterministic computation tree. Each branch of that tree has length at most t(n), and thus constructing and searching it takes O(t(n)) steps. And the number of branches is b O(t(n)), where b is the maximum number of legal choices given by N’s transition function. But b  2 c for some constant c. So, the number of branches is in fact 2 cO(t(n)) =2 O(t(n)). Thus, the overall number of steps is O(t(n))  2 O(t(n)) = 2 O(t(n)).