1. Asymptotic Notation Q, O, W, o, w
Used to describe the running times of algorithms
Instead of exact running time, say Q(n2)
Defined for functions whose domain is the set of natural numbers
Determine sets of functions, in practice used to compare two functions
There are actually 5 kinds of asymptotic notation. How many of you are familiar with all of these?
What these symbols do is give us a notation for talking about how fast a function goes to infinity, which is just what we want to know when we study the running times of algorithms. Instead of working out a complicated formula for the exact running time, we can just say that the running time is theta of n^2. That is, the running time is proportional to n^2 plus lower order terms. For most purposes, that’s just what we want to know.
One thing to keep in mind is that we’re working with functions defined on the natural numbers. Sometimes I’ll talk (a little) about doing calculus on these functions, but the piont is that we won’t care what, say, f(1/2) is.

2. O-notation
For a given function g(n), we denote by O(g(n)) the set of functions O(g(n)) = {f(n): there exist positive constants c and n0 such that 0  f(n)  cg(n) for all n  n0 }
Sometimes we won’t know the exact order of growth. Sometimes the running time depends on the input, or we might be talking about a number of different algorithms. Then we might want to put an upper or lower bound on the order of growth. That’s what big-O and big-Omega are for.
Except for Theta, the thing to remember is that the English letters are upper bounds, and the Greek letters are lower bounds. (Theta is both, but it’s only a greek letter.) So O(g(n)) is the set of functions that go to infinity no faster than g. The formal definition is the same as for Theta, except that there is only one c, and you have the inequality.
We call g an asymptotic . . .
We say g(n) is an asymptotic upper bound for f(n)

3. -notation
For a given function g(n), we denote by (g(n)) the set of functions (g(n)) = {f(n): there exist positive constants c and n0 such that 0  cg(n)  f(n) for all n  n0 }
In the same way, Omega(g(n)) is the set of functions that go to infinity no slower than g(n). Again, the definition is the same except that the inequality reads “0 le c g(n) le f(n)” for all n ge n0.
Are there any questions?
We say g(n) is an asymptotic lower bound for f(n)

4. -notation
For a given function g(n), we denote by (g(n)) the set of functions
(g(n)) = {f(n): there exist positive constants c1, c2 and n0 such that
0  c1g(n)  f(n)  c2g(n),
for all n  n0 }
Theta notation may be the one we use the most. Sometimes, when people say “O of n”, they really mean theta(n). When we say that one function is theta of another, we mean that neither function goes to infinity faster than the other.
Formally, theta(g(n)) is a set of functions. It’s the set of all functions f(n) such that there exist positive constants c1, c2, and n0 st 0 le c1 g(n) \le f(n) le c2 g(n) for all n ge n0. I’ll stop and let you digest that for a minute.
Questions about notation?
Graphically, we can see what this means. It doesn’t matter what f does for low n. But there’s some threshold, n0, after which f is bounded between these two different multiples of g(n).
Also, the definition tells us that g and f are both nonnegative for large n. This makes sense since we’re interested in running times.
When f(n) is theta of g(n), we say that g(n)…
We say g(n) is an asymptotically tight bound for f(n)

5. Example 10n2 - 3n = Q(n2) (actually = is ϵ)
What constants for n0, c1, and c2 will work?
Make c1 a little smaller than leading coefficient, and c2 a little bigger
To compare orders of growth, look at leading term
So let’s work out an example.
First of all, how can you tell, without using the definition, whether two functions have the same order of growth? What part do you look at?
So, looking at the leading terms, we see that 10n^2 - 3n = theta(n^2). Now I want you to take some time in class and work out the constants to show it via the definition. Once you have an answer, if everyone else isn’t done yet, compare notes with your neighbors.
So there’s different answers. What did people get?
[Get white board. . .
We really have 2 separate inequalities. The first is c1 g(n) le f(n), and the secondis . . .
. . . go back to screen]
The thing to realize is that the high-order term will dominate the others, so you
can always do this trick of making c1 just a little smaller than the leading coefficient, and c2 a little bigger. And this is why you can compare by just looking at the lead term.

6. Relations Between Q, W, O I.e., (g(n)) = O(g(n)) Ç W(g(n))
For any two functions g(n) and f(n), f(n) = (g(n)) if and only if f(n) = O(g(n)) and f(n) = (g(n)).
I.e., (g(n)) = O(g(n)) Ç W(g(n))
If you look at the definitions, it’s pretty clear that the definition for being in \theta(g(n)) requires you to satisfy the definitions for O and \tomega. If you go back, you see that, for big enough n, you have to be *both* le than c2 g(n) and ge c1 g(n). Set theoretically, this means that, for any function, Theta is just the int of big O and big Omega.

7. Running Times “The running time is O(f(n))” Þ Worst case is O(f(n))
“Running time is W(f(n))” Þ Best case is W(f(n))
Can still say “Worst case running time is W(f(n))”
Means worst case running time is given by some unkown function g(n) Î W(f(n))
Now let’s talk a little about how we use these capital-letter notations in practice. Then we’ll go on to talk about little o and omega.
Sometimes people will say “the running time for this algorithm is O(f(n))”, even though the actual running time depends on the input data. Since big O gives an upper bound, what this means is that the worst case running time is O(f(n)) (That is, no worse than c f(n) for large n)
You can talk about the best case running time, and that’s what you mean if you just say “the running time is Omega(f(n)).
However, you might instead say the *worst* case running time is Omega(f(n)). This means that you don’t know what the worst case running time is, but you *do* know that it’s asymptotically bounded below by f(n). This comes up a lot when you want to know what the worst case running time is, and you try to approach it from above and below. Here’s an example.

8. Example
Insertion sort takes Q(n2) worst case time, so sorting (as a problem) is O(n2)
Any sort algorithm must look at each item, so sorting is W(n)
In fact, using (e.g.) merge sort, sorting is Q(n lg n) in the worst case
This is an example of the notion of inherent complexity that I talked about in the introduction to the course. Suppose you need to sort some data, and you want to know how much time it will take. Suppose the only algorithm you know is insertion sort, which takes Thet(n^2) worst case time. This means that sorting, as a problem, is O(n^2).
Remember, this is just telling us that n^2 is an asymptotic upper bound on the running time for sort. That’s why we use the O and not Theta.
On the other hand, any sort has to at lesast do something with each item, so sorting is \Omega(n).
If the only algorithm you knew for sorting was insertion sort, and you hadn’t proved any fancy theorems, then this is what you’d know about the inherent complexity of quicksort. It would make sense to look for quicker algorithms, because your best algorithm is worse than the known lower bound on the complexity of the problem.
However, we know that merge sort is Theta(n lg n) worst-case, and moreover we know that’s the best you can hope for, because the inherent complexity of sorting is Theta n \lg n worst case.

9. Asymptotic Notation in Equations
Used to replace expressions containing lower-order terms
For example,
4n3 + 3n2 + 2n + 1 = 4n3 + 3n2 + (n)
= 4n3 + (n2) = (n3)
In equations, (f(n)) always stands for an anonymous function g(n) Î (f(n))
In the example above, (n2) stands for 3n2 + 2n + 1
Another way we use asymptotic notation is to simplify calculations. We can replace groups of terms in an equation with a theta expression. As an example here, we can rewrite 4n^3 … in several ways: as…
This may seem bad, since we defined Theta(g(n)) as a set of functions. The way we think about this is that \Theta(f(n)) represents some anonymous function in the set -- that is, some unspecified function that will make the equation come out right. This was the interpretation earlier when we used = where \in might have seemed more correct.

10. o-notation
For a given function g(n), we denote by o(g(n)) the set of functions o(g(n)) = {f(n): for any positive constant c > 0, there exists a constant n0 > 0 such that 0  f(n) < cg(n) for all n  n0 } f(n) becomes insignificant relative to g(n) as n approaches infinity: lim [f(n) / g(n)] = 0 n We say g(n) is an upper bound for f(n) that is not asymptotically tight.
So far, Theta(g) is the functions that go to infinity essentially at the same speed, O(g) goes to infinity no faster than g, and Omega(g) goes to inf at least as fast as g. Sometimes, we want to say that a function grows strictly faster or slower than g. That’s what the lower case letters are for.
So o(g(n)) is the set of functions that, in the long run, become insignificant compared to g. The way we say this formally is that o(g(n)) is the set of f(n) such that …
So, if you want g(n) to be twice as big as f(n) you choose a certain n0. If you want it to be 4 times as big, you may have to go to larger values of n, but eventually it will be, and so on.
You can express this in terms of limits -- etc.

11. w-notation
For a given function g(n), we denote by w(g(n)) the set of functions
w(g(n)) = {f(n): for any positive constant c > 0, there exists a constant n0 > 0 such that 0  cg(n) < f(n) for all n  n0 }
f(n) becomes arbitrarily large relative to g(n) as n approaches infinity: lim [f(n) / g(n)] = 
n
We say g(n) is a lower bound for f(n) that is not asymptotically tight.
By now you should know the pattern: little omega of g(n) is the set of functions that grow arbitarily large compared to g(n). (just run through)

12. Limits lim [f(n) / g(n)] = 0 Þ f(n) Î o(g(n))
0 < lim [f(n) / g(n)] <  Þ f(n) Î Q(g(n))
0 < lim [f(n) / g(n)] Þ f(n) Î W(g(n))
lim [f(n) / g(n)] =  Þ f(n) Î w(g(n))
lim [f(n) / g(n)] undefined Þ can’t say
Now, for the lower case letters we were able to characterize the notation in terms of limits. Actually, you can partially do it for all of them, except that the limit doesn’t always exist. So here’s a complete table. If the limit as n goestto infty of f(n)/g(n) is 0, then f is o(g). If the limit is finite, then f is O(g). The limit still could be 0 -- o is a subset of O. If the limit is finite and strictly positive, then f is Theta(g).
If all we can say is that the limit is greater than 0, then f is Omega(g). It may still be Theta(g), or it may be omega(g). Finally, if the limit is infty, then f is omega(g).
But it can happen that the limit doesn’t exist at all. Suppose f(n) oscillates between c1 g and c2(g), like this. But fortunately, that almost never comes up in algorithms, so limits are useful for thinking about all of these cases.

13. Comparison of Functions
f  g  a  b f (n) = O(g(n))  a  b f (n) = (g(n))  a  b f (n) = (g(n))  a = b f (n) = o(g(n))  a < b f (n) = w (g(n))  a > b
One thing you may have noticed by now is that these relations are kind of like the <> relations for the numbers. There is one difference, in that you can’t compare all pairs of functions. F(n) might oscillate back and forth between a function that grows faster than g(n), and one that grows slower. But that’s also a problem that doesn’t show up in practice.
This isn’t just an interesting fact. One thing you need to be able to do is quickly compare the orders of growth of different functions. If two algorithms have fairly complicated running times, you need to know which one is better. Keeping in mind the analogy with ordering of numbers can come in handy.

14. Properties Transitivity Symmetry
f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n))
f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n))
f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n))
f(n) = o (g(n)) & g(n) = o (h(n))  f(n) = o (h(n))
f(n) = w(g(n)) & g(n) = w(h(n))  f(n) = w(h(n))
Symmetry
f(n) = (g(n)) if and only if g(n) = (f(n))
Transpose Symmetry
f(n) = O(g(n)) if and only if g(n) = (f(n))
f(n) = o(g(n)) if and only if g(n) = w((f(n))
One example of this analogy is that some of the properties we’re used to from equations and inequalities of numbers applies here.
[go through]
All of these correspond to properties of the order relations for numbers. These sorts of relationships you’ll use all the time when comparing the order of growth of two functions. If you can compare two functions to some third function, then perhaps you can compare them to each other.

15. Useful Facts
For a ³ 0, b > 0, lim n ( lga n / nb ) = 0, so lga n = o(nb), and nb = w(lga n )
Prove using L’Hopital’s rule repeatedly
lg(n!) = (n lg n)
Here are two other useful facts for comparing different functions. The first one involves lg^a n.
Proving both of them is a good exercise. The first is just mechanical, and the second isn’t too bad iether.

16. Examples A B 5n2 + 100n 3n2 + 2 log3(n2) log2(n3) nlg4 3lg n lg2n n1/2
o , , or w?

17. Examples A B 5n2 + 100n 3n2 + 2 log3(n2) log2(n3) nlg4 3lg n lg2n n1/2
o , , or w?
logba = logca / logcb
alog b = blog a

18. Examples A B 5n2 + 100n 3n2 + 2 A  (B) log3(n2) log2(n3) nlg4 3lg n
A  (n2), n2  (B)  A  (B)
log3(n2) log2(n3)
nlg lg n
lg2n n1/2

19. Examples A B 5n2 + 100n 3n2 + 2 A  (B) log3(n2) log2(n3) A  (B)
A  (n2), n2  (B)  A  (B)
log3(n2) log2(n3) A  (B)
logba = logca / logcb; A = 2lg2n / lg3, B = 3lg2n, A/B =2/(3lg3)
nlg lg n
lg2n n1/2

20. Examples A B 5n2 + 100n 3n2 + 2 A  (B) log3(n2) log2(n3) A  (B)
A  (n2), n2  (B)  A  (B)
log3(n2) log2(n3) A  (B)
logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)
nlg lg n A  w(B)
alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3)   as n
lg2n n1/2

21. Examples A B 5n2 + 100n 3n2 + 2 A  (B) log3(n2) log2(n3) A  (B)
A  (n2), n2  (B)  A  (B)
log3(n2) log2(n3) A  (B)
logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)
nlg lg n A  w(B)
alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3)   as n
lg2n n1/2 A  o (B)
lim ( lga n / nb ) = 0 (here a = 2 and b = 1/2)  A  o (B)
n
Have them work on them. Tell them to try to manipulate the logs and exponents for second and third.

22. Reviews on Summation Why do we need summation formulas?
We need them for computing the running times of some algorithms. (CLRS/Chapter 3)
Example: Maximum Subvector
Given an array a[1…n] of numeric values (can be positive, zero and negative) determine the maximum value of sums to all subvectors a[i…j] (1 i  j  n).
So now we’re going to talk briefly about some summation formulas, but I’m going to put it in the context of a particular algorithm. The problem is . . .
So, let’s take an example.
If the numbers were all pos, we’d just take the whole array. But, here, the maximum subarray goes from entries 3 to 4. So you pretty much have to check all subarrays. 1 -2 2

23. Reviews on Summation
MaxSubvector(a, n) maxsum ¬ 0; for i ¬ 1 to n for j = i to n sum ¬ 0; for k ¬ i to j sum += a[k] maxsum ¬ max(sum, maxsum); return maxsum;
Here is an algorithm to find the maximum subvector sum. The outer loop lets i run from 1 to n, and the next loop in runs j from i to n. This gives you all pairs of i, j with 1 le i le j le n. For each of those pairs, the inner loop computes the sum.
To compute the running time, we use the fact that the running time of a loop is the sum of the running time of all the iterations. The control variables of the sum are the same as for the loop. So, for the inner loop, there is only 1 step, and the loop runs from i to j. So you have the sum of 1 as k goes from i to j.

24. Reviews on Summation T(n) =    1 i=1 j=i k=i MaxSubvector(a, n)
maxsum ¬ 0;
for i ¬ 1 to n
for j = i to n
sum ¬ 0;
for k ¬ i to j
sum += a[k]
maxsum ¬ max(sum, maxsum);
return maxsum;
n n j
T(n) =    1
i=1 j=i k=i
NOTE: This is not a simplified solution. What is the final answer?
Here is an algorithm to find the maximum subvector sum. The outer loop lets i run from 1 to n, and the next loop in runs j from i to n. This gives you all pairs of i, j with 1 le i le j le n. For each of those pairs, the inner loop computes the sum.
To compute the running time, we use the fact that the running time of a loop is the sum of the running time of all the iterations. The control variables of the sum are the same as for the loop. So, for the inner loop, there is only 1 step, and the loop runs from i to j. So you have the sum of 1 as k goes from i to j.

25. Reviews on Summation Constant Series: For n  0, i=a
b  1 = b - a + 1
i=a
Linear Series: For n  0,
n  i = … + n = n(n+1) / 2
i=1
n n j n n n n-i n
T(n) =    1 =   (j-i+1) =   a =  (n-i+1)(n-i+2)/2
i=1 j=i k=i i=1 j=i i=1 a= i=1

26. Reviews on Summation Quadratic Series: For n  0,
 i2 = … + n2 = (2n3 + 3n2 + n) / 6
i=1
Linear-Geometric Series: For n  0,
n n j n n
T(n) =    1 =  (n-i+1)(n-i+2)/2 = ½  b(b+1)
i=1 j=i k=i i= b=1
=  b2+  b = … (reversing order of sum)

27. Review on Summations Cubic Series: For n  0,
Geometric Series: For real x  1,
For |x| < 1,

28. Review on Summations Telescoping Series:
Harmonic Series: nth harmonic number, nI+,
Differentiating Series: For |x| < 1,

29. Review on Summations
nth harmonic number

30. More on Maximum Subvector
Given an array a[1…n] of numeric values (can be positive, zero and negative) determine the maximum value of sums to all subvectors a[i…j] (1 i  j  n)..
(Hint: How should we treat a subvector with negative sum during the search process?)
def max_subarray(A): max_ending_here = max_so_far = 0 for x in A: max_ending_here = max(0, max_ending_here + x) max_so_far = max(max_so_far, max_ending_here) return max_so_far