1. 2.1 Computational Tractability
"For me, great algorithms are the poetry of computation. Just like verse, they can be terse, allusive, dense, and even mysterious. But once unlocked, they cast a brilliant new light on some aspect of computing." - Francis Sullivan

2. Computational Tractability
As soon as an Analytic Engine exists, it will necessarily guide the future course of the science. Whenever any result is sought by its aid, the question will arise - By what course of calculation can these results be arrived at by the machine in the shortest time? - Charles Babbage
Charles Babbage (1864)
Analytic Engine (schematic)

3. Polynomial-Time
Brute force. For many non-trivial problems, there is a natural brute force search algorithm that checks every possible solution.
Typically takes 2N time or worse for inputs of size N.
Unacceptable in practice.
Desirable scaling property. When the input size doubles, the algorithm should only slow down by some constant factor C.
Def. An algorithm is poly-time if the above scaling property holds.
n ! for stable matching with n men and n women
focusing on worst-case poly-time is a subtle modeling decision - we try to justify it here
There exists constants c > 0 and d > 0 such that on every input of size N, its running time is bounded by c Nd steps.
choose C = 2d

4. Worst-Case Analysis
Worst case running time. Obtain bound on largest possible running time of algorithm on input of a given size N.
Generally captures efficiency in practice.
Draconian view, but hard to find effective alternative.
Average case running time. Obtain bound on running time of algorithm on random input as a function of input size N.
Hard (or impossible) to accurately model real instances by random distributions.
Algorithm tuned for a certain distribution may perform poorly on other inputs.

5. Worst-Case Polynomial-Time
Def. An algorithm is efficient if its running time is polynomial.
Justification: It really works in practice!
Although 6.02  1023  N20 is technically poly-time, it would be useless in practice.
In practice, the poly-time algorithms that people develop almost always have low constants and low exponents.
Breaking through the exponential barrier of brute force typically exposes some crucial structure of the problem.
Exceptions.
Some poly-time algorithms do have high constants and/or exponents, and are useless in practice.
Some exponential-time (or worse) algorithms are widely used because the worst-case instances seem to be rare.
simplex method Unix grep

6. Why It Matters
table taken from More Programming Pearls, p. 82, 400MhZ Pentium II scaled up
one step takes one nanosecond
msec = millisecond

7. 2.2 Asymptotic Order of Growth

8. Complexity
The complexity of an algorithm associates a number T(n),
the worst-case time the algorithm takes, with each problem size n.
Mathematically,
T: N+ → R+
that is T is a function that maps positive integers (giving
problem sizes) to positive real numbers (giving number
of steps).

9. Complexity Measures Problem size n
Worst-case complexity: max # steps algorithm takes on any input of size n
Best-case complexity: min # steps algorithm takes on any input of size n
Average-case complexity: avg # steps algorithm takes on inputs of size n
Best-case : unrealistic
Average-case : over what probability distribution?, analysis often hard
Worst-case : a fast algorithm has a comforting guarantee
maybe too pessimistic

10. Asymptotic Order of Growth
Upper bounds. T(n) is O(f(n)) if there exist constants c > 0 and n0  0 such that for all n  n0 we have T(n)  c · f(n).
Lower bounds. T(n) is (f(n)) if there exist constants c > 0 and n0  0 such that for all n  n0 we have T(n)  c · f(n).
Tight bounds. T(n) is (f(n)) if T(n) is both O(f(n)) and (f(n)).
Ex: T(n) = 32n2 + 17n + 32.
T(n) is O(n2), O(n3), (n2), (n), and (n2) .
T(n) is not O(n), (n3), (n), or (n3).

11. Asymptotic Order of Growth in words
A way of comparing functions that ignores constant factors and small input sizes
O(g(n)): class of functions f(n) that grow no faster than g(n)
Θ(g(n)): class of functions f(n) that grow at same rate as g(n)
Ω(g(n)): class of functions f(n) that grow at least as fast as g(n)

12. Notation Slight abuse of notation. T(n) = O(f(n)). Asymmetric:
f(n) = 5n3; g(n) = 3n2
f(n) = O(n3) = g(n)
but f(n)  g(n).
Better notation: T(n)  O(f(n)).
Meaningless statement. Any comparison-based sorting algorithm requires at least O(n log n) comparisons.
Statement doesn't "type-check."
Use  for lower bounds.

13. Properties Transitivity. If f = O(g) and g = O(h) then f = O(h).
If f = (g) and g = (h) then f = (h).
If f = (g) and g = (h) then f = (h).
Additivity.
If f = O(h) and g = O(h) then f + g = O(h).
If f = (h) and g = (h) then f + g = (h).
If f = (h) and g = O(h) then f + g = (h).

14. Asymptotic Bounds for Some Common Functions
Polynomials. a0 + a1n + … + adnd is (nd) if ad > 0.
Polynomial time. Running time is O(nd) for some constant d independent of the input size n.
Logarithms. O(log a n) = O(log b n) for any constants a, b > 0.
Logarithms. For every x > 0, log n = O(nx).
Exponentials. For every r > 1 and every d > 0, nd = O(rn).
can avoid specifying the base
log grows slower than every polynomial
every exponential grows faster than every polynomial

15. More Examples
10n2-16n+100 is O(n2) also O(n3)
10n2-16n+100 ≤ 11n2 for all n ≥ 10
10n2-16n+100 is Ω (n2) also Ω (n)
10n2-16n+100 ≥ 9n2 for all n ≥16
Therefore also 10n2-16n+100 is Θ (n2)
10n2-16n+100 is not O(n) also not Ω (n3)

16. Time efficiency of nonrecursive algorithms
General Plan for Analysis
- Decide on parameter n indicating input size
- Identify algorithm’s basic operation
- Determine worst, average, and best cases for input of size n
- Set up a sum for the number of times the basic operation is executed
- Simplify the sum using standard formulas and rules

17. Useful summation formulas and rules
liu1 = 1+1+…+1 = u - l + 1
In particular, liu1 = n = n  (n)
1in i = 1+2+…+n = n(n+1)/2  n2/2  (n2)
1in i2 = …+n2 = n(n+1)(2n+1)/6  n3/3  (n3)
0in ai = 1 + a +…+ an = (an+1 - 1)/(a - 1) for any a  1
In particular, 0in 2i = …+ 2n = 2n  (2n )
(ai ± bi ) = ai ± bi cai = cai liuai = limai + m+1iuai

18. Example Problem
Problem: Given N integers stored in an array X (int X[N]), find the sum of the numbers
How can you design an algorithm for this problem?
Iterative (Non-Recursive) Solution
Use a for or while loop and add the numbers one by one
Recursive Solution
A solution that calls itself on smaller problems to solve the big problem

19. Finding the sum of a set of numbers: Non-Recursive Algorithm
Int X[N];
Sum = 0;
For (int i = 0; i < N; i++)
Sum = Sum + X[i];

20. Finding the sum of a set of numbers: Recursive Algorithm
int sum ( int A[], int N) {
if (N == 0) return 0; Stopping rule
else return sum(A, N-1) + A[N-1]; -- Key Step
} /* end-sum */
Why recursion?
Simplifies the code drastically

21. Analyzing Running Time
RT: the amount of time it takes for the algorithm to finish execution on a particular input size
More precisely, the RT of an algorithm on a particular input is the number of primitive operations or steps executed.
We define a step to be a unit of work that can be executed in constant amount of time in a machine.

22. Finding the sum of a set of numbers: Iterative Algorithm and its analysis
Assume int X[N] of integer is our data set
Cost Times
Sum = 0; C
For (int i = 0; i < N; i++) C N
Sum = Sum + X[i]; C N
T(n) = C0 + C1*N + C2*N
Since C0, C1 and C2 are constants, T(n) can be expressed as a linear function n, i.e.,
T(n) = a + b*n, for some constants a, b

23. Another Example: Searching for a number in an array of numbers
Assume int X[N] of integer is our data set and we are searching for “key”
Cost Times
Found = 0; C
I = 0; C
while (!found && i < N){ C <= L < N
If (key ==X[I]) found = 1; C <= L <= N
I++; C <= L <= N }
T(n) = C0 + C1 + L*(C2 + C3 + C4), where 1 <= L <= N is the number of times that the loop is iterated.

24. Example2: Searching for a number in an array of numbers (continued)
What’s the best case? Loop iterates just once =>
T(n) = C0 + C1 + C2 + C3 + C4
What’s the average (expected) case? Loop iterates N/2 times =>
T(n) = C0 + C1 + N/2 * (C2 + C3 + C4)
Notice that this can be written as T(n) = a + b*n where a, b are constants
What’s the worst case? Loop iterates N times =>
T(n) = C0 + C1 + N * (C2 + C3 + C4)

25. Worst Case Analysis of Algorithms
We will only look at WORST CASE running time of an algorithm. Why?
Worst case is an upper bound on the running time. It gives us a guarantee that the algorithm will never take any longer
For some algorithms, the worst case happens fairly often. As in this search example, the searched item is typically not in the array, so the loop will iterate N times
The “average case” is often roughly as bad as the “worst case”. In our search algorithm, both the average case and the worst case are linear functions of the input size “n”

26. Asymptotic Notation
We will study the asymptotic efficiency of algorithms
To do so, we look at input sizes large enough to make only the order of growth of the running time relevant
That is, we are concerned with how the running time of an algorithm increases with the size of the input in the limit as the size of the input increases without bound.
Usually an algorithm that is asymptotically more efficient will be the best choice for all but very small inputs.
3 asymptotic notations
Big O, Q, W Notations

27. Big-Oh Notation: Asymptotic Upper Bound
T(n) = f(n) = O(g(n))
if f(n) <= c*g(n) for all n > n0, where c & n0 are constants > 0 n
c*g(n)
f(n) n0
Example: T(n) = 2n + 5 is O(n). Why?
2n+5 <= 3n, for all n >= 5
T(n) = 5*n2 + 3*n + 15 is O(n2). Why?
5*n2 + 3*n + 15 <= 6*n2, for all n >= 6

28. W Notation: Asymptotic Lower Bound
T(n) = f(n) = W(g(n))
if f(n) >= c*g(n) for all n > n0, where c and n0 are constants > 0 n
f(n)
c*g(n) n0
Example: T(n) = 2n + 5 is W(n). Why?
2n+5 >= 2n, for all n > 0
T(n) = 5*n2 - 3*n is W(n2). Why?
5*n2 - 3*n >= 4*n2, for all n >= 4

29. Q Notation: Asymptotic Tight Bound
T(n) = f(n) = Q(g(n))
if c1*g(n) <= f(n) <= c2*g(n) for all n > n0, where c1, c2 and n0 are constants > 0 n
f(n)
c1*g(n)
c2*g(n) n0
Example: T(n) = 2n + 5 is Q(n). Why?
2n <= 2n+5 <= 3n, for all n >= 5
T(n) = 5*n2 - 3*n is Q(n2). Why?
4*n2 <= 5*n2 - 3*n <= 5*n2, for all n >= 4

30. Big-Oh, Theta, Omega Tips to guide your intuition:
Think of O(f(N)) as “less than or equal to” f(N)
Upper bound: “grows slower than or same rate as” f(N)
Think of Ω(f(N)) as “greater than or equal to” f(N)
Lower bound: “grows faster than or same rate as” f(N)
Think of Θ(f(N)) as “equal to” f(N)
“Tight” bound: same growth rate
(True for large N and ignoring constant factors)

31. Common Functions we will encounter
Name
Big-Oh
Comment
Constant
O(1)
Can’t beat it!
Log log
O(loglogN)
Extrapolation search
Logarithmic
O(logN)
Typical time for good searching algorithms
Linear
O(N)
This is about the fastest that an algorithm can run given that we need O(n) just to read the input
N logN
O(NlogN)
Most sorting algorithms
Quadratic
O(N2)
Acceptable when the data size is small (N<1000)
Cubic
O(N3)
Exponential
O(2N)
Only good for really small input sizes (n<=20)
Increasing cost
Polynomial time

32. Time and Space Tradeoffs
In turns out in most algorithm design, there is a tradeoff between time and space
To make an algorithm faster, you might have to use more space
Trade space away (use less space), then the algorithm will run slower

33. 2.4 A Survey of Common Running Times

34. Linear Time: O(n)
Linear time. Running time is at most a constant factor times the size of the input.
Computing the maximum. Compute maximum of n numbers a1, …, an.
max  a1
for i = 2 to n {
if (ai > max)
max  ai }

35. Linear Time: O(n)
Merge. Combine two sorted lists A = a1,a2,…,an with B = b1,b2,…,bn into sorted whole.
Claim. Merging two lists of size n takes O(n) time.
Pf. After each comparison, the length of output list increases by 1.
i = 1, j = 1
while (both lists are nonempty) {
if (ai  bj) append ai to output list and increment i
else(ai  bj)append bj to output list and increment j }
append remainder of nonempty list to output list

36. O(n log n) Time
O(n log n) time. Arises in divide-and-conquer algorithms.
Sorting. Mergesort and heapsort are sorting algorithms that perform O(n log n) comparisons.
Largest empty interval. Given n time-stamps x1, …, xn on which copies of a file arrive at a server, what is largest interval of time when no copies of the file arrive?
O(n log n) solution. Sort the time-stamps. Scan the sorted list in order, identifying the maximum gap between successive time-stamps.
also referred to as linearithmic time

37. Quadratic Time: O(n2) Quadratic time. Enumerate all pairs of elements.
Closest pair of points. Given a list of n points in the plane (x1, y1), …, (xn, yn), find the pair that is closest.
O(n2) solution. Try all pairs of points.
Remark. (n2) seems inevitable, but this is just an illusion.
min  (x1 - x2)2 + (y1 - y2)2
for i = 1 to n {
for j = i+1 to n {
d  (xi - xj)2 + (yi - yj)2
if (d < min)
min  d }
don't need to take square roots
see chapter 5

38. Cubic Time: O(n3) Cubic time. Enumerate all triples of elements.
Set disjointness. Given n sets S1, …, Sn each of which is a subset of 1, 2, …, n, is there some pair of these which are disjoint?
O(n3) solution. For each pairs of sets, determine if they are disjoint.
foreach set Si {
foreach other set Sj {
foreach element p of Si {
determine whether p also belongs to Sj }
if (no element of Si belongs to Sj)
report that Si and Sj are disjoint

39. Polynomial Time: O(nk) Time
Independent set of size k. Given a graph, are there k nodes such that no two are joined by an edge?
O(nk) solution. Enumerate all subsets of k nodes.
Check whether S is an independent set = O(k2).
Number of k element subsets =
O(k2 nk / k!) = O(nk).
k is a constant
foreach subset S of k nodes {
check whether S in an independent set
if (S is an independent set)
report S is an independent set }
poly-time for k=17, but not practical

40. Exponential Time
Independent set. Given a graph, what is maximum size of an independent set?
O(n2 2n) solution. Enumerate all subsets.
S*  
foreach subset S of nodes {
check whether S in an independent set
if (S is largest independent set seen so far)
update S*  S }