1. Lecture 2 Analysis of Algorithms
How to estimate time complexity?
Analysis of algorithms
Techniques based on Recursions
ACKNOWLEDGEMENTS: Some contents in this lecture source from slides of Yuxi Fu

2. ROADMAP How to estimate time complexity? Analysis of algorithms
worst case, best case?
Big-O notation
9/3/2018
Xiaojuan Cai

3. Observation System dependent effects. Hardware: CPU, memory, cache, …
Software: compiler, interpreter, garbage collector, …
System: operating system, network, other applications, …
Easy, but difficult to get precise measurements.
9/3/2018
Xiaojuan Cai

4. Mathematical model
Total running time = sum of cost × frequency for all operations.
Cost depends on machine, compiler.
Frequency depends on algorithm, input data.
variable declaration 2
assignment statement 2
less than compare N+1
equal to compare N
array access N
increment N to 2N
The idea that we measure only the number of fundamental operations is a classical notion that dates back at least to Turing.
Rounding-off errors in matrix processing. A. M. Turing, QJ Mechanics Appl Math (1948)
% 1-sum
int count = 0;
for (int i = 0; i < N; i++)
if (a[i] == 0)
count++;
9/3/2018
Xiaojuan Cai

5. Mathematical model Relative rather than Absolute. Machine independent.
About large input instance
Time complexity of an algorithm is a function of the size of the input n.
9/3/2018
Xiaojuan Cai

6. Input size -- convention
Sorting and searching: the size of the array
Graph: the number of the vertices and edges
Computational geometry: the number points or line segments
Matrix operation: the dimensions of the input matrices
Number theory and Cryptography: the number of bits of the input.
9/3/2018
Xiaojuan Cai

7. Where are we? How to estimate time complexity? Analysis of algorithms
worst case, best case?
Big-O notation
9/3/2018
Xiaojuan Cai

8. Linear searching v.s. Binary searching
Problem: Searching
Input: An integer array A[1...n], and an integer x
Output: Does x exist in A?
Does 45 exist in:
9, 8, 9, 6, 2, 56, 12, 5, 4, 30, 67, 93, 25, 44, 96
2, 4, 5, 6, 8, 9, 9, 12, 25, 30, 44, 56, 67, 93, 96
Linear searching v.s. Binary searching
9/3/2018
Xiaojuan Cai

9. Binary searching
2, 4, 5, 6, 8, 9, 9, 12, 25, 30, 44, 56, 67, 93, 96
mid
low
high
9/3/2018
Xiaojuan Cai

10. Quiz Linear Searching Binary Searching Best case 1 Worst case n
log n+1
A B. logn. C. n D. none of above
9/3/2018
Xiaojuan Cai

11. Computational Complexity
Linear Searching
Binary Searching
Worst case n
log n
10-6s per instruction
Linear Searching
Binary Searching
n=1024
0.001s s
n=240
12.7day s
9/3/2018
Xiaojuan Cai

12. Where are we? How to estimate time complexity? Analysis of algorithms
worst case, best case?
Big-O notation
9/3/2018
Xiaojuan Cai

13. Big-O Notation
Definition (O-notation)
Let f(n) and g(n) :
f(n) is said to be O(g(n)), written f (n) = O(g(n)), if∃c>0.∃n0.∀n ≥ n0.f (n) ≤ cg(n)
The O-notation provides an upper bound of the running time;
f grows no faster than some constant times g.
9/3/2018
Xiaojuan Cai

14. Examples 10n2 + 20n = O(n2). logn2 =O(logn). log(n!) = O(nlogn).
Hanoi: T(n) = O(2n), n is the input size.
Searching: T(n) = O(n)
9/3/2018
Xiaojuan Cai

15. Big-Ω Notation
Definition (Ω-notation)
Let f(n) and g(n) :
f(n) is said to be Ω(g(n)), written f (n) = Ω(g(n)), if∃c>0.∃n0.∀n ≥ n0.f (n) ≥ cg(n)
The Ω-notation provides an lower bound of the running time;
f(n) = O(g(n)) iff g(n) = Ω(f(n)).
9/3/2018
Xiaojuan Cai

16. Does there exist f,g, such that f = O(g) and f = Ω(g)?
Examples
log nk = Ω(logn).
log n! = Ω(nlogn).
n! = Ω(2n).
Searching: T(n) = Ω(1)
Does there exist f,g, such that f = O(g) and f = Ω(g)?
9/3/2018
Xiaojuan Cai

17. Big-Θ notation log n! = Θ(nlogn). 10n2 + 20 n = Θ(n2)
Definition (Θ-notation)
f(n)=Θ(g(n)), if both f(n)=O(g(n)) and f(n) = Ω (g(n)).
log n! = Θ(nlogn).
10n n = Θ(n2)
9/3/2018
Xiaojuan Cai

18. Small o-notation log n! = o(nlogn)? 10n2 + 20 n = o(n2)?
Definition (o-notation)
Let f(n) and g(n) :
f(n) is said to be o(g(n)), written f (n) = o(g(n)), if∀c.∃n0.∀n ≥ n0.f (n) < cg(n)
$1\prec \log\log n \prec log n\prec \sqrt{n}\prec n^{3/4} \prec n \prec n\log n \prec n^2 \prec 2^n \prec n!\prec 2^{n^2}$
log n! = o(nlogn)?
10n n = o(n2)?
nlog n = o(n2)?
9/3/2018
Xiaojuan Cai

19. Small ω-notation Definition (ω-notation) Let f(n) and g(n) :
f(n) is said to be ω(g(n)), written f (n) = ω(g(n)), if∀c.∃n0.∀n ≥ n0.f (n) > cg(n)
9/3/2018
Xiaojuan Cai

20. Quiz f(n) g(n) n1.01 nlog2n nlogn log(n!) logn n2n 3n
A. O B. Ω. C. Θ D. none of above
9/3/2018
Xiaojuan Cai

21. Approximation by Integration
\[\int_{m-1}^{n}f(x)dx \le \sum_{j=m}^{n}f(j)\le \int_{m}^{n+1}f(x)dx\]
\[\int_{m}^{n+1}f(x)dx \le \sum_{j=m}^{n}f(j)\le \int_{m-1}^{n}f(x)dx\]
\[\sum_{j=1}^{n}j^{k} = \Theta(n^{k+1})\]
\[H_{n}=\sum_{j=1}^{n}\frac{1}{j}=\Theta(\ln n)=\Theta(\log
n)\]
\[\sum_{j=1}^{n}\log j = \Theta(n\log n)\]
9/3/2018
Xiaojuan Cai

22. In terms of limits Suppose exists 9/3/2018 Xiaojuan Cai
\begin{eqnarray*}
lim_{n\rightarrow\infty} \frac{f(n)}{g(n)} \not = \infty & {\ implies\ } & f(n) = O(g(n))\\
lim_{n\rightarrow\infty} \frac{f(n)}{g(n)} \not = 0 & {\ implies\ } & f(n) = \Omega(g(n))\\
lim_{n\rightarrow\infty} \frac{f(n)}{g(n)} = c & {\ implies\ } & f(n) = \theta(g(n))\\
lim_{n\rightarrow\infty} \frac{f(n)}{g(n)} = 0 & {\ implies\ } & f(n) = o(g(n))\\
lim_{n\rightarrow\infty} \frac{f(n)}{g(n)} = \infty & {\ implies\ } & f(n) = \omega(g(n))
\end{eqnarray*}
9/3/2018
Xiaojuan Cai

23. f Rg if and only if f (n) = Θ(g (n)).
Complexity class
Definition (Complexity class)
An equivalence relation R on the set of complexity functions is defined as follows:
f Rg if and only if f (n) = Θ(g (n)).
9/3/2018
Xiaojuan Cai

24. Assume the computer consumes 10-6s per instruction
Order of growth
order
name
example
N = 1000
N = 2000 1
constant
add two numbers
instant
log N
logarithmic
binary search N
linear
find the maximum
N log N
linearithmic
mergesort N2
quadratic
2-sum
~1s
~2s 2N
exponential
Hanoi
forever
Assume the computer consumes 10-6s per instruction
9/3/2018
Xiaojuan Cai

25. Order of growth
9/3/2018
Xiaojuan Cai

26. Quiz A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above 9/3/2018
\[n + \frac{n}{2} + \frac{n}{2^2} + \cdots + \frac{n}{2^j} + \cdots + \frac{n}{2^{\log n}} = 2n - 1 = \theta(n)\]
A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above
9/3/2018
Xiaojuan Cai

27. Quiz A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above 9/3/2018
\[n + \frac{n}{2} + \frac{n}{2^2} + \cdots + \frac{n}{2^j} + \cdots + \frac{n}{2^{\log n}} = 2n - 1 = \theta(n)\]
A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above
9/3/2018
Xiaojuan Cai

28. Quiz A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above 9/3/2018
\[\sum_{i=1}^{n}(\frac{n}{i}-1) \le \sum_{i=1}^{n}\lfloor\frac{n}{i}\rfloor \le \sum_{i=1}^{n}\frac{n}{i} \]
A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above
9/3/2018
Xiaojuan Cai

29. Quiz A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above 9/3/2018
\[\sum_{i=1}^{n}(\frac{n}{i}-1) \le \sum_{i=1}^{n}\lfloor\frac{n}{i}\rfloor \le \sum_{i=1}^{n}\frac{n}{i} \]
A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above
9/3/2018
Xiaojuan Cai

30. Quiz A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above 9/3/2018
\[\sum_{i=1}^{n}(\frac{n}{i}-1) \le \sum_{i=1}^{n}\lfloor\frac{n}{i}\rfloor \le \sum_{i=1}^{n}\frac{n}{i} \]
A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above
9/3/2018
Xiaojuan Cai

31. Quiz θ(nloglogn) A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above
\[\sum_{i=1}^{n}(\frac{n}{i}-1) \le \sum_{i=1}^{n}\lfloor\frac{n}{i}\rfloor \le \sum_{i=1}^{n}\frac{n}{i} \]
θ(nloglogn)
A. Θ(n2) B. Θ(n). C. Θ(nlogn) D. none of above
9/3/2018
Xiaojuan Cai

32. Amortized analysis The total number of append is n.
The total number of delete is between 0 and n-1. So the time complexity is O(n).
9/3/2018
Xiaojuan Cai

33. Memory Time: Time complexity Memory: Space complexity
faster, better.
Memory: Space complexity
less, better.
Space complexity <= Time complexity
9/3/2018
Xiaojuan Cai

34. Conclusion How to estimate time complexity? Analysis of algorithms
worst case, best case?
Big-O notation
9/3/2018
Xiaojuan Cai