1. COSC 3100 Fundamentals of Analysis of Algorithm Efficiency
Instructor: Tanvir

2. Outline (Ch 2) Input size, Orders of growth,
Worst-case, Best-case, Average-case (Sec 2.1)
Asymptotic notations and Basic efficiency classes (Sec 2.2)
Analysis of Nonrecursive Algorithms (Sec 2.3)
Analysis of Recursive Algoithms (Sec 2.4)
Example: Computing n-th Fibonacci Number (Sec 2.5)
Empirical Analysis, Algorithm Visualization (Sec 2.6, 2.7)

3. Analysis of Algoithms
An investigation of an algorithm’s efficiency with respect to two resources:
Running time (time efficiency or time complexity)
Memory space (space efficiency or space complexity)

4. Analysis Framework Measuring an input’s size Measuring Running Time
Orders of Growth (of algorithm’s efficiency function)
Worst-case, Best-case, and Average-case efficiency

5. Measuring Input Size Efficiency is defined as a function of input size
Input size depends on the problem
Example 1: what is the input size for sorting n numbers?
Example 2: what is the input size for evaluating 𝑝 𝑥 = 𝑎 𝑛 𝑥 𝑛 + 𝑎 𝑛−1 𝑥 𝑛−1 +⋯ + 𝑎 1 𝑥+ 𝑎 0
Example 3: what is the input size for multiplying two n×n matrices?

6. Units of measuring running time
Measure running time in milliseconds, seconds, etc.
Depends on which computer
Count the number of times each operation is executed
Difficult and unnecessary
Count the number of times an algorithm’s “basic operation” is executed

7. Measure running time in terms of # of basic operations
Basic operation: the operation that contributes the most to the total running time of an algorithm
Usually the most time consuming operation in the algorithm’s innermost loop

8. Input size and basic operation examples
Problem
Measure of input size
Basic operation
Search for a key in a list of n items
# of items in the list
Key comparison
Add two n×n matrices
Dimensions of the matrices, n
Addition
Polynomial evaluation
Order of the polynomial
Multiplication

9. Theoretical Analysis of Time Efficiency
Count the number of times the algorithm’s basic operation is executed on inputs of size n: C(n)
T(n) ≈ cop × C(n)
Input size
Ignore cop,
Focus on
orders of growth
# of times basic op.
is executed
Running time
Execution time for
basic operation
If C(n) = 𝟏 𝟐 𝒏 𝒏−𝟏 , how much longer it runs if the input size is doubled?

10. Orders of Growth
Why do we care about the order of growth of an algorithm’s efficiency function, i.e., the total number of basic operations?
Euclid’s
Consecutive Integer Checking
gcd(60, 24) 3 13
gcd(31415, 14142) 10
14142
gcd( , ) 97
> 1020
How fast efficiency function grows
as n gets larger and larger…

11. Orders of Growth (contd.)
lgn
n×lgn n2 n3 2n n! 10
3.3
3.3×10
102
103
3.6×106
6.6
6.6×102
104
106
1.3×1030
9.3×10157
10×103
109 13
13×104
108
1012
105 17
17×105
1010
1015 20
20×106
1018

12. Orders of Growth (contd.)
Plots of growth…
Consider only the leading term
Ignore the constant coefficients

13. Worst, Best, Average Cases
Efficiency depends on input size n
For some algorithms, efficiency depends on the type of input
Example: Sequential Search
Given a list of n elements and a search key k, find if k is in the list
Scan list, compare elements with k until either found a match (success), or list is exhausted (failure)

14. Sequential Search Algorithm
ALGORITHM SequentialSearch(A[0..n-1], k) //Input: A[0..n-1] and k //Output: Index of first match or -1 if no match is //found i <- 0 while i < n and A[i] ≠ k do i <- i+1 if i < n return i //A[i] = k else return -1

15. Different cases Worst case efficiency Best case efficiency
Efficiency (# of times the basic op. will be executed) for the worst case input of size n
Runs longest among all possible inputs of size n
Best case efficiency
Runs fastest among all possible inputs of size n
Average case efficiency
Efficiency for a typical/random input of size n
NOT the average of worst and best cases
How do we find average case efficiency?

16. Average Case of Sequential Search
Two assumptions:
Probability of successful search is p (0 ≤ p ≤ 1)
Search key can be at any index with equal probability (uniform distribution)
Cavg(n) = Expected # of comparisons = Expected # of comparisons for success + Expected # of comparisons if k is not in the list

17. Summary of Analysis Framework
Time and space efficiencies are functions of input size
Time efficiency is # of times basic operation is executed
Space efficiency is # of extra memory units consumed
Efficiencies of some algorithms depend on type of input: requiring worst, best, average case analysis
Focus is on order of growth of running time (or extra memory units) as input size goes to infinity

18. Asymptotic Growth Rate
3 notations used to compare orders of growth of an algorithm’s basic operation count
O(g(n)): Set of functions that grow no faster than g(n)
Ω(g(n)): Set of functions that grow at least as fast as g(n)
Θ(g(n)): Set of functions that grow at the same rate as g(n)

19. O(big oh)-Notation
c × g(n)
t(n)
Doesn’t
matter n n0
t(n) є O(g(n))

20. t(n) ≤ c × g(n) for all n ≥ n0
O-Notation (contd.)
Definition: A function t(n) is said to be in O(g(n)), denoted t(n) є O(g(n)), if t(n) is bounded above by some positive constant multiple of g(n) for sufficiently large n.
If we can find +ve constants c and n0 such that:
t(n) ≤ c × g(n) for all n ≥ n0

21. O-Notation (contd.) Is 100n+5 є O(n2) ?
Is 2n+1 є O(2n) ? Is 22n є O(2n) ?
Is 1 2 n(n-1) є O(n2) ?

22. Ω(big omega)-Notation
t(n)
c × g(n)
Doesn’t
matter n n0
t(n) є Ω(g(n))

23. t(n) ≥ c × g(n) for all n ≥ n0
Ω-Notation (contd.)
Definition: A function t(n) is said to be in Ω(g(n)) denoted t(n) є Ω(g(n)), if t(n) is bounded below by some positive constant multiple of g(n) for all sufficiently large n.
If we can find +ve constants c and n0 such that
t(n) ≥ c × g(n) for all n ≥ n0

24. Ω-Notation (contd.) Is n3 є Ω(n2) ? Is 100n+5 є Ω(n2) ?
Is 1 2 n(n-1) є Ω(n2) ?

25. Θ(big theta)-Notation
c1 × g(n)
t(n)
c2 × g(n)
Doesn’t
matter n n0
t(n) є Θ(g(n))

26. c2×g(n) ≤ t(n) ≤ c1×g(n) for all n ≥ n0
Θ-Notation (contd.)
Definition: A function t(n) is said to be in Θ(g(n)) denoted t(n) є Θ(g(n)), if t(n) is bounded both above and below by some positive constant multiples of g(n) for all sufficiently large n.
If we can find +ve constants c1, c2, and n0 such that
c2×g(n) ≤ t(n) ≤ c1×g(n) for all n ≥ n0

27. Θ-Notation (contd.) Is 1 2 n(n-1) є Θ(n2) ? Is n2+sin(n) є Θ(n2) ?
Is an2+bn+c є Θ(n2) for a > 0?
Is (n+a)b Θ(nb) for b > 0 ?

28. O, Θ, and Ω (≥) Ω(g(n)): functions that (=) Θ(g(n)): functions that
grow at least as fast as g(n)
(=) Θ(g(n)): functions that
grow at the same rate as g(n)
g(n)
(≤) O(g(n)): functions that
grow no faster that g(n)

29. t1(n) + t2(n) є O(max{g1(n),g2(n)})
Theorem
If t1(n) є O(g1(n)) and t2(n) є O(g2(n)), then
t1(n) + t2(n) є O(max{g1(n),g2(n)})
Analogous assertions are true for Ω and Θ notations.
Implication: if sorting makes no more than n2 comparisons and then binary search makes no more than log2n comparisons, then efficiency is O(max{n2, log2n}) = O(n2)

30. t1(n) + t2(n) є O(max{g1(n),g2(n)})
Theorem (contd.)
If t1(n) є O(g1(n)) and t2(n) є O(g2(n)), then
t1(n) + t2(n) є O(max{g1(n),g2(n)})
t1(n) ≤ c1g1(n) for n ≥ n01 and t2(n) ≤ c2g2(n) for n ≥ n02
t1(n) + t2(n) ≤ c1g1(n) + c2g2(n) for n ≥ max { n01, n02 }
t1(n) + t2(n) ≤ max{c1, c2} g1(n) + max{c1, c2} g2(n) for n ≥ max { n01, n02 }
t1(n) + t2(n) ≤ 2max{c1, c2} max{g1(n), g2(n) },
for n ≥ max{n01, n02} c n0

31. Using Limits for Comparing Orders of Growth
0 implies that t(n) grows slower than g(n)
c implies that t(n) grows at the same order
as g(n)
∞ implies that t(n) grows faster than g(n)
1. First two cases (0 and c) means t(n) є O(g(n))
2. Last two cases (c and ∞) means t(n) є Ω(g(n))
3. The second case (c) means t(n) є Θ(g(n))

32. Taking Limits
L’Hôpital’s rule: if limits of both t(n) and g(n) are ∞ as n goes to ∞,
lim 𝑛→∞ 𝑡(𝑛) 𝑔(𝑛) = lim 𝑛→∞ 𝑡′(𝑛) 𝑔′(𝑛) = lim 𝑛→∞ 𝑑𝑡(𝑛) 𝑑𝑛 𝑑𝑔(𝑛) 𝑑𝑛
Stirling’s formula: For large n, we have n! ≈ 2𝜋𝑛 𝑛 𝑒 𝑛

33. Stirling’s Formula n! = n (n-1) (n-2) … 1 ln(n!) = 𝑘=1 𝑛 ln⁡(𝑘)
Trapezoidal rule: 1 𝑛 ln 𝑥 𝑑𝑥 ≈ ln ln …+ ln 𝑛− ln⁡(𝑛) 1 𝑛 ln 𝑥 𝑑𝑥 ≈ ln ln …+ ln 𝑛− ln⁡(𝑛)
Add 1 2 ln⁡(𝑛) to both sides,
𝑛𝑙𝑛 𝑛 −𝑛 ln⁡(𝑛) ≈ 𝑘=1 𝑛 ln⁡(𝑘)
Exponentiate both sides, n! ≈ C nne-nn1/ = C 𝒏 𝒏 𝒆 𝒏 … 1 2 3 4 n

34. Taking Limits (contd.) Compare orders of growth of 22n and 3n
Compare orders of growth of lgn and 𝒏
Compare orders of growth of n! and 2n

35. Summary of How to Establish Order of Growth of Basic Operation Count
Method 1: Using Limits
Method 2: Using Theorem
Method 3: Using definitions of O-, Ω-, and Θ-notations.

36. Basic Efficiency Classes
Name
Comments 1
constant
May be in best cases
lgn
logarithmic
Halving problem size at each iteration n
linear
Scan a list of size n
n×lgn
linearithmic
Divide and conquer algorithms, e.g., mergesort n2
quadratic
Two embedded loops, e.g., selection sort n3
cubic
Three embedded loops, e.g., matrix multiplication 2n
exponential
All subsets of n-elements set n!
factorial
All permutations of an n-elements set

37. Important Summation Formulas
𝑖=𝑙 𝑢 1 =𝑢−𝑙+1
𝑖=1 𝑛 𝑖 = 𝑛(𝑛+1) 2 ≈ 𝑛 2
𝑖=1 𝑛 𝑖 2 = 𝑛(𝑛+1)(2𝑛+1) 6 ≈ 𝑛 3
𝑖=0 𝑛 𝑎 𝑖 = 𝑎 𝑛+1 −1 𝑎−1 (𝑎≠1)
𝑖=1 𝑛 𝑖 𝑎 𝑖 =𝑎 𝑑 𝑑𝑎 𝑎 𝑛+1 −1 𝑎−1
𝒊=𝒍 𝒖 𝒄 𝒂 𝒊 =𝒄 𝒊=𝒍 𝒖 𝒂 𝒊
𝒊=𝒍 𝒖 𝒂 𝒊 ± 𝒃 𝒊 = 𝒊=𝒍 𝒖 𝒂 𝒊 ± 𝒊=𝒍 𝒖 𝒃 𝒊

38. Analysis of Nonrecursive Algorithms
ALGORITHM MaxElement(A[0..n-1]) //Determines largest element maxval <- A[0] for i <- 1 to n-1 do if A[i] > maxval maxval <- A[i] return maxval
Input size: n
Basic operation: > or <-
C(n) = 𝒊=𝟏 𝒏−𝟏 𝟏 =𝒏−𝟏−𝟏+𝟏
=𝒏−𝟏 є Θ(n)

39. General Plan for Analysis of Nonrecursive algorithms
Decide on a parameter indicating an input’s size
Identify the basic operation
Does C(n) depends only on n or does it also depend on type of input?
Set up sum expressing the # of times basic operation is executed.
Find closed form for the sum or at least establish its order of growth

40. Analysis of Nonrecursive (contd.)
ALGORITHM UniqueElements(A[0..n-1]) //Determines whether all elements are //distinct for i <- 0 to n-2 do for j <- i+1 to n-1 do if A[i] = A[j] return false return true
Input size: n
Basic operation: A[i] = A[j]
Does C(n) depend on type of input?

41. UniqueElements (contd.)
for i <- 0 to n-2 do
for j <- i+1 to n-1 do
if A[i] = A[j]
return false
return true
Cworst(n) = 𝑖=0 𝑛−2 𝑗=𝑖+1 𝑛−1 1 = 𝑛−1 𝑛 2 ≈ 1 2 𝑛 2 ∈Θ 𝑛 2
Why Cworst(n) ∈𝜣 𝒏 𝟐 is better than
saying Cworst(n) ∈𝑶 𝒏 𝟐 ?

42. Analysis of Nonrecursive (contd.)
ALGORITHM MatrixMultiplication(A[0..n-1, 0..n-1], B[0..n-1, 0..n-1])
//Output: C = AB
for i <- 0 to n-1 do
for j <- 0 to n-1 do
C[i, j] = 0.0
for k <- 0 to n-1 do
C[i, j] = C[i, j] + A[i, k]×B[k, j]
return C
T(n) ≈ cmM(n)+caA(n)
= cmn3+can3
= (cm+ca)n3
Input size: n
Basic operation: ×
M(n) = 𝒊=𝟎 𝒏−𝟏 𝒋=𝟎 𝒏−𝟏 𝒌=𝟎 𝒏−𝟏 𝟏

43. Analysis of Nonrecursive (contd.)
ALGORITHM Binary(n) // Output: # of digits in binary // representation of n count <- 1 while n > 1 do count <- count+1 n <- 𝑛 2 return count
Input size: n
Basic operation:
C(n) = ?
Each iteration halves n
Let m be such that 2m ≤ n < 2m+1
Take lg: m ≤ lg(n) < m+1, so
m = 𝐥𝐠⁡(𝒏) and m+1 = 𝐥𝐠⁡(𝒏) +1 є Θ(lg(n))

44. Analysis of Nonrecursive (contd.)
ALGORITHM Mystery(n) S <- 0 for i <- 1 to n do S <- S + i×i return S
What does this algorithm compute?
What is the basic operation?
How many times is the basic operation executed?
What’s its efficiency class?
Can you improve it further, or
Can you prove that no improvement
is possible?

45. Analysis of Nonrecursive (contd.)
ALGORITHM Secret(A[0..n-1]) r <- A[0] s <- A[0] for i <- 1 to n-1 do if A[i] < r r <- A[i] if A[i] > s s <- A[i] return s-r
What does this algorithm compute?
What is the basic operation?
How many times is the
basic operation executed?
What’s its efficiency class?
Can you improve it further,
Or Could you prove that
no improvement is possible?

46. Analysis of Recursive Algorithms
ALGORITHM F(n) // Output: n! if n = 0 return 1 else return n×F(n-1)
Input size: n
Basic operation: ×
M(n) = M(n-1) + 1 for n > 0
to compute
F(n-1)
to multiply
n and F(n-1)

47. Analysis of Recursive (contd.)
M(n) = M(n-1) + 1
M(0) = 0
M(n) = M(n-1) + 1 = M(n-2) = … …
= M(n-n) … = 0 + n = n
We could compute it nonrecursively, saves function call overhead
Recurrence relation
By Backward substitution
n 1’s

48. General Plan for Recursive
Decide on input size parameter
Identify the basic operation
Does C(n) depends also on input type?
Set up a recurrence relation
Solve the recurrence or, at least establish the order of growth of its solution

49. Analysis of Recursive (contd.): Tower of Hanoi

50. Tower of Hanoi (contd.) M(n-1) M(n) = M(n-1) + 1 + M(n-1) M(1) = 1

51. Tower of Hanoi (contd.)
ALGORITHM ToH(n, s, m, d) if n = 1 print “move from s to d” else ToH(n-1, s, d, m) ToH(n-1, m, s, d)
M(1) = 1
M(n-1)
M(1)
M(n-1)

52. Tower of Hanoi (contd.) M(n) = 2M(n-1) + 1 for n > 1 M(1) = 1
M(n) = 2M(n-1) + 1 [Backward substitution…]
= 2[ 2M(n-2)+1] + 1 = 22M(n-2)+2+1
= 22[2M(n-3)+1]+2+1 = 23M(n-3)
… … …
= 2n-1M(n-(n-1))+2n-2+2n-3+……
= 2n-1+ 2n-2+2n-3+…… = 𝑖=0 𝑛−1 2 𝑖 = 2n-1
M(n) є Θ(2n)
Be careful! Recursive algorithm may look simple
But might easily be exponential in complexity.

53. Tower of Hanoi (contd.) Recursion tree (# of function calls)… 2 1
C(n) = 𝒍=𝟎 𝒏−𝟏 𝟐 𝒍 = 2n-1

54. Analysis of Recursive (contd.)
ALGORITHM BinRec(n) //Output: # of binary digits in n’s //binary representation if n = 1 return 1 else return BinRec( 𝑛 2 )+1
Input size: n = 2k
Basic operation: +
A(n) = A(2k) = A(2k-1) + 1 = [A(2k-2) + 1] + 1
= … = A(2k-k)+k = A(1) + k = k = lg(n)
A(n) є Θ(lg(n))

55. Analysis of Recursive (contd.)
Maximum how many slices of pizza can a person obtain by making n straight cuts with a pizza knife?
L1 = 2
L2 = 4
L0 = 1
difference 1 2 3
L3 = 7
1, 2, 4, 7, …
Ln = Ln-1 + n for n > 0
L0 = 1
Bonus problem in HW2

56. EXAMPLE: nth Fibonacci Number
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, …
F(n) = F(n-1) + F(n-2) for n > 1
F(0) = 0, F(1) = 1
Let’s try backward substitution…
F(n) = F(n-2)+F(n-3)+F(n-3)+F(n-4)
= F(n-2)+2F(n-3)+F(n-4)
= F(n-3)+3F(n-4)+3F(n-5)+F(n-6) …

57. nth Fibonacci (contd.)
ax(n)+bx(n-1)+cx(n-2) = 0 Homogeneous linear second-order recurrence with constant coefficients Guess: x(n) = rn arn+brn-1+crn-2 = 0 ar2+br+c = 0 THEOREM: If r1, r2 are two real distinct roots of characteristic equation then x(n) = αr1n+βr2n where α, β are arbitrary constants
Characteristic equation

58. nth Fibonacci (contd.)
F(n)-F(n-1)-F(n-2) = 0 r2-r-1 = 0 => r1,2 = 1± 1−4 −1 2 = 1± 5 2 F(n) = α 𝑛 +β 1− 5 2 𝑛 F(0) = α β 1− = 0 => α+β=0 F(1) = α β 1− = 1 F(n) = 𝑛 − 1 5 1− 5 2 𝑛 = 1 5 𝜑 𝑛 − 𝜑 𝑛 where 𝜑= ≈ and 𝜑 = - 1 𝜑

59. nth Fibonacci (contd.)
F(n) = 𝜑 𝑛 − 𝜑 𝑛 where 𝜑 ≈ and 𝜑 ≈
F(n) є Θ(φn)
ALGORITHM F(n)
if n ≤ 1
return n
else
return F(n-1)+F(n-2)
A(n) = A(n-1)+A(n-2)+1
A(0) = 0, A(1) = 0
[A(n)+1] –[A(n-1)+1]-[A(n-2)+1]=0
Let B(n) = A(n)+1
B(n) –B(n-1)-B(n-2)=0
B(0)=1, B(1)=1
Notice, B(n) = F(n+1)
A(n) = B(n)-1=F(n+1)-1
= 𝜑 𝑛+1 − 𝜑 𝑛+1 -1
A(n) є Θ(φn)
Basic op: +

60. nth Fibonacci (contd.) F(5) F(4) F(3) F(2) F(1) F(0)
Only n-1 additions, Θ(n)!
ALGORITHM Fib(n)
F[0] <- 0
F[1] <- 1
for i <- 2 to n do
F[i] <- F[i-1]+F[i-2]
return F[n]
ALGORITHM Fib(n)
f <- 0
fnext <- 1
for i <- 2 to n do
tmp <- fnext
fnext <- fnext+f
f <- tmp
return fnext

61. Empirical Analysis Mathematical Analysis Empirical Analysis Advantages
Machine and input independence
Disadvantages
Average case analysis is hard
Empirical Analysis
Applicable to any algorithm
Machine and input dependency

62. Empirical Analysis (contd.)
General Plan
Understand Experiment’s purpose:
What is the efficiency class?
Compare two algorithms for same problem
Efficiency metric: Operation count vs. time unit
Characteristic of input sample (range, size, etc.)
Write a program
Generate sample inputs
Run on sample inputs and record data
Analyze data

63. Empirical Analysis (contd.)
Insert counters into the program to count C(n)
Or, time the program, (tfinish-tstart), in Java System.currentTimeMillis()
Typically not very accurate, may vary
Remedy: make several measurements and take mean or median
May give 0 time, remedy: run in a loop and take mean

64. Empirical Analysis (contd.)
Increase sample size systematically, like 1000, 2000, 3000, …, 10000
Impact is easier to analyze
Better is to generate random sizes within desired range
Because, odd sizes may affect
Pseudorandom number generators
In Java Math.random() gives uniform random number in [0, 1)
If you need number in [min, max]:
min + (int)( Math.random() * ( (max-min) + 1 ) )

65. Empirical Analysis (contd.)
Pseudorandom in [min, max]
Math.random() * (max-min) gives [0, max-min)
min + Math.random() * (max-min) gives [min, max)
(int)[Math.random() * (max-min+1)] gives [0, max-min]
min+ (int)[Math.random() * (max-min+1)] gives [min, max]
To get [5, 10]
(int)Math.random()*(10-5+1) gives [0, 10-5] or [0, 5]
5+ (int)Math.random()*(10-5+1) gives [5, 10]

66. Empirical Analysis (contd.)
Profiling: find out most time-consuming portion, Eclipse and Netbeans have it
Tabulate data
If M(n)/g(n) converges to a +ve constant, we know M(n) є Θ(g(n))
Or look at M(2n)/M(n); e.g., if M(n) є Θ(lgn) then M(2n)/M(n) will be low n
M(n)
g(n)
M(n)/g(n)

67. Empirical Analysis (contd.)
Scatterplot
M(n) ≈ can
lgM(n) ≈ nlga + lgc
count/time
count/time n n n
lgn
count/time
nlgn or n2 n

68. Algorithm Visualization (contd.)
Sequence of still pictures or animation
Sorting Out Sorting on YouTube
Nice animation of 9 sorting algorithms

69. Summary Time and Space Efficiency
C(n): Count of # of times the basic operation is executed for input of size n
C(n) may depend on type of input and then we need worst, average, and best case analysis
Usually order of growth (O, Ω, Θ) is all that matters: logarithmic, linear, linearithmic, quadratic, cubic, and exponential
Input size, basic op., worst case?, sum or recurrence, solve it…
We run on computers for empirical analysis

70. Homework 2
DUE ON FEBRUARY 07 (TUESDAY) IN CLASS