1. Thinking about Algorithms Abstractly
Lecture 2. Relevant Mathematics: Classifying Functions
Input Size
Time

2. Assumed Knowledge See J. Edmonds, How to Think About Algorithms (HTA):
Chapter 22. Existential and Universal Quantifiers
Chapter 24. Logarithms and Exponentials
COSC 3101B, PROF. J. ELDER

3. Classifying Functions
Giving an idea of how fast a function grows without going into too much detail.

4. Which are more alike?
COSC 3101B, PROF. J. ELDER

5. Which are more alike?
Mammals
COSC 3101B, PROF. J. ELDER

6. Which are more alike?
COSC 3101B, PROF. J. ELDER

7. Which are more alike?
Dogs
COSC 3101B, PROF. J. ELDER

8. Classifying Animals Vertebrates Fish Reptiles Birds Mammals Dogs
Giraffe
COSC 3101B, PROF. J. ELDER

9. Classifying Functions
T(n) 10
100
1,000
10,000
log n   3 6 9 13
n1/2 3 10 31
100 n 10
100
1,000
10,000
n log n 30
600
9,000
130,000 n2
100
10,000
106
108 n3
1,000
106
109
1012 2n
1,024
1030
10300
103000
Note: The universe contains approximately 1050 particles.
COSC 3101B, PROF. J. ELDER

10. Which are more alike?
n1000 n2 2n
COSC 3101B, PROF. J. ELDER

11. Which are more alike? n1000 n2 2n Polynomials
COSC 3101B, PROF. J. ELDER

12. Which are more alike?
1000n2
3n2
2n3
COSC 3101B, PROF. J. ELDER

13. Which are more alike? 1000n2 3n2 2n3 Quadratic
COSC 3101B, PROF. J. ELDER

14. Classifying Functions?
COSC 3101B, PROF. J. ELDER

15. Classifying Functions
Exp
Constant
Polynomial
Logarithmic
Exponential
Double Exp
Poly Logarithmic
25n 2 2 n5 5
5 log n
(log n)5 n5
25n
COSC 3101B, PROF. J. ELDER

16. Classifying Functions?
Polynomial
COSC 3101B, PROF. J. ELDER

17. Classifying Functions
Polynomial ?
Cubic
Linear
Quadratic 5n
5n2
5n3
5n4
COSC 3101B, PROF. J. ELDER

18. Differ only by a multiplicative constant.
Logarithmic
log10n = # digits to write n
log2n = # bits to write n = 3.32 log10n
log(n1000) = 1000 log(n)
Differ only by a multiplicative constant.
COSC 3101B, PROF. J. ELDER

19. Poly Logarithmic
(log n)5 = log5 n
COSC 3101B, PROF. J. ELDER

20. Logarithmic << Polynomial
log1000 n << n0.001
For sufficiently large n
COSC 3101B, PROF. J. ELDER

21. Linear << Quadratic
10000 n << n2
For sufficiently large n
COSC 3101B, PROF. J. ELDER

22. Polynomial << Exponential
n1000 << n
For sufficiently large n
COSC 3101B, PROF. J. ELDER

23. << << << << << << <<
Ordering Functions
Functions
Exp
Constant
Logarithmic
<<
<<
Polynomial
<<
Exponential
<<
<<
Double Exp
<<
Poly Logarithmic
25n 2 2 n5 5
5 log n
(log n)5 n5
25n
<<
<<
<<
<<
<<
<<
COSC 3101B, PROF. J. ELDER

24. Which Functions are Constant?
Yes No 5
1,000,000,000,000 -5
8 + sin(n)
COSC 3101B, PROF. J. ELDER

25. Which Functions are “Constant”?
The running time of the algorithm is a “Constant” It does not depend significantly on the size of the input.
Yes 5
1,000,000,000,000 -5
8 + sin(n)
Yes
Yes
Lies in between 7 9
Yes No No
COSC 3101B, PROF. J. ELDER

26. End of Lecture 1
Sept 7, 2006

27. CSE 3101B Lecture 2
Sept 12, 2006

28. The Time Complexity of an Algorithm
Specifies how the running time depends on the size of the input.

29. Purpose To estimate how long a program will run.
To estimate the largest input that can reasonably be given to the program.
To compare the efficiency of different algorithms.
To help focus on the parts of code that are executed the largest number of times.
To choose an algorithm for an application.
COSC 3101B, PROF. J. ELDER

30. Time Complexity Is a Function
Specifies how the running time depends on the size of the input.
A function mapping “size” of input  “time” T(n) executed .
COSC 3101B, PROF. J. ELDER

31. Example: Insertion Sort
COSC 3101B, PROF. J. ELDER

32. Example: Insertion Sort
COSC 3101B, PROF. J. ELDER

33. Example: Insertion Sort
COSC 3101B, PROF. J. ELDER

34. Example: Merge Sort
COSC 3101B, PROF. J. ELDER

35. COSC 3101B, PROF. J. ELDER

36. Example: Merge Sort
COSC 3101B, PROF. J. ELDER

37. Example: Merge Sort
COSC 3101B, PROF. J. ELDER

38. Classifying Functions
Exp
Constant
Polynomial
Logarithmic
Exponential
Double Exp
Poly Logarithmic 2
nθ(1)
2θ(n) 2
θ(1)
θ(log n)
(log n)θ(1)
nθ(1)
2θ(n)
COSC 3101B, PROF. J. ELDER

39. Classifying Functions
T(n) 10
100
1,000
10,000
log n   3 6 9 13
n1/2 3 10 31
100 n 10
100
1,000
10,000
n log n 30
600
9,000
130,000 n2
100
10,000
106
108 n3
1,000
106
109
1012 2n
1,024
1030
10300
103000 80
Note: The universe contains approximately 1050 particles.
COSC 3101B, PROF. J. ELDER

40. Digression: The Number of Particles in the Universe
= 2×136×2256 = 2×136×2223 = 17×2260
≅ ×1079
This is the Eddington number. According to Arthur Eddington in his book Mathematical Theory of Relativity (1923, London, Cambridge University press), it is the number of particles in the universe.
COSC 3101B, PROF. J. ELDER

41. Digression: Number of Particles in the Universe
Many other estimates of the number of particles in the universe have been computed, all in the range from 1078 to Here is an example (which is way too simplistic for physicists but shows that the Eddington number was fairly close):
r = radius of visible universe = age of universe × speed of light = speed of light / Hubble constant = 1.42 × 1026 meters
volume of universe = 4/3 π r3 = 1.2 × 1079 cubic meters
average density of universe = 3 hydrogen atoms per cubic meter (from models that give the minimum mass of a "closed" universe) 1 hydrogen atom = 4 particles (proton + electron, a proton is 3 quarks)
number of particles = 1.44 × 1080
Source: Robert Munfao (
COSC 3101B, PROF. J. ELDER

42. Classifying Functions
Exp
Constant
Polynomial
Logarithmic
Exponential
Double Exp
Poly Logarithmic 2
nθ(1)
2θ(n) 2
θ(1)
θ(log n)
(log n)θ(1)
nθ(1)
2θ(n)
COSC 3101B, PROF. J. ELDER

43. Which Functions are Constant?
Yes No 5
1,000,000,000,000 -5
8 + sin(n)
COSC 3101B, PROF. J. ELDER

44. Which Functions are “Constant”?
The running time of the algorithm is a “Constant” It does not depend significantly on the size of the input.
Yes 5
1,000,000,000,000 -5
8 + sin(n)
Yes
Yes
Lies in between 7 9
Yes No No
COSC 3101B, PROF. J. ELDER

45. Which Functions are Quadratic?
… ?
COSC 3101B, PROF. J. ELDER

46. Which Functions are Quadratic?
Some constant times n2.
COSC 3101B, PROF. J. ELDER

47. Which Functions are Quadratic?
5n2 + 3n + 2log n
COSC 3101B, PROF. J. ELDER

48. Which Functions are Quadratic?
5n2 + 3n + 2log n
Lies in between
COSC 3101B, PROF. J. ELDER

49. Which Functions are Quadratic?
5n2 + 3n + 2log n
Ignore low-order terms
Ignore multiplicative constants.
Ignore "small" values of n.
Write θ(n2).
COSC 3101B, PROF. J. ELDER

50. Which Functions are Polynomial?
… ?
COSC 3101B, PROF. J. ELDER

51. Which Functions are Polynomial?
n to some constant power.
COSC 3101B, PROF. J. ELDER

52. Which Functions are Polynomial?
5n2 + 8n + 2log n
5n2 log n
5n2.5
COSC 3101B, PROF. J. ELDER

53. Which Functions are Polynomial?
5n2 + 8n + 2log n
5n2 log n
5n2.5
Lie in between
COSC 3101B, PROF. J. ELDER

54. Which Functions are Polynomials?
5n2 + 8n + 2log n
5n2 log n
5n2.5
Ignore low-order terms
Ignore power constant.
Ignore "small" values of n.
Write nθ(1)
COSC 3101B, PROF. J. ELDER

55. Which Functions are Exponential?
… ?
COSC 3101B, PROF. J. ELDER

56. Which Functions are Exponential?
2 raised to a linear function of n.
COSC 3101B, PROF. J. ELDER

57. Which Functions are Exponential?
2n / n100
2n · n100
too small?
too big?
COSC 3101B, PROF. J. ELDER

58. Which Functions are Exponential?
2n / n100
2n · n100
Lie in between
= 23n
> 20.5n
< 22n
COSC 3101B, PROF. J. ELDER

59. Which Functions are Exponential?
2n / n100
2n · n100
= 23n
> 20.5n
< 22n
20.5n > n100
2n = 20.5n · 20.5n > n100 · 20.5n
2n / n100 > 20.5n
COSC 3101B, PROF. J. ELDER

60. Which Functions are Exponential?
2n / n100
2n · n100
Ignore low-order terms
Ignore base.
Ignore "small" values of n.
Ignore polynomial factors.
Write 2θ(n)
COSC 3101B, PROF. J. ELDER

61. Classifying Functions
f(n) = 8·24n / n ·n3
Tell me the most significant thing about your function
Rank constants in significance.
COSC 3101B, PROF. J. ELDER

62. Classifying Functions
f(n) = 8·24n / n ·n3
Tell me the most significant thing about your function
2θ(n)
23n <
< 24n
f(n)
because
COSC 3101B, PROF. J. ELDER

63. Classifying Functions
f(n) = 8·24n / n ·n3
Tell me the most significant thing about your function
2θ(n)
24n/nθ(1)
θ(24n/n100)
8·24n / n nθ(1)
8·24n / n θ(n3)
8·24n / n ·n3
COSC 3101B, PROF. J. ELDER

64. Notations f(n) = θ(g(n)) f(n) ≈ c g(n) f(n) = O(g(n)) f(n) ≤ c g(n)
Theta
f(n) = θ(g(n))
f(n) ≈ c g(n)
BigOh
f(n) = O(g(n))
f(n) ≤ c g(n)
Omega
f(n) = Ω(g(n))
f(n) ≥ c g(n)
Little Oh
f(n) = o(g(n))
f(n) << c g(n)
Little Omega
f(n) = ω(g(n))
f(n) >> c g(n)
COSC 3101B, PROF. J. ELDER

65. Definition of Theta
f(n) = θ(g(n))
COSC 3101B, PROF. J. ELDER

66. f(n) is sandwiched between c1g(n) and c2g(n)
Definition of Theta
f(n) = θ(g(n))
f(n) is sandwiched between c1g(n) and c2g(n)
COSC 3101B, PROF. J. ELDER

67. f(n) = θ(g(n)) Definition of Theta
f(n) is sandwiched between c1g(n) and c2g(n)
for some sufficiently small c1 (= )
for some sufficiently large c2 (= 1000)
COSC 3101B, PROF. J. ELDER

68. For all sufficiently large n
Definition of Theta
f(n) = θ(g(n))
For all sufficiently large n
COSC 3101B, PROF. J. ELDER

69. f(n) = θ(g(n)) Definition of Theta For all sufficiently large n
For some definition of “sufficiently large”
COSC 3101B, PROF. J. ELDER

70. Definition of Theta
3n2 + 7n + 8 = θ(n2)
COSC 3101B, PROF. J. ELDER

71. Definition of Theta 3n2 + 7n + 8 = θ(n2) c1·n2 £ 3n2 + 7n + 8 £ c2·n2? ?
COSC 3101B, PROF. J. ELDER

72. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·n2 £ 3n2 + 7n + 8 £ 4·n2 3
COSC 3101B, PROF. J. ELDER

73. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·12 £ 3·12 + 7·1 + 8 £ 4·12
3·12 £ 3·12 + 7· £ 4·12 1
False
COSC 3101B, PROF. J. ELDER

74. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·72 £ 3·72 + 7·7 + 8 £ 4·72
3·72 £ 3·72 + 7· £ 4·72 7
False
COSC 3101B, PROF. J. ELDER

75. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·82 £ 3·82 + 7·8 + 8 £ 4·82
3·82 £ 3·82 + 7· £ 4·82 8
True
COSC 3101B, PROF. J. ELDER

76. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·92 £ 3·92 + 7·9 + 8 £ 4·92
3·92 £ 3·92 + 7· £ 4·92 9
True
COSC 3101B, PROF. J. ELDER

77. Definition of Theta 3n2 + 7n + 8 = θ(n2) 10 True
3·102 £ 3· · £ 4·102 10
True
COSC 3101B, PROF. J. ELDER

78. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·n2 £ 3n2 + 7n + 8 £ 4·n2 ?
COSC 3101B, PROF. J. ELDER

79. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·n2 £ 3n2 + 7n + 8 £ 4·n2
COSC 3101B, PROF. J. ELDER

80. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·n2 £ 3n2 + 7n + 8 £ 4·n2
True
COSC 3101B, PROF. J. ELDER

81. Definition of Theta 3n2 + 7n + 8 = θ(n2) 3·n2 £ 3n2 + 7n + 8 £ 4·n2
COSC 3101B, PROF. J. ELDER
True

82. Definition of Theta 3n2 + 7n + 8 = θ(n2) True
3·n2 £ 3n2 + 7n £ 4·n2 3 4 8
n ³ 8
COSC 3101B, PROF. J. ELDER

83. End of Lecture 2
Sept 12, 2006

84. Asymptotic Notation f(n) = θ(g(n)) f(n) ≈ c g(n) f(n) = O(g(n))
Theta
f(n) = θ(g(n))
f(n) ≈ c g(n)
BigOh
f(n) = O(g(n))
f(n) ≤ c g(n)
Omega
f(n) = Ω(g(n))
f(n) ≥ c g(n)
Little Oh
f(n) = o(g(n))
f(n) << c g(n)
Little Omega
f(n) = ω(g(n))
f(n) >> c g(n)
COSC 3101B, PROF. J. ELDER

85. Definition of Theta
3n2 + 7n + 8 = θ(n)
COSC 3101B, PROF. J. ELDER

86. Definition of Theta 3n2 + 7n + 8 = θ(n) c1·n £ 3n2 + 7n + 8 £ c2·n ? ?
COSC 3101B, PROF. J. ELDER

87. Definition of Theta 3n2 + 7n + 8 = θ(n) 3 3·n £ 3n2 + 7n + 8 £ 100·n
COSC 3101B, PROF. J. ELDER

88. Definition of Theta 3n2 + 7n + 8 = θ(n) ? 3·n £ 3n2 + 7n + 8 £ 100·n
COSC 3101B, PROF. J. ELDER

89. Definition of Theta 3n2 + 7n + 8 = θ(n) False
3·100 £ 3· · £ 100·100
100
False
COSC 3101B, PROF. J. ELDER

90. Definition of Theta 3n2 + 7n + 8 = θ(n) 3
3·n £ 3n2 + 7n £ 10,000·n 3
10,000
COSC 3101B, PROF. J. ELDER

91. Definition of Theta 3n2 + 7n + 8 = θ(n) False 10,000
3·10,000 £ 3·10, ·10, £ 100·10,000
10,000
False
COSC 3101B, PROF. J. ELDER

92. Definition of Theta 3n2 + 7n + 8 = θ(n) What is the reverse statement?
COSC 3101B, PROF. J. ELDER

93. Understand Quantifiers!!!
Ø[" b, Loves(b, MM)]
" b, Loves(b, MM)
Ø[$ b, ØLoves(b, MM)]
$ b, ØLoves(b, MM)
Sam
Mary
Bob
Beth
John
Marilyn Monroe
Fred
Ann
Sam
Mary
Bob
Beth
John
Marilyn Monroe
Fred
Ann
COSC 3101B, PROF. J. ELDER

94. Definition of Theta 3n2 + 7n + 8 = θ(n) The reverse statement
COSC 3101B, PROF. J. ELDER

95. Definition of Theta
3n2 + 7n + 8 = θ(n) ?
COSC 3101B, PROF. J. ELDER

96. Order of Quantifiers
f(n) = θ(g(n)) ?
COSC 3101B, PROF. J. ELDER

97. Understand Quantifiers!!!
Could be a separate girl for each boy.
One girl
Sam
Mary
Bob
Beth
John
Marilin Monro
Fred
Ann
Sam
Mary
Bob
Beth
John
Marilin Monro
Fred
Ann
COSC 3101B, PROF. J. ELDER

98. No! It cannot be a different c1 and c2 for each n.
Order of Quantifiers
f(n) = θ(g(n))
No! It cannot be a different c1 and c2 for each n.
COSC 3101B, PROF. J. ELDER

99. Other Notations f(n) = θ(g(n)) f(n) ≈ c g(n) f(n) = O(g(n))
Theta
f(n) = θ(g(n))
f(n) ≈ c g(n)
BigOh
f(n) = O(g(n))
f(n) ≤ c g(n)
Omega
f(n) = Ω(g(n))
f(n) ≥ c g(n)
Little Oh
f(n) = o(g(n))
f(n) << c g(n)
Little Omega
f(n) = ω(g(n))
f(n) >> c g(n)
COSC 3101B, PROF. J. ELDER

100. Time Complexity Is a Function
Specifies how the running time depends on the size of the input.
A function mapping “size” of input  “time” T(n) executed .
COSC 3101B, PROF. J. ELDER

101. Definition of Time?
COSC 3101B, PROF. J. ELDER

102. Definition of Time # of seconds (machine dependent).
# lines of code executed.
# of times a specific operation is performed (e.g., addition).
These are all reasonable definitions of time, because they are within a constant factor of each other.
COSC 3101B, PROF. J. ELDER

103. Definition of Input Size
A good definition: #bits
Examples:
COSC 3101B, PROF. J. ELDER

104. Time Complexity Is a Function
Specifies how the running time depends on the size of the input.
A function mapping:
“size” of input  “time” T(n) executed .
Or more precisely:
# of bits n needed to represent the input  # of operations T(n) executed .
COSC 3101B, PROF. J. ELDER

105. Time Complexity of an Algorithm
The time complexity of an algorithm is the largest time required on any input of size n.
O(n2): Prove that for every input of size n, the algorithm takes no more than cn2 time.
Ω(n2): Find one input of size n, for which the algorithm takes at least this much time.
θ (n2): Do both.
COSC 3101B, PROF. J. ELDER

106. What is the height of tallest person in the class?
Bigger than this?
Need to look at only one person
Need to look at every person
Smaller than this?
COSC 3101B, PROF. J. ELDER

107. Time Complexity of a Problem
The time complexity of a problem is the time complexity of the fastest algorithm that solves the problem.
O(n2): Provide an algorithm that solves the problem in no more than this time.
Ω(n2): Prove that no algorithm can solve it faster.
θ (n2): Do both.
COSC 3101B, PROF. J. ELDER

108. End of Lecture 3
Sept 14, 2006