1. CS2013 Lecture 5
John Hurley
Cal State LA

2. 2
Execution Time
Suppose two algorithms perform the same task such as search (linear search vs. binary search) and sorting (selection sort vs. insertion sort). Which one is better?
One approach is to implement these algorithms in Java and run the programs to get execution time. But there are two problems:
First, there are many tasks running concurrently on a computer. The execution time of a particular program is dependent on the system load.
Second, the execution time is usually dependent on specific input. Consider linear search and binary search: if the key happens to be the first element in the list, linear search will find the element quicker than binary search, which is unrepresentative of the general performance of the algorithms.

3. 3
Growth Rate
In addition to the problems mentioned in the last slide, the importance of algorithm performance increases as input size increases.
We may not care that linear search is less efficient than binary search if we only have a few values to search.
The standard approach to measuring algorithm efficiency approximates the effect of increasing the size of the input. In this way, you can see how an algorithm’s execution time increases with input size.

4. 4
Math Ahead!
The rest of this lecture uses a few math principles that you learned in HS but may have forgotten.
Do not worry (too much) if your math background is shaky. We introduce mathematical material at a gentle pace. When I started working on my MS here, I had not taken a math class in 25 years, and I managed to learn this material. You can too.
On the other hand, if you want to study this material in more detail, you will not be disappointed. You just have to wait until you take CS3120.

5. 5
Summations
Summation is the operation of adding a sequence of numbers; the result is their sum or total. Summation is designated with the Greek symbol sigma (∑)
Stop at 100
Find the sum
Iterate through values of i
Start from 1
This summation means "the sum of all integers between 1 and 100 inclusive"

6. Useful Mathematic Summations6
Useful Mathematic Summations

7. Useful Mathematic Summations7
Useful Mathematic Summations
The first summation on the previous slide would usually be expressed this way:
For the following values, the value of the summation is 5050, since 100(101)/2 = 10100/2 = 5050:

8. 8
Logarithms
The logarithm of a number with respect to a base number is the exponent to which the base must be raised to yield the number.
Here is the notation:
logb(y) = x
where b is the base. The parentheses are usually left out in practice.
Examples:
log2 8 = 3
log1010,000 = 4
The word "logarithm" is derived (by an early-modern mathematician) from Greek and means roughly "succession reasoning." Interestingly (to me, anyway) it is completely unrelated to its anagram "algorithm," which is derived from an Arabic (prefixed with al-) version of the name of the medieval Persian mathematician Kwarizmi

9. 9
Logarithms
In other fields, log without an stated base is understood to refer to log10 or loge.
In CS, log without further qualification is understood to refer to log2, pronounced "log base 2" or "binary logarithm."
The base is not important in comparing algorithms, but, as you will see, the base is almost always 2 when we are calculating the complexity of programming algorithms.

10. 10
Big O Notation
Linear search compares the key with the elements in the array sequentially until the key is found or the array is exhausted. If the key is not in the array, it requires n comparisons for an array of size n. If the key is in the array, it requires, in the average case, n/2 comparisons.
This algorithm’s execution time is proportional to the size of the array. If you double the size of the array, you will expect the number of comparisons to double in the average case as well as in the worst case. The algorithm grows at a linear rate. More precisely, as input size increases asymptotically (towards infinity), the expense of the calculation won’t grow faster than linearly.
Computer scientists use the Big O notation as an abbreviation for “order,” a function that sets an upper bound on the growth of the function we are measuring Using this notation, the complexity of the linear search algorithm is O(n), pronounced as “order of n.”

11. Big-O Notation
Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n0 such that
f(n)  cg(n) for n  n0
Example: 2n + 10 is O(n)
2n + 10  cn
(c  2) n  10
n  10/(c  2)
Pick c = 3 and n0 = 10
2n+10 is  3n for all n  10
Both axes in this chart are logarithmic
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 11

12. Big-Oh Example Example: the function n2 is not O(n) n2  cn n  c
The above inequality cannot be satisfied since c must be a constant
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 12

13. More Big-O Examples 7n - 2 3 n3 + 20 n2 + 5 3 log n + 5 7n-2 is O(n)
need c > 0 and n0  1 such that 7 n - 2  c n for n  n0
this is true for c = 7 and n0 = 1
3 n n2 + 5
3 n n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3 n n2 + 5  c n3 for n  n0
this is true for c = 4 and n0 = 21
3 log n + 5
3 log n + 5 is O(log n)
need c > 0 and n0  1 such that 3 log n + 5  c log n for n  n0
this is true for c = 8 and n0 = 2 13

14. More Big-O Examples 7n - 2 7n-2 is O(n)
need c > 0 and n0  1 such that 7 n - 2  c n for n  n0
this is true for c = 7 and n0 = 1
7n - 2  cn
7n  cn + 2
Make c = 7
7n  7n + 2
n0 = 1 14

15. More Big-O Examples 3 n3 + 20 n2 + 5 Set c = 4
3 n n2 + 5 is O(n3)
3 n n2 + 5  n3
20 n2 + 5  (c - 3)n3
Set c = 4
20 n2 + 5  n3
n0 must be more than 20.
n0 = 21 works 15

16. More Big-O Examples 3 log n + 5 3 log n + 5 is O(log n)
need c > 0 and n0  1 such that 3 log n + 5  c log n for n  n0
this is true for c = 8 and n0 = 2:
5 <= (c-3) log n
Log 2 = 1, so for n >= 2
5 <= c - 3
Is true where c >= 8 16

17. Big-Oh and Growth Rate f(n) is O(g(n)) g(n) is O(f(n)) g(n) grows more
The big-Oh notation gives an upper bound on the growth rate of a function
The statement “f(n) is O(g(n))” means that the growth rate of f(n) is no more than the growth rate of g(n)
We can use the big-Oh notation to rank functions according to their growth rate
f(n) is O(g(n))
g(n) is O(f(n))
g(n) grows more
Yes No
f(n) grows more
Same growth
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 17

18. 18
Constant Time
The Big O notation estimates the execution time of an algorithm in relation to the input size. If the time is not related to the input size, the algorithm is said to take constant time with the notation O(1). For example, a method that retrieves an element at a given index in an array takes constant time, because it does not grow as the size of the array increases.

19. Ignore Multiplicative Constants19
Ignore Multiplicative Constants
The linear search algorithm requires n comparisons in the worst-case and n/2 comparisons in the average-case.
Using the Big O notation, both cases require O(n) time. The multiplicative constant (1/2) can be omitted. Multiplicative constants have no impact on the order of magnitude of the growth rate. The growth rate for n/2 or 100n is the same as n, i.e., O(n) = O(n/2) = O(100n).

20. Ignore Non-Dominating Terms20
Ignore Non-Dominating Terms
Consider the algorithm for finding the maximum number in an array of n elements. If n is 2, it takes one comparison to find the maximum number. If n is 3, it takes two comparisons to find the maximum number. In general, it takes n-1 comparisons to find maximum number in a list of n elements.
Algorithm analysis is concerned with large input size. If the input size is small, there is no need to estimate an algorithm’s efficiency. As n grows larger, the n part in the expression n-1 dominates the complexity. The Big O notation allows you to ignore the non-dominating part (e.g., -1 in the expression n-1) and highlight the important part (e.g., n in the expression n-1). So, the complexity of this algorithm is O(n).

21. Big-Oh Rules
If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e.,
Drop lower-order terms
Drop constant factors
Use the smallest possible class of functions
Say “2n is O(n)” instead of “2n is O(n2)”
Use the simplest expression of the class
Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 21

22. 22
Best, Worst, and Average
An input that results in the shortest execution time for a given input size is called the best-case input and an input that results in the longest execution time is called the worst-case input.
Best-case and worst-case are not representative, but worst-case analysis is very useful. You can show that the algorithm will never be slower than the worst-case.
An average-case analysis attempts to determine the average amount of time among all possible input of the same size.
Average-case analysis is ideal, but difficult to perform, because it is hard to determine the relative probabilities and distributions of various input instances for many problems. Worst-case analysis is easier to obtain and is thus common. So, the analysis is generally conducted for the worst-case.

23. Asymptotic Algorithm Analysis
The asymptotic analysis of an algorithm determines the running time in big-Oh notation
To perform the asymptotic analysis
We find the worst-case number of primitive operations executed as a function of the input size
We express this function with big-Oh notation
Example:
We say that algorithm arrayMax “runs in O(n) time”
Since constant factors and lower-order terms are eventually dropped anyhow, we can disregard them when counting primitive operations
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 23 23

24. Primitive Operations Basic computations performed by an algorithm
Identifiable in pseudocode
Largely independent from the programming language
Exact definition not important (we will see why later)
Assumed to take a constant amount of time in the RAM model
Examples:
Evaluating an expression
Assigning a value to a variable
Indexing into an array
Calling a method
Returning from a method
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 24

25. Ignore multiplicative constants (e.g., “c”).
Repetition: Simple Loops
for (i = 1; i <= n; i++) {
k = k + 5; }
executed
n times
constant time
Time Complexity
T(n) = (a constant c) * n = cn = O(n)
Ignore multiplicative constants (e.g., “c”). 25

26. Ignore multiplicative constants (e.g., “c”).
Repetition: Nested Loops
for (i = 1; i <= n; i++) {
for (j = 1; j <= n; j++) {
k = k + i + j; }
executed
n times
inner loop
executed
n times
Time Complexity
T(n) = (a constant c) * n * n = cn2 = O(n2)
Ignore multiplicative constants (e.g., “c”). 26

27. Repetition: Nested Loops
for (i = 1; i <= n; i++) {
for (j = 1; j <= i; j++) {
k = k + i + j; }
executed
n times
inner loop
executed
i times
Time Complexity
T(n) = c + 2c + 3c + 4c + … + nc = cn(n+1)/2 = (c/2)n2 + (c/2)n = O(n2)
Ignore non-dominating terms
Ignore multiplicative constants 27

28. Ignore multiplicative constants (e.g., 20*c)
Repetition: Nested Loops
for (i = 1; i <= n; i++) {
for (j = 1; j <= 20; j++) {
k = k + i + j; }
executed
n times
inner loop
executed
20 times
Time Complexity
T(n) = 20 * c * n = O(n)
Ignore multiplicative constants (e.g., 20*c) 28

29. Sequence for (j = 1; j <= 10; j++) { k = k + 4; }
executed
10 times
for (i = 1; i <= n; i++) {
for (j = 1; j <= 20; j++) {
k = k + i + j; }
executed
n times
inner loop
executed
20 times
Time Complexity
T(n) = c * * c * n = O(n) 29

30. Selection if (list.contains(e)) { System.out.println(e); } else
for (Object t: list) {
System.out.println(t);
Assuming linear search,
O(n)
Let n be
list.size().
Executed
n times.
Time Complexity
T(n) = test time + worst-case (if, else)
= O(n) + O(n)
= O(n) 30

31. Analysis of Algorithms
Slide by Matt Stallmann included with permission.
Why Growth Rate Matters
if runtime is...
time for n + 1
time for 2 n
time for 4 n
c lg n
c lg (n + 1)
c (lg n + 1)
c(lg n + 2)
c n
c (n + 1)
2c n
4c n
c n lg n
~ c n lg n
+ c n
2c n lg n + 2cn
4c n lg n + 4cn
c n2
~ c n2 + 2c n
4c n2
16c n2
c n3
~ c n3 + 3c n2
8c n3
64c n3
c 2n
c 2 n+1
c 2 2n
c 2 4n
runtime
quadruples
when problem
size doubles
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 31

32. Comparing Common Growth Functions32
Comparing Common Growth Functions
Constant time
Logarithmic time
Linear time
Log-linear time
Quadratic time
Cubic time
Exponential time

33. Comparing Common Growth Functions33
Comparing Common Growth Functions

34. Functions Graphed Using “Normal” Scale
Slide by Matt Stallmann included with permission.
Functions Graphed Using “Normal” Scale
g(n) = 2n
g(n) = 1
g(n) = n lg n
g(n) = lg n
g(n) = n2
g(n) = n
g(n) = n3
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 34

35. Time Complexity for ArrayList and LinkedList35
Time Complexity for ArrayList and LinkedList

36. 36
Recurrence Relations
A recurrence relation is a rule by which a sequence is generated
Eg, the sequence 5, 8, 11, 14, 17, 20…
Is described by the recurrence relation
a0 = 5
an = an-1 + 3
Divide-and-conquer algorithms are often described in terms of recurrence relations

37. Analyzing Binary Search37
Analyzing Binary Search
Binary Search searches an array or list that is *sorted*
In each step, the algorithm compares the search key value with the key value of the middle element of the array. If the keys match, then a matching element has been found and its index, or position, is returned.
Otherwise, if the search key is less than the middle element's key, then the algorithm repeats its action on the sub-array to the left of the middle element or, if the search key is greater, on the sub-array to the right.
If the remaining array at any step to be searched is empty, then the key cannot be found in the array and a special "not found" indication is returned.

38. Logarithmic Time: Analyzing Binary Search38
Logarithmic Time: Analyzing Binary Search
Each iteration in binary search contains a fixed number of operations, denoted by c.
Let T(n) denote the time complexity for a binary search on a list of n elements. Since we are studying the rate of growth of execution time, we can define T(1) to equal 1.
Assume n is a power of 2; this makes the math simpler and, if it is not true, the difference is irrelevant to the order of complexity.
Let k=log n. In other words, n = 2k
Since binary search eliminates half of the input after each comparison,
c is the cost of one iteration
CS-style recurrence relation

39. 39
Logarithmic Time
Ignoring constants and smaller terms, the complexity of the binary search algorithm is O(log n). An algorithm with the O(log n) time complexity is called a logarithmic algorithm.
The base of the log is 2, but the base does not affect a logarithmic growth rate, so it can be omitted.
The time to execute a logarithmic algorithm grows slowly as the problem size increases. If you square the input size (with base = 2), the time taken doubles.

40. Relatives of Big-Oh big-Omega
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1 such that
f(n)  c g(n) for n  n0
big-Theta
f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and an integer constant n0  1 such that
c’g(n)  f(n)  c’’g(n) for n  n0
© 2014 Goodrich, Tamassia, Goldwasser
Analysis of Algorithms 40