CS2013 Lecture 5 John Hurley Cal State LA

2 Execution Time Suppose two algorithms perform the same task such as search (linear search vs. binary search) and sorting (selection sort vs. insertion sort). Which one is better? One approach is to implement these algorithms in Java and run the programs to get execution time. But there are two problems: First, there are many tasks running concurrently on a computer. The execution time of a particular program is dependent on the system load. Second, the execution time is usually dependent on specific input. Consider linear search and binary search: if the key happens to be the first element in the list, linear search will find the element quicker than binary search, which is unrepresentative of the general performance of the algorithms.

3 Growth Rate In addition to the problems mentioned in the last slide, the importance of algorithm performance increases as input size increases. We may not care that linear search is less efficient than binary search if we only have a few values to search. The standard approach to measuring algorithm efficiency approximates the effect of increasing the size of the input. In this way, you can see how an algorithm’s execution time increases with input size.

4 Math Ahead! The rest of this lecture uses a few math principles that you learned in HS but may have forgotten. Do not worry (too much) if your math background is shaky. We introduce mathematical material at a gentle pace. When I started working on my MS here, I had not taken a math class in 25 years, and I managed to learn this material. You can too. On the other hand, if you want to study this material in more detail, you will not be disappointed. You just have to wait until you take CS3120.

5 Summations Summation is the operation of adding a sequence of numbers; the result is their sum or total. Summation is designated with the Greek symbol sigma (∑) Stop at 100 Find the sum Iterate through values of i Start from 1 This summation means "the sum of all integers between 1 and 100 inclusive"

Useful Mathematic Summations 6 Useful Mathematic Summations

Useful Mathematic Summations 7 Useful Mathematic Summations The first summation on the previous slide would usually be expressed this way: For the following values, the value of the summation is 5050, since 100(101)/2 = 10100/2 = 5050:

8 Logarithms The logarithm of a number with respect to a base number is the exponent to which the base must be raised to yield the number. Here is the notation: logb(y) = x where b is the base. The parentheses are usually left out in practice. Examples: log2 8 = 3 log1010,000 = 4 The word "logarithm" is derived (by an early-modern mathematician) from Greek and means roughly "succession reasoning." Interestingly (to me, anyway) it is completely unrelated to its anagram "algorithm," which is derived from an Arabic (prefixed with al-) version of the name of the medieval Persian mathematician Kwarizmi

9 Logarithms In other fields, log without an stated base is understood to refer to log10 or loge. In CS, log without further qualification is understood to refer to log2, pronounced "log base 2" or "binary logarithm." The base is not important in comparing algorithms, but, as you will see, the base is almost always 2 when we are calculating the complexity of programming algorithms.

10 Big O Notation Linear search compares the key with the elements in the array sequentially until the key is found or the array is exhausted. If the key is not in the array, it requires n comparisons for an array of size n. If the key is in the array, it requires, in the average case, n/2 comparisons. This algorithm’s execution time is proportional to the size of the array. If you double the size of the array, you will expect the number of comparisons to double in the average case as well as in the worst case. The algorithm grows at a linear rate. More precisely, as input size increases asymptotically (towards infinity), the expense of the calculation won’t grow faster than linearly. Computer scientists use the Big O notation as an abbreviation for “order,” a function that sets an upper bound on the growth of the function we are measuring Using this notation, the complexity of the linear search algorithm is O(n), pronounced as “order of n.”

Big-O Notation Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n0 such that f(n)  cg(n) for n  n0 Example: 2n + 10 is O(n) 2n + 10  cn (c  2) n  10 n  10/(c  2) Pick c = 3 and n0 = 10 2n+10 is  3n for all n  10 Both axes in this chart are logarithmic © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 11

Big-Oh Example Example: the function n2 is not O(n) n2  cn n  c The above inequality cannot be satisfied since c must be a constant © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 12

More Big-O Examples 7n - 2 3 n3 + 20 n2 + 5 3 log n + 5 7n-2 is O(n) need c > 0 and n0  1 such that 7 n - 2  c n for n  n0 this is true for c = 7 and n0 = 1 3 n3 + 20 n2 + 5 3 n3 + 20 n2 + 5 is O(n3) need c > 0 and n0  1 such that 3 n3 + 20 n2 + 5  c n3 for n  n0 this is true for c = 4 and n0 = 21 3 log n + 5 3 log n + 5 is O(log n) need c > 0 and n0  1 such that 3 log n + 5  c log n for n  n0 this is true for c = 8 and n0 = 2 13

More Big-O Examples 7n - 2 7n-2 is O(n) need c > 0 and n0  1 such that 7 n - 2  c n for n  n0 this is true for c = 7 and n0 = 1 7n - 2  cn 7n  cn + 2 Make c = 7 7n  7n + 2 n0 = 1 14

More Big-O Examples 3 n3 + 20 n2 + 5 Set c = 4 3 n3 + 20 n2 + 5 is O(n3) 3 n3 + 20 n2 + 5  n3 20 n2 + 5  (c - 3)n3 Set c = 4 20 n2 + 5  n3 n0 must be more than 20. n0 = 21 works 15

More Big-O Examples 3 log n + 5 3 log n + 5 is O(log n) need c > 0 and n0  1 such that 3 log n + 5  c log n for n  n0 this is true for c = 8 and n0 = 2: 5 <= (c-3) log n Log 2 = 1, so for n >= 2 5 <= c - 3 Is true where c >= 8 16

Big-Oh and Growth Rate f(n) is O(g(n)) g(n) is O(f(n)) g(n) grows more The big-Oh notation gives an upper bound on the growth rate of a function The statement “f(n) is O(g(n))” means that the growth rate of f(n) is no more than the growth rate of g(n) We can use the big-Oh notation to rank functions according to their growth rate f(n) is O(g(n)) g(n) is O(f(n)) g(n) grows more Yes No f(n) grows more Same growth © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 17

18 Constant Time The Big O notation estimates the execution time of an algorithm in relation to the input size. If the time is not related to the input size, the algorithm is said to take constant time with the notation O(1). For example, a method that retrieves an element at a given index in an array takes constant time, because it does not grow as the size of the array increases.

Ignore Multiplicative Constants 19 Ignore Multiplicative Constants The linear search algorithm requires n comparisons in the worst-case and n/2 comparisons in the average-case. Using the Big O notation, both cases require O(n) time. The multiplicative constant (1/2) can be omitted. Multiplicative constants have no impact on the order of magnitude of the growth rate. The growth rate for n/2 or 100n is the same as n, i.e., O(n) = O(n/2) = O(100n).

Ignore Non-Dominating Terms 20 Ignore Non-Dominating Terms Consider the algorithm for finding the maximum number in an array of n elements. If n is 2, it takes one comparison to find the maximum number. If n is 3, it takes two comparisons to find the maximum number. In general, it takes n-1 comparisons to find maximum number in a list of n elements. Algorithm analysis is concerned with large input size. If the input size is small, there is no need to estimate an algorithm’s efficiency. As n grows larger, the n part in the expression n-1 dominates the complexity. The Big O notation allows you to ignore the non-dominating part (e.g., -1 in the expression n-1) and highlight the important part (e.g., n in the expression n-1). So, the complexity of this algorithm is O(n).

Big-Oh Rules If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e., Drop lower-order terms Drop constant factors Use the smallest possible class of functions Say “2n is O(n)” instead of “2n is O(n2)” Use the simplest expression of the class Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)” © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 21

22 Best, Worst, and Average An input that results in the shortest execution time for a given input size is called the best-case input and an input that results in the longest execution time is called the worst-case input. Best-case and worst-case are not representative, but worst-case analysis is very useful. You can show that the algorithm will never be slower than the worst-case. An average-case analysis attempts to determine the average amount of time among all possible input of the same size. Average-case analysis is ideal, but difficult to perform, because it is hard to determine the relative probabilities and distributions of various input instances for many problems. Worst-case analysis is easier to obtain and is thus common. So, the analysis is generally conducted for the worst-case.

Asymptotic Algorithm Analysis The asymptotic analysis of an algorithm determines the running time in big-Oh notation To perform the asymptotic analysis We find the worst-case number of primitive operations executed as a function of the input size We express this function with big-Oh notation Example: We say that algorithm arrayMax “runs in O(n) time” Since constant factors and lower-order terms are eventually dropped anyhow, we can disregard them when counting primitive operations © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 23 23

Primitive Operations Basic computations performed by an algorithm Identifiable in pseudocode Largely independent from the programming language Exact definition not important (we will see why later) Assumed to take a constant amount of time in the RAM model Examples: Evaluating an expression Assigning a value to a variable Indexing into an array Calling a method Returning from a method © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 24

Ignore multiplicative constants (e.g., “c”). Repetition: Simple Loops for (i = 1; i <= n; i++) { k = k + 5; } executed n times constant time Time Complexity T(n) = (a constant c) * n = cn = O(n) Ignore multiplicative constants (e.g., “c”). 25

Ignore multiplicative constants (e.g., “c”). Repetition: Nested Loops for (i = 1; i <= n; i++) { for (j = 1; j <= n; j++) { k = k + i + j; } executed n times inner loop executed n times Time Complexity T(n) = (a constant c) * n * n = cn2 = O(n2) Ignore multiplicative constants (e.g., “c”). 26

Repetition: Nested Loops for (i = 1; i <= n; i++) { for (j = 1; j <= i; j++) { k = k + i + j; } executed n times inner loop executed i times Time Complexity T(n) = c + 2c + 3c + 4c + … + nc = cn(n+1)/2 = (c/2)n2 + (c/2)n = O(n2) Ignore non-dominating terms Ignore multiplicative constants 27

Ignore multiplicative constants (e.g., 20*c) Repetition: Nested Loops for (i = 1; i <= n; i++) { for (j = 1; j <= 20; j++) { k = k + i + j; } executed n times inner loop executed 20 times Time Complexity T(n) = 20 * c * n = O(n) Ignore multiplicative constants (e.g., 20*c) 28

Sequence for (j = 1; j <= 10; j++) { k = k + 4; } executed 10 times for (i = 1; i <= n; i++) { for (j = 1; j <= 20; j++) { k = k + i + j; } executed n times inner loop executed 20 times Time Complexity T(n) = c *10 + 20 * c * n = O(n) 29

Selection if (list.contains(e)) { System.out.println(e); } else for (Object t: list) { System.out.println(t); Assuming linear search, O(n) Let n be list.size(). Executed n times. Time Complexity T(n) = test time + worst-case (if, else) = O(n) + O(n) = O(n) 30

Analysis of Algorithms Slide by Matt Stallmann included with permission. Why Growth Rate Matters if runtime is... time for n + 1 time for 2 n time for 4 n c lg n c lg (n + 1) c (lg n + 1) c(lg n + 2) c n c (n + 1) 2c n 4c n c n lg n ~ c n lg n + c n 2c n lg n + 2cn 4c n lg n + 4cn c n2 ~ c n2 + 2c n 4c n2 16c n2 c n3 ~ c n3 + 3c n2 8c n3 64c n3 c 2n c 2 n+1 c 2 2n c 2 4n runtime quadruples when problem size doubles © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 31

Comparing Common Growth Functions 32 Comparing Common Growth Functions Constant time Logarithmic time Linear time Log-linear time Quadratic time Cubic time Exponential time

Comparing Common Growth Functions 33 Comparing Common Growth Functions

Functions Graphed Using “Normal” Scale Slide by Matt Stallmann included with permission. Functions Graphed Using “Normal” Scale g(n) = 2n g(n) = 1 g(n) = n lg n g(n) = lg n g(n) = n2 g(n) = n g(n) = n3 © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 34

Time Complexity for ArrayList and LinkedList 35 Time Complexity for ArrayList and LinkedList

36 Recurrence Relations A recurrence relation is a rule by which a sequence is generated Eg, the sequence 5, 8, 11, 14, 17, 20… Is described by the recurrence relation a0 = 5 an = an-1 + 3 Divide-and-conquer algorithms are often described in terms of recurrence relations

Analyzing Binary Search 37 Analyzing Binary Search Binary Search searches an array or list that is *sorted* In each step, the algorithm compares the search key value with the key value of the middle element of the array. If the keys match, then a matching element has been found and its index, or position, is returned. Otherwise, if the search key is less than the middle element's key, then the algorithm repeats its action on the sub-array to the left of the middle element or, if the search key is greater, on the sub-array to the right. If the remaining array at any step to be searched is empty, then the key cannot be found in the array and a special "not found" indication is returned.

Logarithmic Time: Analyzing Binary Search 38 Logarithmic Time: Analyzing Binary Search Each iteration in binary search contains a fixed number of operations, denoted by c. Let T(n) denote the time complexity for a binary search on a list of n elements. Since we are studying the rate of growth of execution time, we can define T(1) to equal 1. Assume n is a power of 2; this makes the math simpler and, if it is not true, the difference is irrelevant to the order of complexity. Let k=log n. In other words, n = 2k Since binary search eliminates half of the input after each comparison, c is the cost of one iteration CS-style recurrence relation

39 Logarithmic Time Ignoring constants and smaller terms, the complexity of the binary search algorithm is O(log n). An algorithm with the O(log n) time complexity is called a logarithmic algorithm. The base of the log is 2, but the base does not affect a logarithmic growth rate, so it can be omitted. The time to execute a logarithmic algorithm grows slowly as the problem size increases. If you square the input size (with base = 2), the time taken doubles.

Relatives of Big-Oh big-Omega f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1 such that f(n)  c g(n) for n  n0 big-Theta f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and an integer constant n0  1 such that c’g(n)  f(n)  c’’g(n) for n  n0 © 2014 Goodrich, Tamassia, Goldwasser Analysis of Algorithms 40