Algorithm Analysis Data Structures and Algorithms (60-254)

Quantification of Performance We want to understand the behaviour of our algorithms (both runtime and space utilization) on different inputs. This will enable us to compare different algorithms that solve the same problem. To do this, we characterize the performance as a function of the number of inputs (why? reasonable? pitfalls?) The functions may be complex. We want to understand their growth rate in simpler terms. 2

Growth of Functions Let f(n) be a function of a positive integer n. The dominant term of f(n) determines the behavior of f(n) as n  . For example, let f(n) = 2n 3 + 3n 2 + 4n + 1 The dominant term of f(n) is 2n 3. This means that as n becomes large (n   ): 2n 3 dominates the behavior of f(n) The other terms’ contributions become much less significant. 3

More Examples Example 2: The term dominates the behavior of f(n) as n   Example 3: The term 2 n dominates the behavior of f(n) as n   4

Rate of Growth The rate of growth means how function behaves as n  . It is determined by its dominant term. The big-Oh notation is a short-hand way of expressing this. The relationship f(n) is O(n 2 ) is interpreted as: f(n) grows no faster than n 2 as n becomes large (n   ). The dominant term of f(n) does not grow faster than n 2. 5

More Examples Also, O(n 2 ) is true for: We could make up as many such functions as we wish. 6

Definition of Big Oh Problem: Find the most accurate description for any function in terms of the big-Oh notation. A formal definition of: f(n) is O(g(n)) is that the inequality: f(n)  c * g(n) holds for all n  n 0, where n 0 and c are positive constants f(n) and g(n) are functions mapping nonnegative integers to real numbers Informally“f(n) is order g(n)” 7

Graphical Interpretation of Big Oh 8

f(n) = 7n 2 +.5n + 6 and g(n) = n 2 f(n) is O(n 2 ), provided that c = 10 and n 0 = 2 n 0 = 2 Example 9 7n 2 +.5n n 2

In General In general, if f(n) = a 0 + a 1 n + … + a d-1 n d-1 + a d n d Then, f(n) is O(n d ) We will see other functions too: For example: O(log n), O(n log n),, etc. Having defined the big-Oh notation, now … 10

Quantifying Performance 11 Input sizeTime I1I1 T1T1 I2I2 T2T2 …… We want to quantify the behavior of an algorithm. Use this to compare efficiency of two algorithms on the same problem. Observations: 1.A program (algorithm) consumes resources: time and space 2.Amount of resources directly related to size of input. Say, we have the following table: How can we derive the function T(I) ?

Quantifying Performance How can we derive the function T(I) ? Problems: Too many parameters for using interpolation It depends on machine in which program is run Compiler used Programming language Programmer who writes the code Solution: Imagine our algorithm runs on an “algorithm machine” that accepts pseudo-code. Assumptions: READs and WRITEs take constant time Arithmetic operations take constant time Logical operations “ “ “ 12

Worst Case Even though we made various assumptions, it is still complicated. Instead, we quantify our algorithm on the worst-case input. This is called “worst-case analysis” Also, the “average-case analysis” exists: Requires probability distribution of set of inputs which is usually unknown. Not studied in this course. 13

Input Size not always easy to determine, and problem dependent Some examples: Graph-theoretic problem: Number of vertices, V, and number of edges, E. Matrix multiplication: Number of rows and columns of input matrices. Sorting: The number of elements, n. 14

Reporting Performance Typically we don’t try to find exactly what T(I) is. Instead, we can say: T(I) is O(g(I)) For example, time complexity of mergesort of n elements: T(n) is O(n log n) Behavior of mergesort is better than a constant times n log n, where n  n 0. 15

Analysis of Examples 1.Given a list of n elements, find the minimum (or maximum). Then, T(n) is O(n) We look at all elements to determine minimum (maximum). 2.Given n points in the plane, find the closest pair of points. In this case, T(n) is O(n 2 ) Why? A brute-force algorithm that looks at all n 2 pairs of points. 16

Analysis of Examples 3.Given n points in a plane, determine if any three points are contained in a straight line. In this case, T(n) is O(n 3 ) Why? A brute-force algorithm that searches all n 3 triplets. 17

WAIT A MINUTE! What is T(n) for finding the GCD of m and n? The naïve brute force algorithm was O(n) but the Euclidean algorithm was O(log n)? Hmmm… And how about this? Find the gcd(m=1989,n=1590). Algorithm: Step 1. Output 3. So T(n) is O(1). We must distinguish between the complexity of an algorithm and the complexity of a class of problems. 18

Maximum Contiguous Subsequence (MCS) Problem Given a sequence of n integers: a 1, a 2, a 3, …, a n-1, a n a contiguous subsequence is: a i, a i+1, …, a j-1, a j, where 1  i  j  n. The problem: Determine a contiguous subsequence such that: a i + a i+1 + … + a j-1 + a j  0 is maximal. Some examples: -1, -2, -3, -4, -5, -6 MCS is empty, has value 0 by definition. 19

More Examples For the sequence: -1, 2, 3, -3, 2, an MCS is 2, 3 whose value is = 5. Note: There may be more than one MCS. For example: -1, 1, -1, 1, -1, 1 has six MCS whose value is 1 20

An O(n 2 ) Algorithm for MCS Search problems have an associated search space. To figure out: How large the search space is. For the MCS problem: How many sequences need be examined? For example, -1, 2, 3, -3, 2 Then, the subsequences that begin with –1 are: -1, 2 -1, 2, 3 -1, 2, 3, -3 -1, 2, 3, -3, 2 21

An O(n 2 ) Algorithm for MCS The ones beginning with 2 are: 2 2, 3 2, 3, -3 2, 3, -3, 2 Those beginning with 3 are: 3 3, -3 3, -3, 2 The ones beginning with –3: -3 -3, 2 and beginning with 2, just one: 2 22

An O(n 2 ) Algorithm for MCS Then, including the empty sequence, a total of 16 examined. In general, given a 1, a 2, a 3, …, a n-1, a n We have n sequences beginning with a 1 : a 1 a 1, a 2 a 1, a 2, a 3 …. a 1, a 2, a 3, …, a n-1, a n n-1 beginning with a 2 : a 2 a 2, a 3 …. a 2, a 3, …, a n-1, a n 23

An O(n 2 ) Algorithm for MCS and so on. Then, two subsequences beginning with a n-1 : a n-1 a n-1, a n and, finally, one beginning with a n a n Total of possible subsequences: … n-1 + n + 1 = n(n+1)/2 + 1 Analysis: The dominant term is n 2 /2, hence search space is O(n 2 ). A “brute-force” algorithm follows… 24

An O(n 2 ) Algorithm for MCS Algorithm MCSBruteForce Input: A sequence a 1, a 2, a 3, …, a n-1, a n. Output: value, start and end of MCS. Set maxSum  0 for i = 1 to n do Set sum  0 for j = i to n do sum  sum + a j if (sum > maxSum). maxSum  sum start  i end  j Print start, end, maxSum and STOP. 25

Improved MCS Algorithm Think of avoiding looking at all the subsequences. Introduce the following notion. Given: a i, a i+1, …, a k, a k+1, …, a j (1) the subsequence: a i, a i+1, …, a k is a prefix of (1), where i  k  j. The prefix sum is: a i + a i+1 + … + a k Observation: In an MCS no prefix sum can be negative. 26

Improved MCS Algorithm In the previous example, -1, 2, 3, -3, 2, we exclude: -1, 2 -1, 2, 3 -1, 2, 3, -3 -1, 2, 3, -3, 2 and -3 -3, 2 as being possible candidates. 27

Improved MCS Algorithm In general: If ever sum < 0, skip over index positions from i+1, …, j Also, if sum  0 always for a starting position i, none of positions i+1, …, n is a candidate start position, since all prefix sums are non-negative. The improved MCS algorithm inspects a i just once. The algorithm follows…. 28

Improved MCS Algorithm Algorithm MCSImproved Set i  1; Set start  end   1 Set maxSum  sum  0 for j = 1 to n do sum  sum + a j if (sum > maxSum) maxSum  sum start  i end  j if (sum < 0) i  j + 1 sum  0 Print start, end, maxSum and STOP. 29

Analysis of the Algorithms Algorithm MCSBruteForce: The outer loop is executed n times For each i, the inner loop is executed n – i + 1 times Thus, the total number of times the inner loop is executed: Algorithm MCSImproved: It has a single for loop, which visits all n elements. Hence, 30