ALGORITHM ANALYSIS Analysis of Resource consumption by processor –Time –Memory –Number of processors –Power Proof that the algorithm works correctly Debasis Mitra 1

ALGORITHM ANALYSIS Time is measured with Step Counts in an Algorithm –Why? Not a constant number, but as a function of Input Size –Why? We are interested primarily in Worst Case Scenario: –Why? Debasis Mitra 2

WHY STEP-COUNTED? Code run time varies depending on –Processor speed –Language –I/O –…–… What is the number of steps by the following fragment? –For i=1:100 { for j= 1: 2 {print i}}; Debasis Mitra 3

Why time-complexity as function of input-size: n? What is the number of steps by the following fragment? –For i=1:n { for j= 1: m {print i}}; Algorithms are to work on different problem sizes What is the “input size” –Number of elements in input, e.g. array size in a search problem Debasis Mitra 4

Why Worst-case scenario? What is the time-complexity (#steps) search alg below? Input array A[1..n] = [3, 9, 5, 7], search key k =3; For i=1:n { if k = = A[i] {print i; return}}; Best case time complexity is “normally” meaningless Sometimes Average-case analysis is done, assuming some distribution of input types Debasis Mitra 5

Upper bound (Big-Oh): t(N) = O(f(N)) if there exists a positive constant c (real) and a corresponding integer n 0 such that t(N)  c.f(N) when N  n 0, for some integer n 0. Related definitions: Lower bound: t(N) =  (g(N)) if there exists a positive constant c (real) and a corresponding integer n 0 such that t(N)  c.g(N) when N  n 0. Tight bound: T(N) =  (h(N)) if and only if t(N) = O(h(N)) and t(N) =  (h(N)). Reminding of Complexity Functions Debasis Mitra 6

Lim n->inf [f(n) / g(n)] = constant, implies both f(n) and g(n) have the same order of terms in n, or, f(n) =  (g(n)) Example: f(n) = 3n 2 + 5n, g(n) = 7n 2, the limit is 3/7 Lim n->inf [f(n) / g(n)] = 0, implies g(n) has higher order term in n than that in f(n), or, f(n) = O(g(n)) Example: f(n) = 3n 2 + 5n, g(n) = 2n 3, the limit is 0 Lim n->inf [f(n) / g(n)] = inf, implies f(n) has higher order term in n than that in g(n), or, f(n) =  (g(n)) Example: f(n) = 3n 2 + 5n, g(n) = 22n, the limit is infinity A short-cut way of comparing two functions f(n) and g(n): Debasis Mitra 7

Points to Note Order complexities are typically monotonically increasing functions with respect to problem-input size n. We typically consider only dominant term ignoring the other ones: 2(n 2 ) + 3n we ignore the 3n, and the constant multiplicand 2 (absorbed in c). So, 2(n 2 ) + 3n = O(n 2 ). It is actually  ( n 2 ). O is often loosely (and confusingly) used for  in the literature, even in this class! n 2 + 3n = O(n 2 ), also = O(n 3 ), and = O(n 4 ), … However, we will prefer the closest one O(n 2 ) as the acceptable order notation for the function [actually we are referring to  ( n 2 )]. Increasing order of functions: Constant or O(1), O(log N), O(log 2 N), …, O(N), O(N logN), O(N 2 ), O(N 2 log N), O(N 3 ), …, O(2 N ), O(3 N ), … Debasis Mitra 8

GENERAL RULES: Loops: number of iterations/ recursions with respect to the input problem instance size N Nested loops: multiplied for I = 1 through N do for j = 1 through N do -whatever- Analysis: O(N*N) Consecutive statements: add steps, which leads to the max for I = 1 through N do -whatever- for I = 1 through N do for j = 1 through N do -moreover- Analysis: O(N) + O(N 2 ) --> O(N 2 ) Conditional statements: max of the alternative paths (worst-case) If (condition) S1 Else S2 Debasis Mitra 9

Why Study Complexity? Code Fibonacci Series recursive & iterative algorithms; and time against input sizes Debasis Mitra 10

Background Needed Data Structures: Big-O notations, Linked list, Sorting, Searching, recursion,,, Discrete Mathematics: Set theory, Logic, Recurrence equation, Matrix operations,,, Programming experience Debasis Mitra11

Syllabus overview Introduction Four algorithm types: divide-conquer, greedy, dynamic programming, branch-bound Graph algorithms (a few only!) Complexity theory – NP-completeness An advanced topic, e.g. linear programming Undergrad: GPU coding project; Grad: An advanced topic self-study report Debasis Mitra12

Syllabus objective More conscious problem solver Exposure to proving correctness of algorithm (and its connection to both solving a problem, and computing its complexity) Fundamental understanding of computing problem complexity Undergrad project: GPU computing; Grad project: introduction to a current topic Debasis Mitra13

Example: MAXIMUM SUBSEQUENCE SUM (MSQS) Problem MSSQ: Given an array of numbers find a subsequence whose sum is maximum out of all such subsequences. Example: 3, 4, –7, 1, 9, -2, 3, -1 (e.g. rise and fall in stock-market) Answer: 11 (for subsequence = 11) [Note: for all positive integers the answer is sum of the whole array.] Debasis Mitra 14

MSQS Algorithm 1 Input: Array A[N], e.g. [3, 4, –7, 1, 9, -2, 3, -1 ] Output: Maximum subsequence sum, e.g., 11 Algorithm 1: 1. maxSum = 0; // We expect non-negative value here at the end 2.for (i = 0 through N-1) do 3.for (j = i through N-1) do // choice of subsequence i through j 4.{thisSum = 0; 5.for (k = i through j) do // addition loop 6.thisSum = thisSum + a[k]; // O(N 3 ) 7. 8.if (thisSum > maxSum) then 9.maxSum = thisSum; // O(N 2 ) 10.} 11.return maxSum; End Algorithm 1. Analysis of Algorithm 1:  i=0 N-1  j=i N-1  k=i j 1 = … = O(N 3 ) Debasis Mitra 15

MSQS Algorithm 2 Input: Array A[N], e.g. [3, 4, –7, 1, 9, -2, 3, -1 ] Output: Maximum subsequence sum, e.g., 11 Algorithm 2: 1. maxSum = 0; // We expect non-negative value here at the end 2.for (i = 0 through N-1) do 3.thisSum = 0; 4.for (j = i through N-1) do 5.{thisSum = thisSum + a[j]; // reuse the partial sum from 6.// the previous iteration 7.if (thisSum > maxSum) then 8.maxSum = thisSum; 9.} 10.return maxSum; End Algorithm 2. Analysis of Algorithm 2:  i=0 N-1  j=i N-1 1 = … = O(N 2 ) Debasis Mitra 16

MSQS Algorithm 3 Weiss, textbook, 1999 Complexity: T(n) = 2T(n/2) + O(N) T(1) = 1 Solve: T(n) = O(n log n) Inductive Proof: (base) n=1; (hypothesis) lines 15, 16 & work correctly for n=2^k (step) These are only three options; So, lines returns correct sum. Debasis Mitra 17

MSQS Algorithm 4 Input: Array A[N], e.g. [3, 4, –7, 1, 9, -2, 3, -1 ] Output: Maximum subsequence sum, e.g., 11 Algorithm 4: 1.maxSum = 0;thisSum = 0; 2.for (j = 0 through N-1) do 3.{thisSum = thisSum + a[j]; // reuse the partial sum from 4.// the previous iteration 5.if (thisSum > maxSum) then 6. maxSum = thisSum; 7.else if (thisSum < 0) then 8. thisSum =0;// ignore computations so far 9.} 10.return maxSum; End Algorithm 4. Analysis of Algorithm 4:  i=0 N-1 O(1) = … = O(N) Exercise: (1) How to track the start-end indices for the maxSum solution? (2) Prove the algorithm 4, stating any underlying assumption/hypothesis. Debasis Mitra 18

Proof of MSQS Algo-4 Lemma 1: Maximum-sub-sequence-sum is non negative. Proof: trivial. (the problem is defined as such) Lemma 2: No pre-fix of a non-negative maximum-sub-sequence-sum (MSQS) can be negative. Proof by contradiction: Presume such a MSQS P =(W Q), W is prefix and Q is suffix parts. –We know, ∑P = ∑W + ∑Q; if ∑W <0 then ∑P < ∑Q; Hence, ∑P is not MSQS. Thm: MSQS Algo-4 finds the MSQS correctly. Proof sketch by induction on loop invariant maxSum: –Base: for null input {} maxSum=0 is MSQS; –Hypothesis: at every iteration j = k, maxSum is MSQS up to (a 1, a 2, …, a k-1 ) –Step: if maxSum+a k <0 then by Lemma 2 it cannot be prefix in any MSQS, –and step 8 rejects the sequence built so far. Input: Array A[N], e.g. [3, 4, –7, 1, 9, -2, 3, -1 ] Output: Maximum subsequence sum, e.g., 11 Algorithm 4: 1.maxSum = 0;thisSum = 0; 2.for (j = 0 through N-1) do 3.{thisSum = thisSum + a[j]; // reuse the partial sum from 4.// the previous iteration 5.if (thisSum > maxSum) then 6. maxSum = thisSum; 7.else if (thisSum < 0) then 8. thisSum =0;// ignore computations so far 9.} 10.return maxSum; End Algorithm 4. Debasis Mitra 19

What do we analyze in an algorithm, why, and how do we analyze? Basic O-notations How complexity matters Exercise: Code the following two fibonacci series algorithms and increase input values, check time. What are the step/time-complexities? Space-complexities? Input: an integer n;Output: Fibonacci number for n Summary Recursive Algorithm: Fib(n) { 1. if n = = 0 or n = = 1 return 1; 2. else return (Fib(n-1) + Fib(n-2)); End Fib. Iterative version: FibIt(n) { 1. temp1 = 1; temp2=1; 2. for i=3:n do { 3. temp3 = temp1 + temp2; 4. temp1 = temp2; temp2 = temp3; } 5. end for; 6. return temp2; End FibIt. Debasis Mitra 20