David Luebke 1 8/17/2015 CS 332: Algorithms Asymptotic Performance

David Luebke 2 8/17/2015 Review: Asymptotic Performance Asymptotic performance: How does algorithm behave as the problem size gets very large? oRunning time oMemory/storage requirements  Remember that we use the RAM model: oAll memory equally expensive to access oNo concurrent operations oAll reasonable instructions take unit time ] Except, of course, function calls oConstant word size ] Unless we are explicitly manipulating bits

David Luebke 3 8/17/2015 Review: Running Time Number of primitive steps that are executed  Except for time of executing a function call most statements roughly require the same amount of time  We can be more exact if need be Worst case vs. average case

David Luebke 4 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } }

David Luebke 5 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i =  j =  key =  A[j] =  A[j+1] = 

David Luebke 6 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 1key = 10 A[j] = 30 A[j+1] = 10

David Luebke 7 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

David Luebke 8 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

David Luebke 9 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 0key = 10 A[j] =  A[j+1] = 30

David Luebke 10 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 0key = 10 A[j] =  A[j+1] = 30

David Luebke 11 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 0key = 10 A[j] =  A[j+1] = 10

David Luebke 12 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 0key = 10 A[j] =  A[j+1] = 10

David Luebke 13 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 0key = 40 A[j] =  A[j+1] = 10

David Luebke 14 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 0key = 40 A[j] =  A[j+1] = 10

David Luebke 15 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

David Luebke 16 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

David Luebke 17 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

David Luebke 18 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 40 A[j] = 30 A[j+1] = 40

David Luebke 19 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

David Luebke 20 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

David Luebke 21 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

David Luebke 22 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

David Luebke 23 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

David Luebke 24 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

David Luebke 25 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

David Luebke 26 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

David Luebke 27 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

David Luebke 28 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

David Luebke 29 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

David Luebke 30 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

David Luebke 31 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

David Luebke 32 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20

David Luebke 33 8/17/2015 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20 Done!

David Luebke 34 8/17/2015 Animating Insertion Sort Check out the Animator, a java applet at: Try it out with random, ascending, and descending inputs

David Luebke 35 8/17/2015 Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } What is the precondition for this loop?

David Luebke 36 8/17/2015 Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } How many times will this loop execute?

David Luebke 37 8/17/2015 Insertion Sort Statement Effort InsertionSort(A, n) { for i = 2 to n { c 1 n key = A[i] c 2 (n-1) j = i - 1; c 3 (n-1) while (j > 0) and (A[j] > key) { c 4 T A[j+1] = A[j] c 5 (T-(n-1)) j = j - 1 c 6 (T-(n-1)) } 0 A[j+1] = key c 7 (n-1) } 0 } T = t 2 + t 3 + … + t n where t i is number of while expression evaluations for the i th for loop iteration

David Luebke 38 8/17/2015 Analyzing Insertion Sort T(n) = c 1 n + c 2 (n-1) + c 3 (n-1) + c 4 T + c 5 (T - (n-1)) + c 6 (T - (n-1)) + c 7 (n-1) = c 8 T + c 9 n + c 10 What can T be?  Best case -- inner loop body never executed ot i = 1  T(n) is a linear function  Worst case -- inner loop body executed for all previous elements ot i = i  T(n) is a quadratic function  Average case o???

David Luebke 39 8/17/2015 Analysis Simplifications  Ignore actual and abstract statement costs  Order of growth is the interesting measure: oHighest-order term is what counts ] Remember, we are doing asymptotic analysis ] As the input size grows larger it is the high order term that dominates

David Luebke 40 8/17/2015 Upper Bound Notation We say InsertionSort’s run time is O(n 2 )  Properly we should say run time is in O(n 2 )  Read O as “Big-O” (you’ll also hear it as “order”) In general a function  f(n) is O(g(n)) if there exist positive constants c and n 0 such that f(n)  c  g(n) for all n  n 0 Formally  O(g(n)) = { f(n):  positive constants c and n 0 such that f(n)  c  g(n)  n  n 0

David Luebke 41 8/17/2015 Insertion Sort Is O(n 2 ) Proof  Suppose runtime is an 2 + bn + c oIf any of a, b, and c are less than 0 replace the constant with its absolute value  an 2 + bn + c  (a + b + c)n 2 + (a + b + c)n + (a + b + c)   3(a + b + c)n 2 for n  1  Let c’ = 3(a + b + c) and let n 0 = 1 Question  Is InsertionSort O(n 3 )?  Is InsertionSort O(n)?

David Luebke 42 8/17/2015 Big O Fact A polynomial of degree k is O(n k ) Proof:  Suppose f(n) = b k n k + b k-1 n k-1 + … + b 1 n + b 0 oLet a i = | b i |  f(n)  a k n k + a k-1 n k-1 + … + a 1 n + a 0

David Luebke 43 8/17/2015 Lower Bound Notation We say InsertionSort’s run time is  (n) In general a function  f(n) is  (g(n)) if  positive constants c and n 0 such that 0  c  g(n)  f(n)  n  n 0 Proof:  Suppose run time is an + b oAssume a and b are positive (what if b is negative?)  an  an + b

David Luebke 44 8/17/2015 Asymptotic Tight Bound A function f(n) is  (g(n)) if  positive constants c 1, c 2, and n 0 such that c 1 g(n)  f(n)  c 2 g(n)  n  n 0 Theorem  f(n) is  (g(n)) iff f(n) is both O(g(n)) and  (g(n))  Proof: someday

David Luebke 45 8/17/2015 Practical Complexity

David Luebke 46 8/17/2015 Practical Complexity

David Luebke 47 8/17/2015 Practical Complexity

David Luebke 48 8/17/2015 Practical Complexity

David Luebke 49 8/17/2015 Practical Complexity

David Luebke 50 8/17/2015 Other Asymptotic Notations A function f(n) is o(g(n)) if  positive constants c and n 0 such that f(n) < c g(n)  n  n 0 A function f(n) is  (g(n)) if  positive constants c and n 0 such that c g(n) < f(n)  n  n 0 Intuitively,  o() is like <  O() is like    () is like >   () is like    () is like =

David Luebke 51 8/17/2015 Up Next Solving recurrences  Substitution method  Master theorem