1. Analysis of Algorithm Efficiency Dr. Yingwu Zhu p5-11, p16-29, p43-53, p93-96

2. What is Algorithm? Any well-defined computational procedure that: – takes input: some value or a set of values and – produces output: some value or a set of values Example: sorting problems – Input: a sequence of numbers {a1, a2, …, an} – Output: a permutation {b1, b2, …, bn} of the input s.t. b1<=b2<=…<=bn

3. Algorithms: Motivation At the heart of programs lie algorithms Algorithms as a key technology Think about: – mapping/navigation – Google search – word processing (spelling correction, layout…) – content delivery and streaming video – games (graphics, rendering…) – big data (querying, learning…)

4. Algorithms: Motivation In a perfect world – for each problem we would have an algorithm – the algorithm would be the fastest possible What would CS look like in this world?

5. Algorithms: Motivation Our world (fortunately) is not so perfect: – for many problems we know embarrassingly little about what the fastest algorithm is multiplying two integers or two matrices factoring an integer into primes determining shortest tour of given n cities – for many problems we suspect fast algorithms are impossible (NP-complete problems) – for some problems we have unexpected and clever algorithms (we will see many of these)

6. Analyzing Algorithms Given computing resources and input data – how fast does an algorithm run? Time efficiency: amount of time required to accomplish the task Our focus – How much memory is required? Space efficiency: amount of memory required Deals with the extra space the algorithm requires

7. Time Efficiency Time efficiency depends on : – size of input – speed of machine – quality of source code – quality of compiler These vary from one platform to another So, we cannot express time efficiency meaningfully in real time units such as seconds!

8. Empirical analysis of time efficiency Select a specific (typical) sample of inputs Use physical unit of time (e.g., milliseconds) or Count actual number of basic operation’s executions Analyze the empirical data Limitation: results dependent on the particular computer and the sample of inputs

9. Theoretical analysis of time efficiency Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size Basic operation: the operation that contributes most towards the running time of the algorithm T(n) ≈ c op C(n ) running time execution time for basic operation Number of times basic operation is executed Input size

10. Input size and basic operation examples ProblemInput size measureBasic operation Searching for key in a list of n items Number of list’s items, i.e. n Key comparison Multiplication of two matrices Matrix dimensions or total number of elements Multiplication of two numbers Checking primality of a given integer n n’size = number of digits (in binary representation) Division Typical graph problem#vertices and/or edges Visiting a vertex or traversing an edge

11. Time Efficiency T(n) = (approximated by) number of times the basic operation is executed. Not only depends on the input size n, but also depends on the arrangement of the input items – Best case: not informative – Average case: difficult to determine – Worst case: is used to measure an algorithm’s performance

12. Example: Sequential Search Best case T(n) Worst case T(n) Average case T(n) – Assume success search probability of p

13. Order of Growth Established framework for analyzing T(n) Order of growth as n→∞ – Highest-order term is what counts Remember, we are doing asymptotic analysis As the input size grows larger it is the high order term that dominates disregard lower-order terms in running time – disregard coefficient on highest order term Asymptotic notations: , O, 

14. Asymptotic Order of Growth A way of comparing functions that ignores constant factors and small input sizes O(g(n)): class of functions f(n) that grow no faster than g(n) Θ (g(n)): class of functions f(n) that grow at same rate as g(n) Ω (g(n)): class of functions f(n) that grow at least as fast as g(n)

15. Big-oh

16. Big-omega

17. Big-theta

18. Upper Bound Notation In general a function – f(n) is O(g(n)) if there exist positive constants c and n 0 such that f(n)  c  g(n) for all n  n 0 – Order of growth of f(n)  order of growth of g(n) (within constant multiple) Formally – O(g(n)) = { f(n):  positive constants c and n 0 such that f(n)  c  g(n)  n  n 0

19. Big-Oh Examples – 10n is O(n 2 ) – 5n+20 is O(n)

20. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } }

21. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i =  j =  key =  A[j] =  A[j+1] = 

22. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 10

23. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

24. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

25. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 30

26. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 30

27. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 10

28. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 10 A[j] =  A[j+1] = 10

29. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 40 A[j] =  A[j+1] = 10

30. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 40 A[j] =  A[j+1] = 10

31. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

32. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

33. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

34. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 40 A[j] = 30 A[j+1] = 40

35. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

36. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

37. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

38. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

39. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

40. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

41. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

42. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

43. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

44. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

45. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

46. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

47. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

48. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10203040 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20

49. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10203040 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20 Done!

50. Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } How many times will this loop execute?

51. T(n) for Insertion Sort Worst case? Best case?

52. Insertion Sort Is O(n 2 ) Proof – Suppose runtime is an 2 + bn + c If any of a, b, and c are less than 0 replace the constant with its absolute value – an 2 + bn + c  (a + b + c)n 2 + (a + b + c)n + (a + b + c) –  3(a + b + c)n 2 for n  1 – Let c’ = 3(a + b + c) and let n 0 = 1 Question – Is InsertionSort O(n 3 )? – Is InsertionSort O(n)?

53. Big O Fact A polynomial of degree k is O(n k ) Proof: – Suppose f(n) = b k n k + b k-1 n k-1 + … + b 1 n + b 0 Let a i = | b i | – f(n)  a k n k + a k-1 n k-1 + … + a 1 n + a 0

54. 10/16/2015 Lower Bound Notation In general a function – f(n) is  (g(n)) if  positive constants c and n 0 such that 0  c  g(n)  f(n)  n  n 0 Proof: – Suppose run time is an + b Assume a and b are positive (what if b is negative?) – an  an + b – Example: n 3 =  (n 2 )

55. 10/16/2015 Asymptotic Tight Bound A function f(n) is  (g(n)) if  positive constants c 1, c 2, and n 0 such that c 1 g(n)  f(n)  c 2 g(n)  n  n 0 Theorem – f(n) is  (g(n)) iff f(n) is both O(g(n)) and  (g(n)) – Proof Engineering – Drop low-order items; ignore leading constants – Example: 3n 3 + 90n 2 -5n + 9999 =  (n 3 )

56. Using limits for comparing orders of growth T(n) has a smaller order of growth than g(n) T(n) has the same order of growth than g(n) T(n) has a larger order of growth than g(n) Example: compare the orders of growth of and n^2

57. Practical Complexity

60. David Luebke 60 10/16/2015 Practical Complexity

61. Properties of Big-O f(n)  O(f(n)) f(n)  O(g(n)) iff g(n)  (f(n)) If f (n)  O(g (n)) and g(n)  O(h(n)), then f(n)  O(h(n)) Note similarity with a ≤ b If f 1 (n)  O(g 1 (n)) and f 2 (n)  O(g 2 (n)), then f 1 (n) + f 2 (n)  O(max{g 1 (n), g 2 (n)})

62. Basic asymptotic efficiency classes 1constant log nlogarithmic nlinear n log nn-log-n n2n2 quadratic n3n3 cubic 2n2n exponential n!n!factorial

63. Summary: Analysis for Algorithms Focus on worst case – well-suited to rigorous analysis, simple Measure time/space complexity using asymptotic notation (“big-oh notation”) – disregard lower-order terms in running time – disregard coefficient on highest order term Non-recursive algorithms Recursive algorithms

64. T(n) for nonrecursive algorithms General Plan for Analysis Decide on parameter n indicating input size Identify algorithm’s basic operation Determine worst, average, and best cases for input of size n Set up a sum for the number of times the basic operation is executed Simplify the sum using standard formulas and rules – Disregard low-order terms & coefficient on the highest-order term

65. Useful summation formulas and rules  l  i  u 1 = 1+1+…+1 = u - l + 1 In particular,  1  i  n 1 = n - 1 + 1 = n   (n)  1  i  n i = 1+2+…+n = n(n+1)/2  n 2 /2   (n 2 )  1  i  n i 2 = 1 2 +2 2 +…+n 2 = n(n+1)(2n+1)/6  n 3 /3   (n 3 )  0  i  n a i = 1 + a +…+ a n = (a n+1 - 1)/(a - 1) for any a  1 In particular,  0  i  n 2 i = 2 0 + 2 1 +…+ 2 n = 2 n+1 - 1   (2 n )  (a i ± b i ) =  a i ±  b i  ca i = c  a i  l  i  u a i =  l  i  m a i +  m+1  i  u a i

66. Example 1: Maximum element

67. Example 2: Element uniqueness problem

68. Example 3: Matrix multiplication

69. Plan for Analysis of Recursive Algorithms Decide on a parameter indicating an input’s size. Identify the algorithm’s basic operation. Check whether the number of times the basic op. is executed may vary on different inputs of the same size. (If it may, the worst, average, and best cases must be investigated separately.) Set up a recurrence relation with an appropriate initial condition expressing the number of times the basic op. is executed. Solve the recurrence (or, at the very least, establish its solution’s order of growth) by backward substitutions or another method.

70. Example 1: Recursive evaluation of n! Definition: n ! = 1  2  …  (n-1)  n for n ≥ 1 and 0! = 1 Recursive definition of n!: F(n) = F(n-1)  n for n ≥ 1 and F(0) = 1 Size: Basic operation: Recurrence relation:

71. Example 2: The Tower of Hanoi Puzzle Recurrence for number of moves: n disks of different sizes and 3 pegs; the goal is to move all the disks to the third peg using the 2 nd one as an auxiliary if necessary. Move on disk at a time and do not place a larger disk on top of a smaller one!

72. The Tower of Hanoi Puzzle T(n) = 2T(n-1) + 1 T(1) = 1

73. Example 3: Binary Search

74. Master Theorem T(n) = aT(n/b) + f (n), where f(n)   (n d ), d  0 Master Theorem: If a < b d, T(n)   (n d ) If a = b d, T(n)   (n d log n) If a > b d, T(n)   (n log b a )