1. Introduction to Algorithms 6.046J/18.401J/SMA5503 Lecture 1 Prof. Charles E. Leiserson

2. Analysis of algorithms The theoretical study of computer-program performance and resource usage. What ’ s more important than performance? modularity user-friendliness correctness programmer time maintainability simplicity functionality extensibility robustness reliability

3. Why study algorithms and performance? Algorithms help us to understand scalability. Performance often draws the line between what is feasible and what is impossible. Algorithmic mathematics provides a language for talking about program behavior. The lessons of program performance generalize to other computing resources. Speed is fun!

4. The problem of sorting Input: sequence of numbers. Output: permutation such that a' 1 ≤ a' 2 ≤ … ≤ a' n Example: Input: 8 2 4 9 3 6 Output: 2 3 4 6 8 9

5. Insertion sort

6. Example of insertion sort 8 2 4 9 3 6

7. Example of insertion sort 8 24 9 3 6

8. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6

9. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6

10. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6

11. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6

12. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6

13. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6

14. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 2 3 4 8 9 6

15. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 2 3 4 8 9 6

16. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 2 3 4 8 9 6 2 3 4 6 8 9 done

17. Running time The running time depends on the input: an already sorted sequence is easier to sort. Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones. Generally, we seek upper bounds on the running time, because everybody likes a guarantee.

18. Insertion Sort Statement Effort InsertionSort(A, n) { for i = 2 to n { c 1 n key = A[i] c 2 (n-1) j = i - 1; c 3 (n-1) while (j > 0) and (A[j] > key) { c 4 T A[j+1] = A[j] c 5 (T-(n-1)) j = j - 1 c 6 (T-(n-1)) } 0 A[j+1] = key c 7 (n-1) } 0 } T = t 2 + t 3 + … + t n where t i is number of while expression evaluations for the i th for loop iteration

19. David Luebke 19 2/21/2016 Analyzing Insertion Sort T(n) = c 1 n + c 2 (n-1) + c 3 (n-1) + c 4 T + c 5 (T - (n-1)) + c 6 (T - (n-1)) + c 7 (n-1) = c 8 T + c 9 n + c 10T(n) = c 1 n + c 2 (n-1) + c 3 (n-1) + c 4 T + c 5 (T - (n-1)) + c 6 (T - (n-1)) + c 7 (n-1) = c 8 T + c 9 n + c 10 What can T be?What can T be? Best case -- inner loop body never executedBest case -- inner loop body never executed t i = 1  T(n) is a linear functiont i = 1  T(n) is a linear function Worst case -- inner loop body executed for all previous elementsWorst case -- inner loop body executed for all previous elements t i = i  T(n) is a quadratic functiont i = i  T(n) is a quadratic function Average caseAverage case ??????

20. Kinds of analyses Worst-case: (usually) T(n) = maximum time of algorithm on any input of size n. Average-case: (sometimes) T(n) = expected time of algorithm over all inputs of size n. Need assumption of statistical distribution of inputs. Best-case: (bogus) Cheat with a slow algorithm that works fast on some input.

21. Machine-independent time What is insertion sort ’ s worst-case time? It depends on the speed of our computer: relative speed (on the same machine), absolute speed (on different machines). BIG IDEA: Ignore machine-dependent constants. Look at growth of T(n) as n → ∞. “Asymptotic Analysis”

22. David Luebke 22 2/21/2016 Asymptotic Performance In this course, we care most about asymptotic performanceIn this course, we care most about asymptotic performance How does the algorithm behave as the problem size gets very large?How does the algorithm behave as the problem size gets very large? Running timeRunning time Memory/storage requirementsMemory/storage requirements Bandwidth/power requirements/logic gates/etc.Bandwidth/power requirements/logic gates/etc.

23. Analysis of Algorithms Analysis is performed with respect to a computational modelAnalysis is performed with respect to a computational model We will usually use a generic uniprocessor random-access machine (RAM)We will usually use a generic uniprocessor random-access machine (RAM) All memory equally expensive to accessAll memory equally expensive to access No concurrent operationsNo concurrent operations All reasonable instructions take unit timeAll reasonable instructions take unit time Except, of course, function callsExcept, of course, function calls Constant word sizeConstant word size Unless we are explicitly manipulating bitsUnless we are explicitly manipulating bits

24. David Luebke 24 2/21/2016 Input Size Time and space complexityTime and space complexity This is generally a function of the input sizeThis is generally a function of the input size E.g., sorting, multiplicationE.g., sorting, multiplication How we characterize input size depends:How we characterize input size depends: Sorting: number of input itemsSorting: number of input items Multiplication: total number of bitsMultiplication: total number of bits Graph algorithms: number of nodes & edgesGraph algorithms: number of nodes & edges EtcEtc

25. David Luebke 25 2/21/2016 Analysis SimplificationsSimplifications Ignore actual and abstract statement costsIgnore actual and abstract statement costs Order of growth is the interesting measure:Order of growth is the interesting measure: Highest-order term is what countsHighest-order term is what counts Remember, we are doing asymptotic analysisRemember, we are doing asymptotic analysis As the input size grows larger it is the high order term that dominatesAs the input size grows larger it is the high order term that dominates

26. Θ-notation Math: Θ(g(n)) = { f (n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n) for all n ≥ n0 } Engineering: Drop low-order terms; ignore leading constants. Example: 3n 3 + 90n 2 – 5n + 6046 = Θ(n 3 )

27. Asymptotic performance When n gets large enough, a Θ(n 2 ) algorithm always beats a Θ(n 3 ) algorithm. We shouldn ’ t ignore asymptotically slower algorithms, however. Real-world design situations often call for a careful balancing of engineering objectives. Asymptotic analysis is a useful tool to help to structure our thinking.

28. David Luebke 28 2/21/2016 Practical Complexity

29. David Luebke 29 2/21/2016 Practical Complexity

30. David Luebke 30 2/21/2016 Practical Complexity

31. David Luebke 31 2/21/2016 Practical Complexity

32. David Luebke 32 2/21/2016 Practical Complexity

33. Insertion sort analysis Worst case: Input reverse sorted. [arithmetic series] Average case: All permutations equally likely. Is insertion sort a fast sorting algorithm? Moderately so, for small n. Not at all, for large n.

34. Merge sort

35. Merge Sort

36. MERGE

37. Merging two sorted 20 12 20 12 13 11 13 11 7 9 7 9 2 1 2 1

38. Merging two sorted 20 12 20 12 13 11 13 11 7 9 7 9 2 1 2 1 1

39. Merging two sorted 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 7 9 7 9 7 9 7 9 2 1 2 2 1 2 1

40. Merging two sorted 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 7 9 7 9 7 9 7 9 2 1 2 2 1 2 1 2 1 2

41. Merging two sorted 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 11 7 9 7 9 7 9 7 9 7 9 7 9 2 1 2 2 1 2 1 2 1 2

42. Merging two sorted arrays 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 11 7 9 7 9 7 9 7 9 7 9 7 9 2 1 2 2 1 2 1 2 7 1 2 7

43. Merging two sorted arrays 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 11 13 11 13 11 7 9 7 9 7 9 9 7 9 7 9 7 9 9 2 1 2 2 1 2 1 2 7 1 2 7

44. ○ Merging two sorted arrays 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 11 13 11 13 11 7 9 7 9 7 9 9 7 9 7 9 7 9 9 2 1 2 2 1 2 1 2 79 1 2 79

45. Merging two sorted arrays 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 11 13 11 13 11 13 11 13 11 7 9 7 9 7 9 9 7 9 7 9 7 9 9 2 1 2 2 1 2 1 2 7 9 1 2 7 9

46. Merging two sorted arrays 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 11 13 11 13 11 13 11 13 11 7 9 7 9 7 9 9 7 9 7 9 7 9 9 2 1 2 2 1 2 1 2 7 9 11 1 2 7 9 11

47. Merging two sorted arrays 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 13 11 13 11 13 11 13 11 13 11 13 7 9 7 9 7 9 9 7 9 7 9 7 9 9 2 1 2 2 1 2 1 2 7 9 11 1 2 7 9 11

48. Merging two sorted arrays 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 13 11 13 11 13 11 13 11 13 11 13 7 9 7 9 7 9 9 7 9 7 9 7 9 9 2 1 2 2 1 2 1 2 7 9 11 12 1 2 7 9 11 12

49. Merging two sorted arrays 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 20 12 13 11 13 11 13 11 13 11 13 11 13 13 11 13 11 13 11 13 11 13 11 13 7 9 7 9 7 9 9 7 9 7 9 7 9 9 2 1 2 2 1 2 1 2 7 9 11 12 Time = Θ(n) to merge a total of n elements (linear time).

50. Analyzing merge sort T(n) MERGE-SORT A[1.. n] Θ(1) 1. If n = 1, done. 2T(n/2) 2. Recursively sort A[ 1.. n/2 ] Abuse and A[ n/2+1.. n ]. Θ(n) 3. “ Merge ” the 2 sorted lists Sloppiness: Should be T( n/2 ) + T( n/2 ), but it turns out not to matter asymptotically.

51. Recurrence for merge sort T(n) = Θ(1) if n = 1; 2T(n/2) + Θ(n) if n > 1. We shall usually omit stating the base case when T(n) = Θ(1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence. CLRS and Lecture 2 provide several ways to find a good upper bound on T(n).

52. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

53. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. T(n)

54. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn T(n/2)

55. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/2 cn/2 T(n/4) T(n/4)

56. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/2 cn/2 T(n/4) T(n/4) T(n/4) T(n/4) Θ(1)

57. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/2 cn/2 h = lg n T(n/4) T(n/4) T(n/4) T(n/4) Θ(1)

58. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn/2 h = lg n T(n/4) T(n/4) T(n/4) T(n/4) Θ(1)

59. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn/2 cn h = lg n T(n/4) T(n/4) T(n/4) T(n/4) Θ(1)

60. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn/2 cn h = lg n T(n/4) T(n/4) T(n/4) T(n/4) cn Θ(1)

61. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn/2 cn h = lg n T(n/4) T(n/4) T(n/4) T(n/4) cn Θ(1) Θ(n) # leaves = n

62. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn/2 cn h = lg n T(n/4) T(n/4) T(n/4) T(n/4) cn Θ(n) Θ(1) Θ(n lg n) # leaves = n

63. Conclusions Θ(n lg n) grows more slowly than Θ( n 2 ). Therefore, merge sort asymptotically beats insertion sort in the worst case. In practice, merge sort beats insertion sort for n > 30 or so. Go test it out for yourself!