1. 1Computer Sciences Department

3. Book: Introduction to Algorithms, by: Thomas H. Cormen Charles E. Leiserson Ronald L. Rivest Clifford Stein Electronic: http://ocw.mit.edu/courses/electrical- engineering-and-computer-science/6-006-introduction- to-algorithms-fall-2011/ Reference 3Computer Sciences Department

4. The Role of Algorithms in Computing 4Computer Sciences Department

5.  The Role of Algorithms in Computing  The problem of sorting  What kinds of problems are solved by algorithms?  Hard problems  Algorithms as a technology  Insertion sort  Analysis of insertion sort  Example of insertion sort  (Best, Worst and Average) case analysis  Merge sort  The divide-and-conquer approach  Analyzing merge sort  Example of merge sort  Recursion tree Lecture Contents (objectives) 5Computer Sciences Department

6.  What are algorithms?  Why is the study of algorithms worthwhile?  What is the role of algorithms relative to other technologies used in computers? The Role of Algorithms in Computing 6Computer Sciences Department

7.  Informally, an algorithm is any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output.  An algorithm is thus a sequence of computational steps that transform the input into the output. Algorithms 7Computer Sciences Department

8. Design and Analysis of Algorithms Analysis: predict the cost of an algorithm in terms of resources and performance Design: design algorithms which minimize the cost 8Computer Sciences Department

9.  Computational problem. Algorithms In general, an instance of a problem consists of the input (satisfying whatever constraints are imposed in the problem statement) needed to compute a solution to the problem. 9Computer Sciences Department

10. The problem of sorting Input: sequence of numbers. Example: Input: 8 1 4 9 3 7 Output: 1 3 4 7 8 9 Output: permutation such that a' 1  a' 2  …  a' n. 10Computer Sciences Department

11.  An algorithm is said to be correct if, for every input instance, it halts with the correct output 11Computer Sciences Department

12.  The Human Genome Project - has the goals of identifying all the 100,000 genes in human DNA, determining the sequences of the 3 billion chemical base pairs that make up human DNA - storing this information in databases, and developing tools for data analysis.  The Internet enables people all around the world to quickly access and retrieve large amounts of information. (to manage and manipulate this large volume of data) What kinds of problems are solved by algorithms? 12Computer Sciences Department

13.  Electronic commerce.  In manufacturing and other commercial settings. What kinds of problems are solved by algorithms? (cont’d) Example: an equation ax ≡ b (mod n), where a, b, and n are integers, and we wish to find all the integers x, modulo n, that satisfy the equation. There may be zero, one, or more than one such solution. We can simply try x = 0, 1,..., n − 1 in order, but Chapter 31 shows a more efficient method. 13Computer Sciences Department

14.  A data structure is a way to store and organize data in order to facilitate access and modifications. Data structure Technique 14Computer Sciences Department

15.  Efficient algorithms.  NP-complete problems. (decision problem) –  First, although no efficient algorithm for an NP- complete problem has ever been found, nobody has ever proven that an efficient algorithm for one cannot exist.  It is unknown whether or not efficient algorithms exist for NP-complete problems. Hard problems 15Computer Sciences Department

16. 16Computer Sciences Department

17.  Would you have any reason to study algorithms? YES.  If computers were infinitely fast, any correct method for solving a problem would do.  Computers may be fast, but: - memory may be cheap, but it is not free. Computing time is therefore a bounded resource, and so is space in memory. These resources should be used wisely, and algorithms that are efficient in terms of time or space will help you do so. Algorithms as a technology 17Computer Sciences Department

18.  Which computer/ algorithm is faster? Very important 18Computer Sciences Department

19. 19Computer Sciences Department

20. 20Computer Sciences Department

21. Getting Started 21Computer Sciences Department

22. Running time The running time depends on the input: an already sorted sequence is easier to sort. Major Simplifying Convention: Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones.  T A (n) = time of A on length n inputs Generally, seek upper bounds on the running time, to have a guarantee of performance. 22Computer Sciences Department

23. Kinds of analyses Worst-case: (usually) T(n) = maximum time of algorithm on any input of size n. (upper bound on the running time) Average-case: (sometimes) T(n) = expected time of algorithm over all inputs of size n. Best-case: T(n) = minimum time of algorithm ((fastest time to complete, with optimal inputs chosen) (lower bound on the running time)) 23Computer Sciences Department

24.  The best case, in which the input array was already sorted, and the worst case, in which the input array was reverse sorted.  The worst-case running time of an algorithm is an upper bound on the running time for any input. Worst-case and average-case analysis How long does it take to determine where in sub-array A[1.. j − 1] to insert element A[ j ]? 24Computer Sciences Department

25.  Suppose that we randomly choose n numbers and apply insertion sort. How long does it take to determine where in sub-array A[1.. j − 1] to insert element A[ j ]?  On average, half the elements in A[1.. j − 1] are less than A[ j ], and half the elements are greater.  On average, therefore, we check half of the sub-array A[1.. j − 1], so t j = j/2. Average case 25Computer Sciences Department

26. Insertion sort The numbers that we wish to sort are also known as the keys Insertion sort, is an efficient algorithm for sorting a small number of elements. 6531872465318724 Start One move / comparing / insert it into the correct position 26Computer Sciences Department

27. Insertion sort (cont’d) 27Computer Sciences Department

28. Insertion sort (cont’d) “Pseudo-code conventions” ij key sorted A:A: 1n Length[A]=n 28Computer Sciences Department

29.  predicting the resources that the algorithm requires.  random-access machine (RAM)- Instructions are (executed one after another, with no concurrent operations)  The data types in the RAM model are integer and floating point and limit on the size of each word of data.  Is exponentiation a constant time instruction? “shift left” instruction. Analyzing algorithms 29Computer Sciences Department

30.  The time taken by the INSERTION-SORT procedure depends on the input.  INSERTION SORT can take different amounts of time: - to sort two input sequences of the same size depending on how nearly sorted they already are. - to sort thousand numbers or three numbers.  The running time of an algorithm on a particular input is the number of primitive operations or “steps” executed. Analysis of insertion sort 30Computer Sciences Department

31. Analysis of insertion sort (cont’d) Best case Worst case 31 Computer Sciences Department

32. Analysis of insertion sort (cont’d) 32Computer Sciences Department

33. Explanation 33Computer Sciences Department a b c =an 2 +bn-c

34. Example of insertion sort 814937 34Computer Sciences Department

35. Example of insertion sort 814937 35Computer Sciences Department

36. Example of insertion sort 814937 184937 36Computer Sciences Department

37. Example of insertion sort 814937 184937 37Computer Sciences Department

38. Example of insertion sort 814937 184937 148937 38Computer Sciences Department

39. Example of insertion sort 814937 184937 148937 39Computer Sciences Department

40. Example of insertion sort 814937 184937 148937 148937 40Computer Sciences Department

41. Example of insertion sort 814937 184937 148937 148937 41Computer Sciences Department

42. Example of insertion sort 814937 184937 148937 148937 134897 42Computer Sciences Department

43. Example of insertion sort 134897 814937 184937 148937 148937 43Computer Sciences Department

44. Example of insertion sort 134789 134897 814937 184937 148937 148937 44Computer Sciences Department

45.  Divide the n-element sequence to be sorted into two subsequences of n/2 elements each.  The merge sort algorithm closely follows the divide-and- conquer paradigm. The divide-and-conquer approach 45Computer Sciences Department

46. 46Computer Sciences Department

47. M ERGE -S ORT M ERGE -S ORT A[1.. n] 1.If n = 1, done. 2.Recursively sort A[ 1..  n/2  ] and A[  n/2  +1.. n ]. 3.“Merge” the 2 sorted lists. 47Computer Sciences Department

48. Analyzing merge sort M ERGE -S ORT A[1.. n] 1.If n = 1, done. 2.Recursively sort A[ 1..  n/2  ] and A[  n/2  +1.. n ]. 3.“Merge” the 2 sorted lists T(n)  (1) 2T(n/2)  (n) T(n) =  (1) if n = 1; 2T(n/2) +  (n) if n > 1. Recurrence for merge sort 48Computer Sciences Department

49. 49Computer Sciences Department

50.  MERGE(A, p, q, r), where A is an array and p, q, and r are indices numbering elements of the array such that p ≤ q < r. Merge sort 50Computer Sciences Department

51. 51Computer Sciences Department

52. Merging two sorted arrays 20 13 7 2 12 11 9 1 52Computer Sciences Department

53. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 53Computer Sciences Department

54. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 54Computer Sciences Department

55. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 55Computer Sciences Department

56. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 56Computer Sciences Department

57. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 7 57Computer Sciences Department

58. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 7 20 13 12 11 9 58Computer Sciences Department

59. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 7 20 13 12 11 9 9 59Computer Sciences Department

60. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 7 20 13 12 11 9 9 20 13 12 11 60Computer Sciences Department

61. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 7 20 13 12 11 9 9 20 13 12 11 61Computer Sciences Department

62. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 7 20 13 12 11 9 9 20 13 12 11 20 13 12 62Computer Sciences Department

63. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 7 20 13 12 11 9 9 20 13 12 11 20 13 12 63Computer Sciences Department

64. Merging two sorted arrays 20 13 7 2 12 11 9 1 1 20 13 7 2 12 11 9 2 20 13 7 12 11 9 7 20 13 12 11 9 9 20 13 12 11 20 13 12 Time =  (n) to merge a total of n elements (linear time). 64Computer Sciences Department

65. Merge sort - “Pseudo-code conventions” 65Computer Sciences Department

66. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. 66Computer Sciences Department

67. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. 67Computer Sciences Department

68. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. T(n)T(n) 68Computer Sciences Department

69. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. T(n/2) cn 69Computer Sciences Department

70. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn T(n/4) cn/2 70Computer Sciences Department

71. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … 71Computer Sciences Department

72. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n 72Computer Sciences Department

73. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn 73Computer Sciences Department

74. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn 74Computer Sciences Department

75. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn … 75Computer Sciences Department

76. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn #leaves = n (n)(n) … 76Computer Sciences Department

77. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn #leaves = n (n)(n) Total  (n lg n) … 77Computer Sciences Department

78. Conclusions  (n lg n) grows more slowly than  (n 2 ). Therefore, merge sort asymptotically beats insertion sort in the worst case. In practice, merge sort beats insertion sort for n > 30 or so. 78Computer Sciences Department