1. Analysis and Design of Algorithms

2. According to math historians the true origin of the word algorism: comes from a famous Persian author named ál-Khâwrázmî.

3. Khâwrázmî (780-850 A.D.) Statue of Khâwrázmî in front of the Faculty of Mathematics, Amirkabir University of Technology, Tehran, Iran. A stamp issued September 6, 1983 in the Soviet Union, commemorating Khâwrázmî's 1200th birthday. A page from his book. Courtesy of Wikipedia

4. Computational Landscape

5. Algorithm An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

6. Analysis of Algorithms How good is the algorithm? – Correctness – Time efficiency – Space efficiency Does there exist a better algorithm? – Lower bounds – Optimality

7. Example Let’s assume: Computer speed = 10 6 IPS, Input: a data base of size n = 10 6 Time complexity shows dependence of algorithm’s running time on input size.

8. Machine Model  Algorithm Analysis:  should reveal intrinsic properties of the algorithm itself.  should not depend on any computing platform, programming language, compiler, computer speed, etc.  Elementary steps:  arithmetic: + –    logic: and or not  comparison:        assigning a value to a scalar variable:   ….

9. Space complexity Time complexity For iterative algorithms: sums For recursive algorithms: recurrence relations Complexity

10. Time Complexity  Time complexity shows dependence of algorithm’s running time on input size.  Worst-case  Average or expected-case  What is it good for? Tells us how efficient our design is before its costly implementation. Reveals inefficiency bottlenecks in the algorithm. Can use it to compare efficiency of different algorithms that solve the same problem. Is a tool to figure out the true complexity of the problem itself! How fast is the “fastest” algorithm for the problem? Helps us classify problems by their time complexity.

11. T(n) =  ( f(n) ) T(n) = 23 n 3 + 5 n 2 log n + 7 n log 2 n + 4 log n + 6. drop lower order terms drop constant multiplicative factor T(n) =  (n 3 ) Why do we want to do this? 1.Asymptotically (at very large values of n) the leading term largely determines function behaviour. 2.With a new computer technology (say, 10 times faster) the leading coefficient will change (be divided by 10). So, that coefficient is technology dependent any way! 3.This simplification is still capable of distinguishing between important but distinct complexity classes, e.g., linear vs. quadratic, or polynomial vs exponential.

12. Asymptotic Notations:   Theta f(n) =  (g(n)) f(n) ≈c g(n) Big Oh f(n) = O(g(n))f(n) ≤c g(n) Big Omega f(n) = Ω(g(n))f(n) ≥ c g(n) Little Oh f(n) = o(g(n)) Little Omega f(n) = ω(g(n)) Rough, intuitive meaning worth remembering:

13. lim n→∞ f(n)/g(n) 0 order of growth of f(n) < order of growth of g(n) f(n)  o(g(n)), f(n)  O(g(n)) c>0 order of growth of f(n) = order of growth of g(n) f(n)   (g(n)), f(n)  O(g(n)), f(n)   (g(n)) ∞ order of growth of f(n) > order of growth of g(n) f(n)   (g(n)), f(n)   (g(n)) =

14. Asymptotics by ratio limit Theta f(n) =  (g(n))0 < L <  Big Oh f(n) = O(g(n)) 0 ≤ L <  Big Omega f(n) = Ω(g(n))0 < L Little Oh f(n) = o(g(n))L = 0 Little Omega f(n) = ω(g(n)) L =  L = lim n  f(n)/g(n). If L exists, then:

15. Examples: log b n vs. log c n log b n = log b c log c n lim n→∞ ( log b n / log c n) = lim n→∞ (log b c) = log b c log b n  (log c n)

16. L’Hôpital’s rule If b b lim n→∞ t(n) = lim n→∞ g(n) = ∞ b b The derivatives f´, g´ exist, Then t(n) g(n) lim n→∞ = t ´(n) g ´(n) lim n→∞ Example: log 2 n vs. n

17. Theta : Asymptotic Tight Bound f(n) =  (g(n)) g(n) c 1 g(n) c 2 g(n) f(n) n0n0 n  c 1, c 2, n 0 >0 :  n  n 0, c 1 g(n)  f(n)  c 2 g(n). ++

18. Big Oh: Asymptotic Upper Bound f(n) =  (g(n)) c g(n) f(n) n0n0 n  c, n 0 >0 :  n  n 0, f(n)  c g(n). g(n) ++

19. Big Omega : Asymptotic Lower Bound f(n) =  (g(n)) g(n) c g(n) f(n) n0n0 n  c, n 0 >0 :  n  n 0, cg(n)  f(n).  +

20. Little oh : Non-tight Asymptotic Upper Bound f(n) =  (g(n)) c g(n) f(n) n0n0 n  c >0,  n 0 >0 :  n  n 0, f(n) < c g(n). No matter how small  +

21. Little omega : Non-tight Asymptotic Lower Bound f(n) =  (g(n)) c g(n) f(n) n0n0 n  c >0,  n 0 >0 :  n  n 0, f(n) > c g(n). No matter how large  +

22. Definitions of Asymptotic Notations f(n) =  (g(n))  c 1,c 2 >0,  n 0 >0:  n  n 0, c 1 g(n)  f(n)  c 2 g(n) f(n) =  (g(n))  c>0,  n 0 >0:  n  n 0, f(n)  c g(n) f(n) =  (g(n))  c>0,  n 0 >0:  n  n 0, c g(n)  f(n) f(n) =  (g(n))  c >0,  n 0 >0:  n  n 0, f(n) < c g(n) f(n) =  (g(n))  c >0,  n 0 >0:  n  n 0, c g(n) < f(n)

25. Ordering Functions Functions Constant Logarithmic Poly Logarithmic Polynomial Exponential (log n) 5 n5n5 2 5n 5 log n5 << Factorial n!

26. Classifying Functions Polynomial LinearQuadratic Cubic ? 5n 2 5n 5n 3 5n 4

27. Example Problem: Sorting Some sorting algorithms and their worst-case time complexities: Quick-Sort:  (n 2 ) Insertion-Sort:  (n 2 ) Selection-Sort:  (n 2 ) Merge-Sort:  (n log n) Heap-Sort:  (n log n) there are infinitely many sorting algorithms! So, Merge-Sort and Heap-Sort are worst-case optimal, and SORTING complexity is Q(n log n).

28. Theoretical analysis of time efficiency Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size b b Basic operation: the operation that contributes most towards the running time of the algorithm. T(n) ≈ c op C(n) running time execution time for basic operation Number of times basic operation is executed input size

29. ProblemInput size measureBasic operation Search for key in list of n items Number of items in list nKey comparison Multiply two matrices of floating point numbers Dimensions of matrices Floating point multiplication Compute a n n Floating point multiplication Graph problem#vertices and/or edges Visiting a vertex or traversing an edge Input size and basic operation examples

30. Theoretical analysis of time efficiency Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size

31. Best-case, average-case, worst-case Worst case: W(n) – maximum over inputs of size n Best case: B(n) – minimum over inputs of size n Average case: A(n) – “average” over inputs of size n – NOT the average of worst and best case – Under some assumption about the probability distribution of all possible inputs of size n, calculate the weighted sum of expected C(n) (numbers of basic operation repetitions) over all possible inputs of size n.

32. 32 Time efficiency of nonrecursive algorithms b b Steps in mathematical analysis of nonrecursive algorithms: Decide on parameter n indicating input’s size Identify algorithm’s basic operation Determine worst, average, & best case for inputs of size n Set up summation for C(n) reflecting algorithm’s loop structure Simplify summation using standard formulas

33. Series b b Proof by Gauss when 9 years old (!):

34. General rules for sums

35. Some Mathematical Facts Some mathematical equalities are:

36. The Execution Time of Algorithms Each operation in an algorithm (or a program) has a cost.  Each operation takes a certain of time. count = count + 1;  take a certain amount of time, but it is constant A sequence of operations: count = count + 1; Cost: c 1 sum = sum + count; Cost: c 2  Total Cost = c 1 + c 2

37. The Execution Time of Algorithms (cont.) Example: Simple If-Statement CostTimes if (n < 0) c1 1 absval = -n c2 1 else absval = n; c3 1 Total Cost <= c1 + max(c2,c3)

38. The Execution Time of Algorithms (cont.) Example: Simple Loop CostTimes i = 1; c1 1 sum = 0; c2 1 while (i <= n) { c3 n+1 i = i + 1; c4 n sum = sum + i; c5 n } Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*c5  The time required for this algorithm is proportional to n

39. The Execution Time of Algorithms (cont.) Example: Nested Loop CostTimes i=1; c1 1 sum = 0; c2 1 while (i <= n) { c3 n+1 j=1; c4 n while (j <= n) { c5 n*(n+1) sum = sum + i; c6 n*n j = j + 1; c7 n*n } i = i +1; c8 n } Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8  The time required for this algorithm is proportional to n 2

40. Growth-Rate Functions – Example1 CostTimes i = 1; c1 1 sum = 0; c2 1 while (i <= n) { c3 n+1 i = i + 1; c4 n sum = sum + i; c5 n } T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*c5 = (c3+c4+c5)*n + (c1+c2+c3) = a*n + b  So, the growth-rate function for this algorithm is O(n)

41. Growth-Rate Functions – Example2 CostTimes i=1; c1 1 sum = 0; c2 1 while (i <= n) { c3 n+1 j=1; c4 n while (j <= n) { c5 n*(n+1) sum = sum + i; c6 n*n j = j + 1; c7 n*n } i = i +1; c8 n } T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8 = (c5+c6+c7)*n 2 + (c3+c4+c5+c8)*n + (c1+c2+c3) = a*n 2 + b*n + c  So, the growth-rate function for this algorithm is O(n 2 )

42. Growth-Rate Functions – Example3 CostTimes for (i=1; i<=n; i++) c1 n+1 for (j=1; j<=i; j++) c2 for (k=1; k<=j; k++) c3 x=x+1; c4 T(n) = c1*(n+1) + c2*( ) + c3* ( ) + c4*( ) = a*n 3 + b*n 2 + c*n + d  So, the growth-rate function for this algorithm is O(n 3 )

43. Sequential Search int sequentialSearch(const int a[], int item, int n){ for (int i = 0; i < n && a[i]!= item; i++); if (i == n) return –1; return i; } Unsuccessful Search:  O(n) Successful Search: Best-Case: item is in the first location of the array  O(1) Worst-Case: item is in the last location of the array  O(n) Average-Case: The number of key comparisons 1, 2,..., n  O(n)

44. Insertion Sort an incremental algorithm 2 2 4 4 5 5 7 7 11 15 2 2 4 4 5 5 7 7 11 15 2 2 11 5 5 7 7 4 4 15 2 2 5 5 11 7 7 4 4 15 2 2 5 5 11 7 7 4 4 15 2 2 5 5 11 7 7 4 4 15

45. Insertion Sort: Time Complexity Algorithm InsertionSort(A[1..n]) for i  2.. n do LI: A[1..i –1] is sorted, A[i..n] is untouched. § insert A[i] into sorted prefix A[1..i–1] by right-cyclic-shift: 2. key  A[i] 3. j  i –1 4. while j > 0 and A[j] > key do 5. A[j+1]  A[j] 6. j  j –1 7. end-while 8. A[j+1]  key 9. end-for end

46. Master theorem 1.a < b d T(n) ∈ Θ(n d ) 2.a = b d T(n) ∈ Θ(n d lg n ) 3.a > b d T(n) ∈ Θ(n log b a )

47. The divide-and-conquer Design Paradigm Divide the problem into subproblems. Conquer the subproblems by solving them recursively. Combine subproblem solutions. Many algorithms use this paradigm.

48. Divide-and-conquer Technique subproblem 2 of size n/2 subproblem 1 of size n/2 a solution to subproblem 1 a solution to the original problem a solution to subproblem 2 a problem of size n

49. 49 Divide and Conquer Examples Sorting: mergesort and quicksort Matrix multiplication-Strassen’s algorithm Binary search Powering a Number Closest pair problem ….etc.

50. Binary search Find an element in a sorted array: – Divide: Check middle element. – Conquer: Recursively search 1 sub array. – Combine: Trivial. Example: Find 9 3 5 7 8 9 12 15

51. Binary search Find an element in a sorted array: – Divide: Check middle element. – Conquer: Recursively search 1 sub array. – Combine: Trivial. Example: Find 9

52. Binary search Find an element in a sorted array: – Divide: Check middle element. – Conquer: Recursively search 1 sub array. – Combine: Trivial. Example: Find 9

53. Binary search Find an element in a sorted array: – Divide: Check middle element. – Conquer: Recursively search 1 sub array. – Combine: Trivial. Example: Find 9

54. Binary search Find an element in a sorted array: – Divide: Check middle element. – Conquer: Recursively search 1 sub array. – Combine: Trivial. Example: Find 9

55. Binary search Find an element in a sorted array: – Divide: Check middle element. – Conquer: Recursively search 1 sub array. – Combine: Trivial. Example: Find 9

56. Binary search Find an element in a sorted array: – Divide: Check middle element. – Conquer: Recursively search 1 sub array. – Combine: Trivial. Example: Find 9

57. Binary Search int binarySearch(int a[], int size, int x) { int low =0; int high = size –1; int mid; // mid will be the index of // target when it’s found. while (low <= high) { mid = (low + high)/2; if (a[mid] < x) low = mid + 1; else if (a[mid] > x) high = mid – 1; else return mid; } return –1; }

58. Recurrence for binary search T(n) = 1T(n/2) + Θ(1) # subproblems subproblem size cost of dividing and combining

59. How much better is O(log 2 n)? n O( log 2 n ) 16 4 64 6 256 8 1024 (1KB) 10 16,384 14 131,072 17 262,144 18 524,288 19 1,048,576 (1MB) 20 1,073,741,824 (1GB) 30

60. Powering a Number Problem: Compute a n, where n ∈ N. Naive algorithm: – Multiply n copies of X: X ·X ·X ··· X ·X ·X. – Complexity ? Θ(n) The Spot Creativity: – Is this the only and the best algorithm? – Any Suggestions on using divide-and-conquer strategy?

61. Powering a Number Starting with X, repeatedly multiply the result by itself. We get: X, X 2, X 4, X 8, X 16, X 32, … Suppose we want to compute X 13. 13 = 8 + 4 + 1. So, X 13 = X 8 ·X 4 ·X.

62. Divide-and-conquer: Complexity: T(n) = T(n/2) + Θ(1) T(n) = Θ(lgn) Powering a Number

63. Matrix Multiplication of n*n

64. Code for Matrix Multiplication for i=1 to n for j=1 to n c ij =0 for k=1 to n c ij =c ij + a ik *b kj Running Time= ?

65. Matrix Multiplication by Divide-&-Conquer n/2 = A 11 A 12 A 21 A 22 B 11 B 12 B 21 B 22 C 11 C 12 C 21 C 22 65

66. Strassen’s Idea How Strassen came up with his magic idea? – We should try get rid of as more multiplications as possible. We could do a hundred additions instead. – We can reduce the numbers of subproblems in T(n)=8 * T(n/2) + Θ(n 2 ) – He must be very clever.

67. Strassen’s Idea Multiply 2*2 matrices with only 7 recursive multiplications. Notes: plus of matrices is commutative, but multiplication is not.

68. Strassen’s Algorithm 1.Divide: Partition A and B into (n/2)*(n/2) submatrices. And form terms to be multiplied using + and –. 2.Conquer: Perform 7 multiplications (P 1 to P 7 ) of (n/2)×(n/2) submatrices recursively. 3.Combine:Form C (r,s,t,u) using + and –on (n/2)×(n/2) submatrices. Write down cost of each step We got T(n)=7 * T(n/2) + Θ(n 2 )

69. Cost of Strassen’s Algorithm T(n)=7 * T(n/2) + Θ(n 2 ) a=7,b=2 =n lg7 case 1 T(n) = Θ(n lg7 ) =O(n 2.81 ) Not so surprising? Strassen’s Algorithm is not the best. The best one has O(n 2.376 ) (Be of theoretical interest only). But it is simple and efficient enough compared with the naïve one when n>=32.

70. Mergesort b Sort arrays B and C b Merge sorted arrays B and C into array A

71. Using Divide and Conquer: Mergesort Mergesort Strategy Sorted Merge Sorted Sort recursively by Mergesort first last  (first  last)  2 

72. Mergesort b Sort arrays B and C b Merge sorted arrays B and C into array A as follows: Repeat the following until no elements remain in one of the arrays: –compare the first elements in the remaining unprocessed portions of the arrays –copy the smaller of the two into A, while incrementing the index indicating the unprocessed portion of that array Once all elements in one of the arrays are processed, copy the remaining unprocessed elements from the other array into A.

73. Algorithm: Mergesort Input: Array E and indices first and last, such that the elements E[i] are defined for first  i  last. Output: E[first], …, E[last] is a sorted rearrangement of the same elements void mergeSort(Element[] E, int first, int last) if (first < last) int mid =  (first+last)/2  ; mergeSort(E, first, mid); mergeSort(E, mid+1, last); merge(E, first, mid, last); return;

74. Merge Sort How to express the cost of merge sort? T(n) =2T(n/2)   (n) for n>1, T(1)=0   (n lg n)

75. Merge Sort 1.Divide:Trivial. 2.Conquer:Recursively sort subarrays. 3.Combine:Linear-time merge. # subproblems subproblem size cost of dividing and combining

76. Efficiency of Mergesort Number of comparisons is close to theoretical minimum for comparison-based sorting: – lg n ! ≈ n lg n - 1.44 n b Space requirement: Θ( n ) (NOT in-place) b Can be implemented without recursion (bottom-up) b All cases have same efficiency: Θ( n log n)