1. Design & Analysis of Algorithms

2. Analysis of Algorithms Comparing the programs (instead of algorithms) has difficulties. What data should the program use? Any analysis must be independent of specific data. Execution time is sensitive to the amount of data manipulated, grows as the amount of data increases. What computer should we use? We should compare the efficiency of the algorithms independently of a particular computer. Because of the execution speed of the processors, the execution times for an algorithm on the same data set on two different computers may differ. How are the algorithms coded? Comparing running times means comparing the implementations. We should not compare implementations, because they are sensitive to programming style.  absolute measure for an algorithm is not appropriate Design & Analysis of Algorithms

3. Analysis of Algorithms When we analyze algorithms, we should employ mathematical techniques that analyze algorithms independently of specific implementations, computers, or data. To analyze algorithms: First, we start to count the number of significant operations in a particular solution to assess its efficiency. Then, we will express the efficiency of algorithms using growth functions. Design & Analysis of Algorithms

4. Why Analysis of Algorithms? For real-time problems, we would like to prove that an algorithm terminates in a given time. Algorithmics may indicate which is the best and fastest solution to a problem and test different solutions Many problems are in a complexity class for which no practical algorithms are known better to know this before wasting a lot of time trying to develop a ”perfect” solution: verification

5. Analysis of Algorithms Analysis in the context of algorithms is concerned with predicting the resources that are required: Computational time Memory/Space However, Time – generally measured in terms of the number of steps required to execute an algorithm - is the resource of most interest By analyzing several candidate algorithms, the most efficient one(s) can be identified

6. But Computers are So Fast These Days??? Do we need to bother with algorithmics and complexity any more? computers are fast, compared to even 10 years ago... Many problems are so computationally demanding that no growth in the power of computing will help very much. Speed and efficiency are still important

7. Algorithm Efficiency There may be several possible algorithms to solve a particular problem each algorithm has a given efficiency There are many possible criteria for efficiency Running time Space There are always tradeoffs between these two efficiencies

8. What is Important? An array-based list retrieve operation is O(1), a linkedlist- based list retrieve operation is O(n). But insert and delete operations are much easier on a linked- list-based list implementation. When selecting the implementation of an Abstract Data Type (ADT), we have to consider how frequently particular ADT operations occur in a given application. If the problem size is always very small, we can probably ignore the algorithm’s efficiency. We have to weigh the trade-offs between an algorithm’s time requirement and its memory requirements. Design & Analysis of Algorithms

9. Analysis of Algorithms A running time function, T(n), yields the time required to execute the algorithm of a problem of size ‘n.’ Often the analysis of an algorithm leads to T(n), which may contain unknown constants, which depend on the characteristics of the ideal machine. So, we cannot determine this function exactly. T(n) = an 2 + bn + c, where a, b, and c are unspecıfıed constants Design & Analysis of Algorithms

10. Empirical vs Theoretical Analysis There are two essential approaches to measuring algorithm efficiency: Empirical analysis Program the algorithm and measure its running time on example instances Theoretical analysis derive a function which relates the running time to the size of instance In this cousre our focus wiil be on Threoretical analysis.

11. Advantages of Theory Theoretical analysis offers some advantages language independent not dependent on skill of programmer

12. Importance of Analyze Algorithm Need to recognize limitations of various algorithms for solving a problem Need to understand relationship between problem size and running time When is a running program not good enough? Need to learn how to analyze an algorithm's running time without coding it Need to learn techniques for writing more efficient code Need to recognize bottlenecks in code as well as which parts of code are easiest to optimize

13. Why do we analyze about them? understand their behavior, and (Job -- Selection, performance, modify) improve them. (Research)

14. What do we analyze about them?  Correctness Does the input/output relation match algorithm requirement?  Amount of work done Basic operations to do task  Amount of space used Memory used  Simplicity, clarity Verification and implementation.  Optimality Is it impossible to do better? Design & Analysis of Algorithms

15. Algorithm Analysis Many criteria affect the running time of an algorithm, including speed of CPU, bus and peripheral hardware language used and coding efficiency of the programmer quality of input (good, bad or average) But Programs derived from two algorithms for solving the same problem should both be Machine independent Language independent Amenable to mathematical study Realistic

16. Analysis of Algorithms (assumptions) Analysis is performed with respect to a computational model To analyze the algorithm, a computational model should also be determined. RAM (Random Access Machine) is used for the purpose.

17. Computation Model for Analysis RAM A RAM is an idealized uni-processor machine with infinite memory. Instruction are executed one by one All memory equally expensive to access No concurrent operations All reasonable instructions (basic operations) take unit time

18. Example of basic operations include –Assigning a value to a variable –Arithmetic operation (+, -, ×, /) on integers –Performing any comparison e.g. a < b –Boolean operations –Accessing an element of an array. Computation Model for Analysis

19. As we evaluate a new algorithm by comparing its performance with that of previous approaches Comparisons are asymtotic analyses of classes of algorithms In theoretical analysis, computational complexity Estimated in asymptotic sense, i.e. estimating for large inputs Big O, Omega, Theta etc. notations are used to compute the complexity Asymptotic notations are used because different implementations of algorithm may differ in efficiency Model of Computation (Assumptions )

20. Analysis Worst case Provides an upper bound on running time An absolute guarantee Average case Provides the expected running time Very useful, but treat with care: what is “average”? Random (equally likely) inputs Real-life inputs

21. Complexity The complexity of an algorithm is simply the amount of work the algorithm performs to complete its task.

22. Complexity Complexity is a function T(n) which measures the time or space used by an algorithm with respect to the input size n. The running time of an algorithm on a particular input is determined by the number of “Elementary Operations” executed.

23. What is an Elementary Operation? An elementary operation is an operation which takes constant time regardless of problem size. The point is that we can now measure running time in terms of number of elementary operations.

24. Elementary Operations Some operations are always elementary variable assignment testing if a number is negative Many can be assumed to be elementary.... addition, subtraction

25. Things to Remember in Analysis Constants or low-order terms are ignore e.g. if f(n) = 2n 2 then f(n) = O(n 2 ) Parameter N, usually referring to number of data items processed, affects running time most significantly. Worst case is amount of time program would take with worst possible input configuration worst case is upper bound for input and easier to find. Average case is amount of time a program is expected to take using "typical" input data definition of "average" can affect results average case is much harder to compute Best case is amount of time program would take with best possible input configuration best case is lower bound for input.

26. General Rules for Analysis Strategy for analysis Analyze from inside out Analyze function calls first If recursion behaves like a for-loop, analysis is trivial; otherwise, use recurrence relation to solve.

27. Methods of Proof Proof by Contradiction Assume a theorem is false; show that this assumption implies a property known to be true is false -- therefore original hypothesis must be true Proof by Counterexample Use a concrete example to show an inequality cannot hold Mathematical Induction Prove a trivial base case, assume true for k, then show hypothesis is true for k+1 Used to prove recursive algorithms

28. Complexity and Input Size Complexity is a function that express relationship between time and input size We are not so much interested in the time and space complexity for small inputs rather the function is only calculated for large inputs.

29. Complexity-Growth of funtion The growth of the complexity functions is what is more important for the analysis. The growth of time and space complexity with increasing input size n is a suitable measure for the comparison of algorithms.

30. Function of Growth rate

31. Average,Worst and Best Case An algorithm may perform very differently on different example instances. For example: a bubble sort algorithm might be presented with data: already in order in random order in the exact reverse order of what is required Average case analysis measures performance over the entire set of possible instances Worst case picks out the worst possible instance We often concentrate on this for theoretical analysis as it provides a clear upper bound of resources Best case picks out the best possible instance The best case occur rarely, it provides a clear lower bound of resources.

32. Average Case Analysis Average case analysis can be difficult in practice to do a realistic analysis we need to know the likely distribution of instances However, it is often very useful and more relevant than worst case for example quicksort has a catastrophic (extremly harmful) worst case, but in practice it is one of the best sorting algorithms known

33. Example-linear search algorithm  Suppose a linear array DATA contains n elements, and suppose a specific ITEM of information is given. We want either to find the location LOC of ITEM in the array DATA, or to send some message, such as LOC = 0, to indicate that ITEM does not appear in DATA.  The linear search algorithm solves this problem by comparing ITEM, one by one, with each element in DATA. That is, we compare ITEM with DATA[1], then DATA[2], and so on, until we find LOC such that ITEM = DATA[LOC].

34. Worst case Worst Case: Clearly the worst case occurs when ITEM is the last element in the array DATA or is not there at all. Accordingly, T(n) = n is the worst-case complexity of the linear search algorithm.

35. Average case  Average Case: Here we assume that ITEM does appear in DATA, and that is equally likely to occur at any position in the array. Accordingly, the number of comparisons can be any of the numbers 1,2,3,..., n, and each number occurs with probability p=1/n.  Then T(n) = n/2  This agrees with our intuitive feeling that the average number of comparisons needed to find the location of ITEM is approximately equal to half the number of elements in the DATA list.

36. Components of an Algorithm Sequences Selections Repetitions

37. Sequence A linear sequence of elementary operations is also performed in constant time. More generally, given two program fragments P 1 and P 2 which run sequentially in times t 1 and t 2 we can use the maximum rule which states that the the larger time dominates Complexity will be max(t1,t2)

38. General Rules for Analysis 1. Consecutive statements Maximum statement is the one counted Block #1 Block #2 t1t1 t2t2 t 1 +t 2 = max(t 1,t 2 )

39. Selection If then P 1 else P 2 structures are a little harder; conditional loops. The maximum rule can be applied here too: max(t 1, t 2 ), assuming t 1, t 2 are times of P 1, P 2 However, the maximum rule may prove too conservative if is always true the time is t 1 if is always false the time is t 2

40. General Rules for Analysis 2. If/Else if cond then S1 else S2 Block #1Block #2t1t1 t2t2 Max(t 1,t 2 )

41. General Rules for Analysis 3. For Loops Running time of a for-loop is at most the running time of the statements inside the for-loop times number of iterations for (i = sum = 0; i < n; i++) sum += a[i]; for loop iterates n times, executes 2 assignment statements each iteration ==> asymptotic complexity of O(n)

42. General Rules for Analysis 4. Nested For-Loops Analyze inside-out. Total running time is running time of the statement multiplied by product of the sizes of all the for- loops e.g. for (i =0; i < n; i++) for (j = 0, sum = a[0]; j <= i ; j++) sum += a[j]; printf("sum for subarray - through %d is %d\n", i, sum);

43. Repetition (Loops) For loop: for i = 1 to m do P(i) Assume that P(i) takes time t, where t is independent of i A reasonable estimate of the time for the whole loop structure, l is: l = mt This is reasonable, provided m is positive If m is zero or negative the relationship l = mt is not valid

44. General Rules for Estimation Loops: The running time of a loop is at most the running time of the statements inside of that loop times the number of iterations. Nested Loops: Running time of a nested loop containing a statement in the inner most loop is the running time of statement multiplied by the product of the size of all loops. Consecutive Statements: Just add the running times of those consecutive statements. If/Else: Never more than the running time of the test plus the larger of running times of c1 and c2. Design & Analysis of Algorithms

45. Example Algorithm: 1. n = read input from user 2. sum = 0 3. i = 0 4. while i < n 5. number = read input from user 6. sum = sum + number 7. i = i + 1 8. mean = sum / n Statement Number of times executed1 21 31 4n+1 5n 6n 7n 81 The computing time for this algorithm in terms on input size n is: T(n) = 4n + 5.

46. The Execution Time of Algorithms (cont.) Example: Simple If-Statement CostTimes if (n < 0) c1 1 absval = -n c2 1 else absval = n; c3 1 Total Cost <= c1 + max(c2,c3) Design & Analysis of Algorithms

47. The Execution Time of Algorithms Example: Simple Loop CostTimes i = 1; c1 1 sum = 0; c2 1 while (i <= n) { c3 n+1 i = i + 1; c4 n sum = sum + i; c5 n } Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*c5  The time required for this algorithm is proportional to n Design & Analysis of Algorithms

48. The Execution Time of Algorithms (cont.) Example: Nested Loop CostTimes i=1; c1 1 sum = 0; c2 1 while (i <= n) { c3 n+1 j=1; c4 n while (j <= n) { c5 n*(n+1) sum = sum + i; c6 n*n j = j + 1; c7 n*n } i = i +1; c8 n } Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8  The time required for this algorithm is proportional to n 2 Design & Analysis of Algorithms

49. Asymptotic Analysis EXACT answers are great when we find them; there is something very satisfying about complete knowledge. But there is also a time when approximations are in order. If we run into sum or a recurrence whose solution is not in closed form, we still would like to know something about the answer. Design & Analysis of Algorithms

50. Analysis of Algorithms We measure the complexity of an algorithm by identifying a basic operation and then counting how any times the algorithm performs that basic operation for an input size n. probleminput of size n basic operation searching a listlists with n elementscomparison sorting a listlists with n elementscomparison multiplying two matricestwo n-by-n matricesmultiplication traversing a treetree with n nodesaccessing a node Towers of Hanoin disksmoving a disk Design & Analysis of Algorithms

51. Algorithm Growth Rates We measure an algorithm’s time requirement as a function of the problem size. Problem size depends on the application: e.g., number of elements in a list for a sorting algorithm, the number disks in Towers of Hanoi problem. So, for instance, we say that (if the problem size is n) Algorithm A requires 5*n 2 time units to solve a problem of size n. Algorithm B requires 7*n time units to solve a problem of size n. The most important thing to learn is how quickly the algorithm’s time requirement grows as a function of the problem size. Algorithm A requires time proportional to n 2. Algorithm B requires time proportional to n. Design & Analysis of Algorithms An algorithm’s proportional time requirement is known as growth rate.

52. Analysis of Algorithms The growth rate of T(n) is referred to the computational complexity of the algorithm. The computational complexity gives a concise way of saying how the running time, T(n), varies with n and is independent of any particular implementation. We can compare the efficiency of two algorithms by comparing their growth rates. The lower the growth rate, the faster the algorithm, at least for large values of n. For example; T(n) = an 2 + bn + c, the growth rate is O(n 2 ) The goal of the algorithm designer should be an algorithm with as low a growth rate of the running time function, T(n), of that algorithm as possible. Design & Analysis of Algorithms

53. Algorithm Growth Rates (cont.) Time requirements as a function of the problem size n Design & Analysis of Algorithms

54. Common Growth Rates FunctionGrowth Rate Name cConstant log NLogarithmic log 2 NLog-squared NLinear N log N N2N2 Quadratic N3N3 Cubic 2N2N Exponential

55. Growth-Rate Functions Design & Analysis of Algorithms O(1) Time requirement is constant, and it is independent of the problem’s size. O(log 2 n) Time requirement for a logarithmic algorithm increases slowly as the problem size increases. O(n) Time requirement for a linear algorithm increases directly with the size of the problem. O(n*log 2 n) Time requirement for a n*log 2 n algorithm increases more rapidly than a linear algorithm. O(n 2 ) Time requirement for a quadratic algorithm increases rapidly with the size of the problem. O(n 3 ) Time requirement for a cubic algorithm increases more rapidly with the size of the problem than the time requirement for a quadratic algorithm. O(2 n ) As the size of the problem increases, the time requirement for an exponential algorithm increases too rapidly to be practical.

56. A Comparison of Growth-Rate Functions (cont.) Design & Analysis of Algorithms

57. Mathematical Functions and Notations lg is logarithm to base 2 (i.e. log 2 ) floor  x  and ceiling  x    3.4  = 3 and  3.4  = 4 open interval (a, b) is {x   | a < x < b} closed interval [a, b] is {x   | a  x  b}

58. Mathematics Review Permutation Set of n elements is an arrangement of the elements in given order e.g. Permutation for elements are a, b, c abc, acb, bac, bca, cab, cba -n! permutation exist for a set of elements 5! = 120 permutation for 5 elements

59. Mathematics Review Properties of logarithms: log b (xy) = log b x + log b y log b (x/y) = log b x - log b y log b x a = alog b x log b a = log x a/log x b Properties of exponentials: a (b+c) = a b a c a bc = (a b ) c a b /a c = a (b-c) b = a log a b b c = a c*log a b

60. Mathematics Review