1. Analyzing algorithms & Asymptotic Notation BIO/CS 471 – Algorithms for Bioinformatics

2. Analyzing Algorithms2 Algorithms and Problems Algorithm: a method or a process followed to solve a problem. A recipe. A problem is a mapping of input to output. An algorithm takes the input to a problem (function) and transforms it to the output. A problem can have many algorithms.

3. Analyzing Algorithms3 Algorithm Properties An algorithm possesses the following properties: It must be correct. It must be composed of a series of concrete steps. There can be no ambiguity as to which step will be performed next. It must be composed of a finite number of steps. It must terminate. A computer program is an instance, or concrete representation, for an algorithm in some programming language.

4. Analyzing Algorithms4 The RAM model of computing  Linear, random access memory  READ/WRITE = one operation  Simple mathematical operations are also unit operations  Can only read one location at a time, by address  Registers 0000 0001 0002 0003 0004 0005 0006 0007 0008 0009 0010 …

5. Analyzing Algorithms5 How fast is an algorithm?  To compare two sorting algorithms, should we talk about how fast the algorithms can sort 10 numbers, 100 numbers or 1000 numbers?  We need a way to talk about how fast the algorithm grows or scales with the input size. Input size is usually called n An algorithm can take 100n steps, or 2n 2 steps, which one is better?

6. Analyzing Algorithms6 Introduction to Asymptotic Notation  We want to express the concept of “about”, but in a mathematically rigorous way  Limits are useful in proofs and performance analyses  notation:  (n 2 ) = “this function grows similarly to n 2 ”.  Big-O notation: O (n 2 ) = “this function grows at least as slowly as n 2 ”. Describes an upper bound.

7. Analyzing Algorithms7 Big-O  What does it mean? If f(n) = O(n 2 ), then:  f(n) can be larger than n 2 sometimes, but…  I can choose some constant c and some value n 0 such that for every value of n larger than n 0 : f(n) < cn 2  That is, for values larger than n 0, f(n) is never more than a constant multiplier greater than n 2  Or, in other words, f(n) does not grow more than a constant factor faster than n 2.

8. Analyzing Algorithms8 Visualization of O(g(n)) n0n0 cg(n) f(n)f(n)

9. Analyzing Algorithms9 Big-O

10. 10 More Big-O  Prove that:  Let c = 21 and n 0 = 4  21n 2 > 20n 2 + 2n + 5 for all n > 4 n 2 > 2n + 5 for all n > 4 TRUE

11. Analyzing Algorithms11 Tight bounds  We generally want the tightest bound we can find.  While it is true that n 2 + 7n is in O(n 3 ), it is more interesting to say that it is in O(n 2 )

12. Analyzing Algorithms12 Big Omega – Notation  () – A lower bound n 2 =  (n) Let c = 1, n 0 = 2 For all n  2, n 2 > 1  n

13. Analyzing Algorithms13 Visualization of  (g(n)) n0n0 cg(n) f(n)f(n)

14. Analyzing Algorithms14  -notation  Big-O is not a tight upper bound. In other words n = O(n 2 )  provides a tight bound  In other words,

15. Analyzing Algorithms15 Visualization of  (g(n)) n0n0 c2g(n)c2g(n) f(n)f(n) c1g(n)c1g(n)

16. Analyzing Algorithms16 A Few More Examples  n = O(n 2 ) ≠  (n 2 )  200n 2 = O(n 2 ) =  (n 2 )  n 2.5 ≠ O(n 2 ) ≠  (n 2 )

17. Analyzing Algorithms17 Some Other Asymptotic Functions  Little o – A non-tight asymptotic upper bound n = o(n 2 ), n = O(n 2 ) 3n 2 ≠ o(n 2 ), 3n 2 = O(n 2 )  () – A lower bound Similar definition to Big-O n 2 =  (n)  () – A non-tight asymptotic lower bound  f(n) =  (n)  f(n) = O(n) and f(n) =  (n)

18. Analyzing Algorithms18 Visualization of Asymptotic Growth n0n0 O(f(n)) f(n)f(n)  (f(n))  (f(n)) o(f(n))  (f(n))

19. Analyzing Algorithms19 Analogy to Arithmetic Operators

20. Analyzing Algorithms20 Example 2  Prove that:  Let c = 21 and n 0 = 10  21n 3 > 20n 3 + 7n + 1000 for all n > 10 n 3 > 7n + 5 for all n > 10 TRUE, but we also need…  Let c = 20 and n 0 = 1  20n 3 < 20n 3 + 7n + 1000 for all n  1 TRUE

21. Analyzing Algorithms21 Example 3  Show that  Let c = 2 and n 0 = 5

22. Analyzing Algorithms22 Looking at Algorithms  Asymptotic notation gives us a language to talk about the run time of algorithms.  Not for just one case, but how an algorithm performs as the size of the input, n, grows.  Tools: Series sums Recurrence relations

23. Analyzing Algorithms23 Running Time Examples (1) Example 1: a = b; This assignment takes constant time, so it is  (1). Example 2: sum = 0; for (i=1; i<=n; i++) sum += n;

24. Analyzing Algorithms24 Running Time Examples (2) Example 2: sum = 0; for (j=1; j<=n; j++) for (i=1; i<=j; i++) sum++; for (k=0; k<n; k++) A[k] = k;

25. Analyzing Algorithms25 Series Sums  The arithmetic series: 1 + 2 + 3 + … + n =  Linearity:

26. Analyzing Algorithms26 Series Sums  0 + 1 + 2 + … + n – 1 =  Example:

27. Analyzing Algorithms27 More Series  Geometric Series: 1 + x + x 2 + x 3 + … + x n  Example:

28. Analyzing Algorithms28 Telescoping Series  Consider the series:  Look at the terms:

29. Analyzing Algorithms29 Telescoping Series  In general:

30. Analyzing Algorithms30 The Harmonic Series

31. Analyzing Algorithms31 Running Time Examples (3) Example 3: sum1 = 0; for (i=1; i<=n; i++) for (j=1; j<=n; j++) sum1++; sum2 = 0; for (i=1; i<=n; i++) for (j=1; j<=i; j++) sum2++;

32. Analyzing Algorithms32 Best, Worst, Average Cases Not all inputs of a given size take the same time to run. Sequential search for K in an array of n integers: Begin at first element in array and look at each element in turn until K is found Best case: Worst case: Average case:

33. Analyzing Algorithms33 Space Bounds Space bounds can also be analyzed with asymptotic complexity analysis. Time: Algorithm Space Data Structure

34. Analyzing Algorithms34 Space/Time Tradeoff Principle One can often reduce time if one is willing to sacrifice space, or vice versa. Encoding or packing information Boolean flags Table lookup Factorials Disk-based Space/Time Tradeoff Principle: The smaller you make the disk storage requirements, the faster your program will run.

35. Analyzing Algorithms35 Faster Computer or Faster Algorithm?  Suppose, for your algorithm, f(n) = 2n 2  In T seconds, you can process k inputs  If you get a computer 64 times faster, how many inputs can you process in T seconds?

36. Analyzing Algorithms36 Faster Computer or Algorithm? If we have a computer that does 10,000 operations per second, what happens when we buy a computer 10 times faster? T(n)T(n)nn’n’Changen’/n 10n1,00010,000n’ = 10n10 20n5005,000n’ = 10n10 5n log n2501,842  10 n < n’ < 10n 7.37 2n22n2 70223 n’ =  10n 3.16 2n2n 1316n’ = n + 3-----

37. Analyzing Algorithms37 Traveling Salesman Problem  n cities Traveling distance between each pair is given Find the circuit that includes all cities A C D G B E F 8 12 20 25 35 33 10 22 21 15 25 23 22 14 19

38. Analyzing Algorithms38 Is there a “real difference”?  10^1  10^2  10^3Number of students in the college of engineering  10^4 Number of students enrolled at Wright State University  10^6 Number of people in Dayton  10^8 Number of people in Ohio  10^10 Number of stars in the galaxy  10^20 Total number of all stars in the universe  10^80 Total number of particles in the universe  10^100 << Number of possible solutions to traveling salesman (100)  Traveling salesman (100) is computable but it is NOT tractable.

39. Analyzing Algorithms39 Growth of Functions

40. Analyzing Algorithms40 Is there a “real” difference?  Growth of functions

41. Analyzing Algorithms41 Approaches to Solving Problems  Direct/iterative SelectionSort Can by analyzed using series sums  Divide and Conquer Recursion and Dynamic Programming Cut the problem in half MergeSort

42. Analyzing Algorithms42 Recursion  Computing factorials sub fact($n) { if ($n <= 1) { return(1); } else { $temp = $fact($n-1); $result = $temp + 1; return($result); } print(fact(4). “\n”); fib(5)

43. Analyzing Algorithms43 Fibonacci Numbers int fib(int N) { int prev, pprev; if (N == 1) { return 0; } else if (N == 2) { return 1; } else { prev = fib(N-1); pprev = fib(N-2); return prev + pprev; }

44. Analyzing Algorithms44 MergeSort  Let M n be the time to MergeSort n items M n = 2(M n-1 ) + n 72946946