1. Recursion

2. Recursion: Definition Function that solves a problem by relying on itself to compute the correct solution for a smaller version of the problem Function that solves a problem by relying on itself to compute the correct solution for a smaller version of the problem Requires terminating condition: Case for which recursion is no longer needed Requires terminating condition: Case for which recursion is no longer needed

3. Factorial Recursion Factorial:n! = n * (n-1)! Factorial:n! = n * (n-1)! Base Case => 0! = 1 Base Case => 0! = 1 Smaller problem => Solving (n-1)! Smaller problem => Solving (n-1)! Implementation: Implementation: long factorial(long inputValue) { if (inputValue == 0) return 1; if (inputValue == 0) return 1; else return inputValue * factorial(inputValue - 1); else return inputValue * factorial(inputValue - 1);}

4. Searching We want to find whether or not an input value is in a sorted list: We want to find whether or not an input value is in a sorted list: 8 in [1, 2, 8, 10, 15, 32, 63, 64]? 33 in [1, 2, 8, 10, 15, 32, 63, 64]? ------------------------------------------------------ int index = 0; while (index < listSize) { if (list[index] == input) return index; index++;} return –1;

5. Searching Better method? Better method? Use fact that we know the list is sorted Use fact that we know the list is sorted Cut what we have to search in half each time Cut what we have to search in half each time Compare input to middle Compare input to middle If input greater than middle, our value has be in the elements on the right side of the middle element If input greater than middle, our value has be in the elements on the right side of the middle element If input less than middle, our value has to be in the elements on the left side of the middle element If input less than middle, our value has to be in the elements on the left side of the middle element If input equals middle, we found the element. If input equals middle, we found the element.

6. Searching: Binary Search for (int left = 0, right = n –1; left <= right;) { middle =(left + right) / 2; if (input == list[middle]) return middle; else if (input < list[middle]) right = middle – 1; else left = middle + 1; } return – 1;

7. Searching: Binary Search 8 in [1, 2, 8, 10, 15, 32, 63, 64]? 1 st iteration: Left = 0, Right = 7, Middle = 3, List[Middle] = 10 Check 8 == 10 => No, 8 No, 8 < 10 2 nd iteration: Left = 0, Right = 2, Middle = 1, List[Middle] = 2 Check 8 == 2 => No, 8 > 2 3 rd iteration: Left = 2, Right = 2, Middle = 2 Check 8 == 8 => Yes, Found It!

8. Searching: Binary Search Binary Search Method: Binary Search Method: Number of operations to find input if in the list: Number of operations to find input if in the list: Dependent on position in list Dependent on position in list 1 operation if middle 1 operation if middle Log 2 n operations maximum Log 2 n operations maximum Number of operations to find input if not in the list: Number of operations to find input if not in the list: Log 2 n operations maximum Log 2 n operations maximum

9. Recursive Binary Search Two requirements for recursion: Two requirements for recursion: Same algorithm, smaller problem Same algorithm, smaller problem Termination condition Termination condition Binary search? Binary search? Search in half of previous array Search in half of previous array Stop when down to one element Stop when down to one element

10. Recursive Binary Search int BinarySearch(int *list, const int input, const int left, const int right) { if (left < right) { middle =(left + right) / 2; if (input == list[middle]) return middle; else if (input < list[middle]) return BinarySearch(list, input, left, middle-1); else return BinarySearch(list,input,middle+1,right); } return – 1; }

11. Fibonacci Computation Fibonacci Sequence: Fibonacci Sequence: 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, … 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, … Simple definition: Simple definition: Fib[0] = 1 Fib[0] = 1 Fib[1] = 1 Fib[1] = 1 Fib[N] = Fib(N-1) + Fib(N-2) Fib[N] = Fib(N-1) + Fib(N-2)

12. Recursive Fibonacci int fibonacci(int input) { if ((input == 0) || (input == 1)) return 1; if ((input == 0) || (input == 1)) return 1; else return (fibonacci(input-1) + fibonacci(input-2)); else return (fibonacci(input-1) + fibonacci(input-2));}

13. Iterative Fibonacci int fibonacci(int input) { int first = 1; int first = 1; int second = 1; int second = 1; int temp; int temp; for (int k = 0; k < input; k++) for (int k = 0; k < input; k++) { temp = first; temp = first; first = second; first = second; second = temp + second; second = temp + second; } return first; return first;}

14. Types of Recursion Linear Recursion: Linear Recursion: 1 recursive call per function 1 recursive call per function Factorial, Binary Search examples Factorial, Binary Search examples Tree Recursion: Tree Recursion: 2 or more recursive calls per function 2 or more recursive calls per function Fibonacci Example Fibonacci Example

15. Efficiency of Recursion Recursion can sometimes be slower than iterative code Recursion can sometimes be slower than iterative code Two main reasons: Two main reasons: Program stack usage Program stack usage When using recursive functions, every recursive call is added to the stack and it grows fast. When using recursive functions, every recursive call is added to the stack and it grows fast. Result generation Result generation

16. Efficiency of Recursion Stack Usage: Stack Usage: When a function is called by a program, that function is placed on the program call stack: When a function is called by a program, that function is placed on the program call stack: Every stack entry maintains information about the function: Every stack entry maintains information about the function: Where to return to when the function completes Where to return to when the function completes Storage for local variables Storage for local variables Pointers or copies of arguments passed in Pointers or copies of arguments passed in main() getData() readFile() Returns file data to be used in getData() Returns formatted data to be printed in main()

17. Efficiency of Recursion Another Reason for Slowdowns [Tree Recursion] Another Reason for Slowdowns [Tree Recursion] Traditional Recursion doesn’t save answers as it executes Traditional Recursion doesn’t save answers as it executes Fib(5) Fib(5) = Fib(4) + Fib(3) = Fib(3) + Fib(2) + Fib(3) = Fib(2) + Fib(1) + Fib(2) + Fib(3) = Fib(1) + Fib(0) + Fib(1) + Fib(2) + Fib(3) = Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(3) = Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(2) + Fib(1) = Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(1) + Fib(0) + Fib(1) Solution: Dynamic programming – saving answers as you go and reusing them Solution: Dynamic programming – saving answers as you go and reusing them

18. Dynamic Programming Fibonacci Problem Fibonacci Problem We know the upper bound we are solving for We know the upper bound we are solving for I.e. Fibonacci (60) = 60 different answers I.e. Fibonacci (60) = 60 different answers Generate an array 60 long and initialize to –1 Generate an array 60 long and initialize to –1 Everytime we find a solution, fill it in in the array Everytime we find a solution, fill it in in the array Next time we look for a solution, if the value in the array for the factorial we need is not –1, use the value present. Next time we look for a solution, if the value in the array for the factorial we need is not –1, use the value present.

19. Algorithmic Complexity

20. Algorithmic Complexity: Two Key Metrics Space Efficiency: Space Efficiency: The maximum amount of memory required to perform a computation and how that maximum amount is related to the size of the input The maximum amount of memory required to perform a computation and how that maximum amount is related to the size of the input Time Efficiency: Time Efficiency: The maximum number of computations required to perform a computation and how that maximum amount is related to the size of the input The maximum number of computations required to perform a computation and how that maximum amount is related to the size of the input We’ll look at these in terms of recursion first, then extend to more general discussion. We’ll look at these in terms of recursion first, then extend to more general discussion.

21. Determining Time Efficiency Two ways to model: Two ways to model: Experimental: Adding in counters to measure the number of operations performed Experimental: Adding in counters to measure the number of operations performed Theoretical: Using mathematical model of a functions computational requirements Theoretical: Using mathematical model of a functions computational requirements Recursion? Some problems have very nice mathematical functions - Let’s start with those first. Recursion? Some problems have very nice mathematical functions - Let’s start with those first.

22. Recursion Time Efficiency: Recurrence Relations Computational Requirements: Three Issues Computational Requirements: Three Issues Amount of work needed for current iteration Amount of work needed for current iteration Cost of basic function requirements Cost of basic function requirements Number of subproblems that have to be solved Number of subproblems that have to be solved Linear recurrence vs Tree recurrence Linear recurrence vs Tree recurrence Size (in terms of input) of subproblems to be solved Size (in terms of input) of subproblems to be solved How much smaller is each of the subproblems How much smaller is each of the subproblems

23. Recursive Binary Search Recursive Binary Search Requirements: Recursive Binary Search Requirements: Amount of work needed for current iteration Amount of work needed for current iteration 1 Comparison (inputValue versus middle index) 1 Comparison (inputValue versus middle index) 1 Adjustment (Changing left or right parameters) 1 Adjustment (Changing left or right parameters) Number of subproblems that have to be solved Number of subproblems that have to be solved 1 Subproblem (the left or right side of the array) 1 Subproblem (the left or right side of the array) Size (in terms of input) of subproblems to be solved Size (in terms of input) of subproblems to be solved Subproblem is half the size of the original problem Subproblem is half the size of the original problem

24. Recurrence Relation General Recurrence Relation: General Recurrence Relation: T(n) = aT(n/b) + cn k a = number of sub problems b = 1/size of subproblems f(n) = current iteration work = constant * n k

25. Recurrence: Master Theorem T(n) = aT(n/b) + f (n) where f (n) ≈ n k 1. a < b k T(n) ~ n k 2. a = b k T(n) ~ n k lg n 3. a > b k T(n) ~ n log b a

26. Recurrence: Master Theorem T 1 (n) = 7 T 1 (n/7)+n a = 7, b = 7, k = 1a ? b k 7 == 7   n k lgn=> n lgn T 2 (n) = 7 T 2 (n/2)+n 2 a = 7, b = 2, k = 2a ? b k 7 > 2 2 => n log b a => n log 2 7 => n 2.81 T 3 (n) = 7 T 3 (n/3)+n 2 a = 7, b = 3, k = 2a ? b k 7 < 3 2 => n k => n 2

27. Recursive Binary Search T(n) = aT(n/b) + f(n) a = number of sub problems= 1 b = 1/size of subproblems= 1/(1/2) => 2 f(n) = current iteration work= 2n 0 so k = 0 Compare a to b k :1 vs 2^0= 1 vs 1 If they are equal, computational cost is: n k log n= 1 * log n=>log n [Formula can be looked up for >,, <, and ==]

28. Space Complexity Why are we interested in space complexity? Why are we interested in space complexity? Most people worry about time complexity. Most people worry about time complexity. In general, problems with high time complexity can still be solved – we just have to wait longer (albeit there are some problems that take so long they are not solvable in our lifetime/multiple lifetimes). In general, problems with high time complexity can still be solved – we just have to wait longer (albeit there are some problems that take so long they are not solvable in our lifetime/multiple lifetimes). Problems with too large space complexity we may not be able to run at all. Our finiteness of memory is more much more evident than our finiteness of time. Problems with too large space complexity we may not be able to run at all. Our finiteness of memory is more much more evident than our finiteness of time.

29. Space Complexity In general, two components: In general, two components: Fixed requirementsC Fixed requirementsC Instruction space (code) Instruction space (code) Constants Constants Simple variables Simple variables Instance requirementsS p Instance requirementsS p Variables whose size is dependent on problem instance Variables whose size is dependent on problem instance Recursion stack space Recursion stack space S(P) = C + S p S(P) = C + S p Ignore C as it is usually dominated by S p Ignore C as it is usually dominated by S p

30. Space Complexity float abc(float a, float b, float c) { return a + b + b*c + (a+b-c) / (a+b) + 4.0; } Assume space required is 1 word for each float. No matter what a, b, c are entered, need same amount of memory to execute. Space required for this function is independent of inputs, so S abc = 0

31. Space Complexity float sum(float *a, const int n) { float s = 0; for (int i =0; i < n; i++) s += a[i]; return s; } Summing over an entire array, problem size = array length [n]. Need four words - one word apiece for a, n, s, i. Again requirements for this function are independent of inputs, so S sum = 0 Note the difference between this function space requirements and program as a whole (somebody has to allocate the four words for the data in the array). Note the difference between this function space requirements and program as a whole (somebody has to allocate the four words for the data in the array).

32. Space Complexity float rsum(float *a, const int n) { if (n <= 0) return 0; else return (rsum(a, n-1) + a[n-1]); } Summing over an entire array, problem size = array length [n]. Need four words - one word apiece for a, n, s, i. Every recursive call needs those four words, and depth of recursion is n + 1 (base = 0, 1..n) Requirements for this function are dependent on inputs, so S rsum = 4(n+1)

33. Time Complexity T(P) = Compile time + Execution Time T(P) = Compile time + Execution Time Compile time is: Compile time is: Not dependent on program inputs Not dependent on program inputs Negligible in comparison to execution time for real world, extensively used programs Negligible in comparison to execution time for real world, extensively used programs Execution time: Execution time: Difficult to accurately obtain Difficult to accurately obtain Hardware dependent, OS dependent, load dependent … Hardware dependent, OS dependent, load dependent … Estimate in terms of program steps Estimate in terms of program steps

34. Time Complexity in C++ Context Comments: Comments: Not executed Not executed Step count = 0 Step count = 0 Declarations: (int b; ) Declarations: (int b; ) Either not executable or lumped into cost of calling function Either not executable or lumped into cost of calling function Step count = 0 Step count = 0

35. C++ Time Complexity C++ Time Complexity Expressions: (for example: a == true) Expressions: (for example: a == true) Generally step count = 1 Generally step count = 1 If expression contains functions, sum of costs for function invocations (defined later) If expression contains functions, sum of costs for function invocations (defined later) Assignments: = Assignments: = Generally, step count for evaluation of expression Generally, step count for evaluation of expression If variable size dependent on instance, If variable size dependent on instance, step count = size of variable + expression step count step count = size of variable + expression step count I.E. a = (2+b); a,b both lists, then: I.E. a = (2+b); a,b both lists, then: step count = cost of evaluating expression (2+b) + cost of copying list b into list a step count = cost of evaluating expression (2+b) + cost of copying list b into list a

36. C++ Time Complexity Iteration: Iteration: Consider loop control elements, not internal statements of the loop body Consider loop control elements, not internal statements of the loop body While do While do Do… while Do… while For ( ; ; ) For ( ; ; ) while, do: Each iteration requires step count of while, do: Each iteration requires step count of for: for: Generally 1 for each iteration unless,, dependent on size Generally 1 for each iteration unless,, dependent on size 1 st time (initialize, test) = steps + steps 1 st time (initialize, test) = steps + steps All other times (update, test) = steps + steps All other times (update, test) = steps + steps Generally have to count iterations + 1 (all true iterations where fall into body plus one where the condition fails) Generally have to count iterations + 1 (all true iterations where fall into body plus one where the condition fails)

37. C++ Time Complexity Switch statement: Switch statement: switch { case cond1: case cond2: default: } switch { case cond1: case cond2: default: } Header: steps Header: steps Conditions: Cost of own check plus cost of all preceding checks Conditions: Cost of own check plus cost of all preceding checks If/else: If/else: If ( ) ; else If ( ) ; else = true: + cost = true: + cost = false: + cost = false: + cost

38. C++ Time Complexity Function invocation: Function invocation: Step cost of 1 generally Step cost of 1 generally Pass by value parameters: If variable size dependent on problem instance, add step cost of size (to do copy) Pass by value parameters: If variable size dependent on problem instance, add step cost of size (to do copy) If recursive, include all copies of local variables that are size dependent because must be generated in recursive call If recursive, include all copies of local variables that are size dependent because must be generated in recursive call

39. C++ Time Complexity Memory management: new/delete Memory management: new/delete Step cost 1 generally Step cost 1 generally If call constructor, destructor, compute similar to function invocation (take into account cost of pass by value parameters) If call constructor, destructor, compute similar to function invocation (take into account cost of pass by value parameters) Jump statements: continue/ break/ goto/ return / return : Jump statements: continue/ break/ goto/ return / return : Step cost 1 for continue/break/goto/return Step cost 1 for continue/break/goto/return return : Cost of return : Cost of

40. Measuring Complexity First approach: First approach: Extend programs to incorporate step count statements Extend programs to incorporate step count statements Add global variable count Add global variable count Whenever a statement is executed, add an additional statement that increments count Whenever a statement is executed, add an additional statement that increments count

41. Measuring Complexity float sum(float *a, const int n) { float s = 0; for (int i =0; i < n; i++) s += a[i]; return s; } float sum(float *a, const int n) { float s = 0; count++;// assignment for (int i =0; i < n; i++) { count++; s += a[i]; count++;}// 1 for for, 1 for += count ++; // last time checking for return s; count ++; // return statement count ++; // return statement}

42. Measuring Complexity Strip out everything except count statements: Strip out everything except count statements: float sum(float *a, const int n) { count++;// assignment for (int i =0; i < n; i++) { count = count + 2;}// 1 for for, 1 for += // last time checking for and return statement count = count + 2; }

43. Measuring Complexity: Sum all count statements: Sum all count statements: 1 at beginning + (N iterations of loop) * (2 within a loop) + 2 at end => 2N + 3

44. Measuring Complexity float rsum(float *a, int n) { if (n <= 0) return 0; else return (rsum(a, n-1) + a[n-1]); } float rsum(float *a, int n) { count ++; // if conditional if (n <= 0) { count++; return 0;} // return else { count++; return (rsum(a,n-1) + a[n-1]); // return statement } How to handle recursion?

45. Measuring Complexity: Recursion float rsum(float *a, cont int n) { count ++; // if conditional if (n <= 0) { count++;} // return else { count++; return (rsum(a,n-1) + a[n-1]); } If (n <= 0) count = 2 Otherwise count = 2 + count(n-1)

46. Measuring Complexity: Recursion 2 + count(n-1)Recurrence relation 2 + count(n-1)Recurrence relation Solve by repeatedly substituting count(n-1) with its recursive definition Solve by repeatedly substituting count(n-1) with its recursive definition => 2+ count(n-1) = 2 + 2 + count(n-2) = 2 + 2 + 2 + count(n – 3) … = 2 * n + count(0)=> 2n + 2

47. Time Complexity Iterative Sum: 2n + 3 Iterative Sum: 2n + 3 Recursive Sum:2n + 2 Recursive Sum:2n + 2 Is recursive sum faster than iterative? Is recursive sum faster than iterative? Not necessarily – Each step of recursive sum may actually be more expensive than a step of iterative sum Not a problem - We are really interested in measurements in relation to input size, which we see are very similar.

48. Time Complexity 2n + 2=> Run time is linear in N 2n + 2=> Run time is linear in N If input grows by a factor of 10, execution time should grow by a factor of 10 If input grows by a factor of 10, execution time should grow by a factor of 10 If input grows by a factor of 10,000, cost should grow by a factor of 10,000 If input grows by a factor of 10,000, cost should grow by a factor of 10,000 Xn + C=> Xn often dominates constant term so we often ignore C

49. Time Complexity Matrix addition example: Matrix addition example: void add(matrix a, matrix b, matrix c, int m, int n) { for (int i = 0; i < m; i++) { for (int j = 0; j < n; j++) { c[i][j] = a[i][j] + b[i][j] }}}

50. Time Complexity void add(matrix a, matrix b, matrix c, int m, int n) { for (int i = 0; i < m; i++) { count++; // for loop for (int j = 0; j < n; j++) { count++; // for loop c[i][j] = a[i][j] + b[i][j] count++; // assignment } count ++; // last time through for loop on j } count++; // last time through for loop on i }

51. Time Complexity void add(matrix a, matrix b, matrix c, int m, int n) { for (int i = 0; i < m; i++) { count = count + 2; // for loop start, last time on j [case 1] for (int j = 0; j < n; j++) { count = count + 2; } // for loop start, assignment [case 2] count = count + 2; } // for loop start, assignment [case 2]} } count++; // last time through for loop on i [case 3] } => 2m [case 1]+ 2mn [case 2] + 1 [case 3]

52. Time Complexity Matrix addition: 2mn + 2m + 1 What does that tell us? Potential speedup: If m >> n, then we might want to switch order of loops, can change 2m to 2n in the middle In general – dominated by 2mn statement

53. Measuring Complexity First approach: Add count statements First approach: Add count statements Second approach: Tabulation method Second approach: Tabulation method Steps per execution of each statement Steps per execution of each statement Frequency of each statement Frequency of each statement

54. Tabular Method for Complexity float sum(float *a, const int n) 1 { 2float s = 0; 3 for (int i =0; i < n; i++) 4 s += a[i]; 5return s; 6} Line S/E Freq Total 101 0 2111 31 n+1 n+1 41 nn 5111 6010 Total: 2n+3

55. Tabular Method for complexity float rsum(float *a, const int n) 1 { 2if (n <= 0) return 0; 3else return (rsum(a, n-1) + a[n- 1]); 4} Line S/E Freq Total =0 >0 =0 >0 10 1 1 0 0 21 1 1 1 1 2b 1 1 0 1 0 31 + t n-1 0 1 0 1+t n-1 401 1 0 0 Total: n = 0=> 2 n > 0=> 2+ t n-1

56. Tabular Method for Complexity void add(matrix a, matrix b, matrix c, int m, int n) { 1for (int i = 0; i < m; i++) { 2for (int j = 0; j < n; j++) { 3c[i][j] = a[i][j] + b[i][j] }}} Line S/E Freq Total 11 m+1 m+1 21 m(n+1) mn+m 31 mnmn Total: 2mn + 2m + 1

57. Time Complexity Best-Case: Minimum number of steps that can be executed for the given parameters Best-Case: Minimum number of steps that can be executed for the given parameters Worst-Case: Maximum number of steps that can be executed for the given parameters Worst-Case: Maximum number of steps that can be executed for the given parameters Average-Case: Average number of steps that can be executed for the given parameters Average-Case: Average number of steps that can be executed for the given parameters

58. Time Complexity T(A) = c 1 n 2 + c 2 nT(B) = c 3 n Consider c 1 = 5, c 2 = 3, c 3 = 100 When is cost of algorithm A > cost of algorithm B? (When do we not want to use A anymore?) 5n 2 + 3n > 100n=> 5n 2 > 97n => 5n > 97 => n > 19.4 19.4 is the breakpoint

59. Big “Oh” Notation Defines relationships between functions Defines relationships between functions F(n) is O(g(n)) if F(n) is O(g(n)) if Exists c and n 0 > 0 such that f(n) = n 0 Exists c and n 0 > 0 such that f(n) = n 0 3n + 2 is O(n) because: 3n + 2 is O(n) because: 3n+2 = 2 (c = 4, n 0 = 2)

60. Big “Oh” notation 10n 2 + 4n + 2 is O(n 2 ) because: 10n 2 + 4n + 2 is O(n 2 ) because: 10n 2 + 4n + 2 = 5 (c = 11, n 0 = 5) 10n 2 + 4n + 2 is not O(n) because: 10n 2 + 4n + 2 is not O(n) because: there do not exist any c or n 0 for which 10n 2 + 4n + 2 is = n 0

61. Big “Oh” notation C = 100? 10n 2 +4n+2 < 100n 10n 2 n n < 9.5 If n = 1 through 9, inequality holds, but breaks around n = 10 C = 1000? C = 1000? 10n 2 +4n+2 < 1000n 10n 2 n n < 99.5 If n = 1 through 99, inequality holds, but breaks around n = 100 Can’t find a n 0 for which it holds for all n >= n 0

62. Big “Oh” Notation f(n) = O(g(n)) statement provides an upper bound on the value of f(n) f(n) = O(g(n)) statement provides an upper bound on the value of f(n) Not necessarily the best upper bound: Not necessarily the best upper bound: Want to use minimum upper bound you can find F(n) = 4n + 2 is O(n) and O(n 2 ) and O(n 3 ).. => Use O(n)

63. Big “Oh” notation O(1)= constant O(1)= constant O(n)= linear O(n)= linear O(n 2 )= quadratic O(n 2 )= quadratic O(n 3 )= cubic O(n 3 )= cubic O(log n)=logarithmic O(log n)=logarithmic O(2 n )=exponential O(2 n )=exponential Commonly see algorithms that are O(n log n) as well Commonly see algorithms that are O(n log n) as well

64. Big “Oh” Notation If a function has multiple terms with n, the first term dominates. If a function has multiple terms with n, the first term dominates. f(n) = 4n 5 + 6n 3 + 2n 2 + n + 8 => O(n 5 ) f(n) = 4n 5 + 6n 3 + 2n 2 + n + 8 => O(n 5 )

65. Omega Notation f(n) = Ω(g(n)) iff f(n) = Ω(g(n)) iff Exists c and n 0 > 0 such that f(n) >= cg(n) for all n, n >= n 0 Exists c and n 0 > 0 such that f(n) >= cg(n) for all n, n >= n 0 3n + 2 = Ω(n) because: 3n + 2 = Ω(n) because: 3n + 2 >= 3n for n >= 1 (c = 3, n = 1) 10n 2 + 4n + 2 = Ω(n 2 ) because: 10n 2 + 4n + 2 = Ω(n 2 ) because: 10n 2 + 4n + 2 >= n 2 for n >= 1 (c = 1, n = 1) Also Ω(n), Ω(1)

66. Omega Notation Ω(g(n)) provides a lower bound on computation time Ω(g(n)) provides a lower bound on computation time Should use tightest lower bound you can find Should use tightest lower bound you can find

67. Theta Notation f(n) = Θ(g(n)) iff f(n) = Θ(g(n)) iff Exist positive constants c 1, c 2, and n 0 such that c 1 g(n) = n 0 Exist positive constants c 1, c 2, and n 0 such that c 1 g(n) = n 0 3n + 2 is Θ(n) because: 3n + 2 is Θ(n) because: 3n + 2 = 2O(n) 3n + 2 >= 3n for all n >= 2Ω(n) Θ(g(n)) provides a tight upper and lower bound on f(n) Θ(g(n)) provides a tight upper and lower bound on f(n)

68. Graphs Points to notice: Points to notice: What happens near the beginning ( n < N ) is not important What happens near the beginning ( n < N ) is not important In the third diagram, c 1 g(n) and c 2 g(n) have the same “shape” (why?) In the third diagram, c 1 g(n) and c 2 g(n) have the same “shape” (why?) f(n) cg(n) f(n) is O(g(n)) N f(n) cg(n) f(n) is  (g(n)) N f(n) c 2 g(n) c 1 g(n) f(n) is  (g(n)) N http://www.cis.upenn.edu/~matuszek/cit594-2004/Lectures/49-analysis-2.ppt

69. Interesting Problems Problems with different O and Ω Problems with different O and Ω Think of Big O: I know it requires at most this much work, so if my algorithm is more than that, it’s a bad algorithm (Best known algorithm can provide this) Think of Big Ω: I know it requires at least this much work, so I should strive to get my algorithm to this point. (Theoretical proof over all algorithms usually provides this). (Theoretical proof over all algorithms usually provides this). Would like to have same Big O and Big Ω=> Then you have optimal algorithm. There are some algorithms where the Big O and Big Ω are different – These are ripe for improvement.

70. For self reading

71. Computational Requirements Require a clocking function with significant resolution Require a clocking function with significant resolution Decide on size of inputs (n) for which times are to be obtained Decide on size of inputs (n) for which times are to be obtained Determine, for each of the above values of n, data sets that exhibit worst case behavior Determine, for each of the above values of n, data sets that exhibit worst case behavior

72. Timing Function clock() function clock() function Returns number of clock ticks since start of program execution Returns number of clock ticks since start of program execution #include #include Tends to be more accurate than other time of day functions because program dependent, not OS dependent. Tends to be more accurate than other time of day functions because program dependent, not OS dependent. clock_t start, end; clock_t start, end; double elapsed; double elapsed; start = clock(); start = clock(); … do something … … do something … end = clock(); end = clock(); elapsed = ((double) (end - start)) / CLOCKS_PER_SEC; elapsed = ((double) (end - start)) / CLOCKS_PER_SEC;

73. Choice of N to Use Assume we are measuring sequential search function Assume we are measuring sequential search function Want to predict how long it will take, in worst case, to search for x in an array of size n. Want to predict how long it will take, in worst case, to search for x in an array of size n. Find a graph and its corresponding function from data Find a graph and its corresponding function from data Asymptotic analysis: O(n) Asymptotic analysis: O(n) Expect plot of times to be a straight line Expect plot of times to be a straight line Asymptotic behavior only guaranteed to hold for sufficiently large values of n Asymptotic behavior only guaranteed to hold for sufficiently large values of n In general, O(n) function may not always have points end up on a line due to lower order costs (such as a log n) In general, O(n) function may not always have points end up on a line due to lower order costs (such as a log n)

74. Choice of N to Use Book suggests: Book suggests: Likely that asymptotic behavior will have started by size n = 100 Likely that asymptotic behavior will have started by size n = 100 Use a fairly separated set of points from 100+ Use a fairly separated set of points from 100+ Use a fairly refined set of points from 0 to 100 to get good idea of low end behavior Use a fairly refined set of points from 0 to 100 to get good idea of low end behavior

75. Repetition Can make up for poor timing resolution through repeated execution timing blocks: int ITERATIONS = 1000000; clock_t start, end; double sum = 0; for (int i= 0; i < ITERATIONS; i++) { start = clock(); start = clock(); … work … … work … end = clock(); sum += ((double) (end - start)) / CLOCKS_PER_SEC; } double averageTime = sum / ITERATIONS.

76. Worst Case Selection Examine algorithm to determine inputs which provide worst case performance Examine algorithm to determine inputs which provide worst case performance Sequential search: Sequential search: When item searched for is not present in the array When item searched for is not present in the array Guarantees have to search entire array Guarantees have to search entire array

77. Worst Case Selection void insertionSort(int* a, int size) { for (int k = 1; k < size; k++) { int temp = a[k]; int position = k; while (position > 0 && a[position-1] > temp) { a[position] = a[position-1]; position--;} a[position] = temp; }}

78. Worst Case Selection Insertion Sort Worst Case? Most comparisons occur when while loop runs all the way to position == 0 With what type of data would this occur? Reverse Sorted

79. Worst Case Selection void selectionSort(int* a, int size) { for (int k = 0; k < size-1; k++) { int index = mininumIndex(a, k, size-1); swap(a[k],a[index]);}} int minimumIndex(int* a, int first, int last) { minIndex = first; for (int j = first + 1; j <= last; j++) { if (a[j] < a[minIndex]) minIndex = j; } return minIndex; }

80. Worst Case Selection Selection Sort Worst Case? For comparisons, there is no case worse than any other. There is no arrangement of data which changes how many comparisons the minIndex function performs, so the same number of spots in the array are always examined for a given input size. However, reverse sorted data does force more assignments (everything is a new min).

81. Worst Case Selection Adding Polynomials: Adding Polynomials: 3x 3 + 2x + 1 + 2x 3 + 3x 2 + 5 2x 3 + 3x 2 + 5================= 5x 3 + 3x 2 + 2x + 6 5x 3 + 3x 2 + 2x + 6

82. Polynomials Adding Polynomials: Adding Polynomials: Iterate through both lists Iterate through both lists If exponent1 == exponent2, If exponent1 == exponent2, CoefficientSum = Coefficient1 + Coefficient2 CoefficientSum = Coefficient1 + Coefficient2 If CoefficentSum != 0, add term to new polynomial representing answer If CoefficentSum != 0, add term to new polynomial representing answer Else, Else, Find higher exponent Find higher exponent Add term to new polynomial representing answer Add term to new polynomial representing answer

83. Polynomials 3 2 11 02 33 25 05 33 22 16 0

84. Worst Case Selection Polynomial addition worst case? Polynomial addition worst case? All exponents are different and interleaved. Requires O(m+n) exponent comparisons, where m, n are list sizes. Best you can do? All same exponents – Requires O(m) comparisons - which is size of either list (have to be same size to have all same exponents).

85. Worst Case/Average Case Generation Not always possible to generate worst case data or average case data by hand Not always possible to generate worst case data or average case data by hand Average case – often times way too many problem instance to test them all and do average Average case – often times way too many problem instance to test them all and do average Best approach: Generate a large number of random instances Best approach: Generate a large number of random instances Worst case – Worst overall time out of instances Worst case – Worst overall time out of instances Average case – Average over instance times Average case – Average over instance times

86. Analysis of Search Functions seqsearch.cpp program seqsearch.cpp program

87. Analysis of Search Functions

89. Analysis of Sorting