More Computational Theory Code analysis Mathematical induction September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien Analyzing Code Types of analysis Bound flavour Upper bound (O) Lower bound (Ω), useful for problems Asymptotically tight (Θ) Analysis case Worst case (adversary) Average case Best case / "lucky" case "common" case Analysis quality Loose bound (any true analysis) Tight bound (no better "meaningful" bound that is asymptotically different) September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien Input Varies The number of operations usually varies based on the size of the input Though not always – consider array lookup In addition algorithm performance may vary based on the organization of the input For example consider searching a large array If the target is the first item in the array the search will be very fast September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien Linear Search // Iterative linear search int linSearch(int arr[], int size, int x) { for (int i = 0; i < size; i++) if (x == arr[i]) return i; } return -1; //target not found The function returns as soon as a match is found -1 is returned if the target item is not found by the time the array end is reached September 12, 2017 Cinda Heeren / Geoffrey Tien

Linear Search Comparisons For different input organization Worst case The target is not in the array or The target is at the last position in the array Make n comparisons in either case Best/lucky case (this rarely occurs) The target is the first element of the array Make 1 comparison Average case Is it (best case + worst case) / 2, i.e. (n + 1) / 2? Average/usual case analysis is tricky and requires several assumptions that may not hold in all situations 𝑂 𝑛 𝑂 1 𝑂 𝑛 September 12, 2017 Cinda Heeren / Geoffrey Tien

Searching Sorted Arrays If we sort the target array first we can change the linear search average cost to around 𝑛 2 Once a value equal to or greater than the target is found the search can end So, if a sequence contains 8 items, on average, linear search compares 4 of them, If a sequence contains 1,000,000 items, linear search compares 500,000 of them, etc. However, if the array is sorted, it is possible to do much better than this by using binary search September 12, 2017 Cinda Heeren / Geoffrey Tien

Binary Search Algorithm int binSearch(int arr[], int size, int target) { int low = 0; int high = size - 1; int mid = 0; while (low <= high) mid = (low + high) / 2; if (target == arr[mid]) return mid; else if (target > arr[mid]) low = mid + 1; else //target < arr[mid] high = mid - 1; } return -1; //target not found While subarray has at least 1 element September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien Worst Case Binary Search What is the worst case for binary search? Either the target is not in the array, or It is found when the search space consists of one element How many times does the while loop iterate in the worst case? binSearch(arr, n, 64); mid = (0+15) / 2 = 7 mid = (8+15) / 2 = 11 mid = (8+10) / 2 = 9 mid = (10+10) / 2 = 10 Done 7 11 15 21 29 32 44 45 57 61 64 73 79 81 86 92 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien Worst Case Analysis Binary Search Each iteration of the while loop halves the search space For simplicity assume that 𝑛 is a power of 2 So 𝑛= 2 𝑘 (e.g. if 𝑛=128, 𝑘=7) How large is the search space? The first iteration halves the search space to 𝑛 2 After the second iteration the search space is 𝑛 4 After the kth iteration the search space consists of just one element, since 𝑛 2 𝑘 = 𝑛 𝑛 =1 Because 𝑛= 2 𝑘 , 𝑘= log 2 𝑛 Therefore at most log 2 𝑛 iterations of the while loop are made in the worst case! 𝑂 log 𝑛 September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien Average Case Binary Search Is the average case more like the best case or the worst case? What is the chance that an array element is the target 1/n the first time through the loop 1/(n/2) the second time through the loop … and so on … It is more likely that the target will be found as the search space becomes small That is, when the while loop nears its final iteration We can conclude that the average case is more like the worst case than the best case 𝑂 log 𝑛 September 12, 2017 Cinda Heeren / Geoffrey Tien

Analysing nested loops Generally, we count the number of times an inner loop is repeated, for each time an outer loop is executed e.g. a function which scans an array looking for duplicates, worst case bool hasDuplicate(int arr[], int size) { for (int i = 0; i < size-1; i++) { for (int j = i+1; j < size; j++) { if (arr[i] == arr[j]) return true; } return false; outer loop: 𝑛−1 times, 𝑂 𝑛 inner loop: between 1 to 𝑛−1 times, 𝑂 𝑛 Is it 𝑂 𝑛 ×𝑂 𝑛 =𝑂 𝑛 2 ? September 12, 2017 Cinda Heeren / Geoffrey Tien

Analysing nested loops hasDuplicate, worst case bool hasDuplicate(int arr[], int size) { for (int i = 0; i < size-1; i++) { for (int j = i+1; j < size; j++) { if (arr[i] == arr[j]) return true; } return false; 𝑖 # of inner loop executions 𝑛−1 1 𝑛−2 2 𝑛−3 ... # of inner loop executions In this case, there is a constant number of elementary operations in the inner loop, so worst case is 𝑂 𝑛 2 ∙𝑂 1 =𝑂 𝑛 2 September 12, 2017 Cinda Heeren / Geoffrey Tien

Mathematical Induction Warning! Clipart lion cannibalism inside! Viewer discretion is advised. September 12, 2017 Cinda Heeren / Geoffrey Tien

Proof techniques The adventure thus far… Witness Counterexample Show an example satisfying an existence theorem Counterexample Provide a specific example showing the falsity of a universal quantification Contradiction Assume the opposite of the theorem Derive a contradiction Mathematical induction Prove a property for one or more base cases (e.g., 𝑛=1) Assume the property is true for one or more anonymous values (e.g. 𝑛=𝑘, 𝑛≤𝑘) Using the assumption above, prove for the next value (e.g. 𝑛=𝑘+1) So, we use an unproven theorem, to prove itself. WHAT?? September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien Proof by Induction Principle and insight Let 𝑃 𝑛 be a property defined on the integers 𝑛, and let 𝑎 be a fixed integer. Suppose the following two statements are true: 𝑃 𝑎 is true. For all integers 𝑘≥𝑎, if 𝑃 𝑘 is true, then 𝑃 𝑘+1 is true. Then the statement: For all integers 𝑛≥𝑎, 𝑃 𝑛 is true. Assume this is true Derive the truth of this "peel away" one item, replace with 𝑃 𝑘 Directly prove base cases Break the property down into smaller cases Assuming property is true for smaller cases, build up to 𝑘 case September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien Lion cuisine Induction example In the African savanna, lions are starving They may eat one another if given the opportunity They want to avoid being eaten by another lion They are experts in mathematical induction Suppose a zebra drops dead surrounded by a group of hungry lions. Does a lion approach and eat the zebra (without being eaten)? September 12, 2017 Cinda Heeren / Geoffrey Tien

Who eats the zebra? 𝑛=1 𝑛=2 𝑛=3 𝑛=4 Pattern? How many lions are there? The lion eats the zebra, will not be attacked A base case! 𝑛=2 Nobody eats the zebra, the lions fear each other Another base case! 𝑛=3 One lion eats the zebra, the other two fear each other The eating lion becomes like the zebra Reduction to the 2-lion case 𝑛=4 If a lion eats, the problem is reduced to the 3-lion case, and the eating lion will be eaten So, nobody eats the zebra Pattern? September 12, 2017 Cinda Heeren / Geoffrey Tien

Lions Theorem: Basis step: Inductive step: And mathematical induction For 𝑛≥1 lions, the zebra will be eaten only if 𝑛 is an odd number. i.e. 𝑃 𝑛 = the zebra will be eaten only if 𝑛 is odd Basis step: 𝑛=1, 𝑛=2, already demonstrated on previous slide. Inductive step: Inductive hypothesis (I.H.): Assume the theorem holds for an arbitrary integer 𝑘≥1, i.e. 𝑃 𝑘 is true Show that 𝑃 𝑘+1 is true September 12, 2017 Cinda Heeren / Geoffrey Tien

Cinda Heeren / Geoffrey Tien A feast for 𝑘+1 lions Inductive step, case 1: 𝑘 is odd By the I.H., one among the first 𝑘 lions will eat the zebra and become vulnerable This lion can be eaten by the 𝑘+1 𝑡ℎ lion, thus backs away to defend himself Therefore, if 𝑘 is odd, 𝑘+1 is even, and nobody eats the zebra Inductive step, case 2: 𝑘 is even By the I.H., nobody among the first 𝑘 lions eats the zebra The 𝑘+1 𝑡ℎ lion safely eats the zebra Therefore, if 𝑘 is even, 𝑘+1 is odd, and the zebra gets eaten QED September 12, 2017 Cinda Heeren / Geoffrey Tien

A numeric example Prove using mathematical induction: Basis step: Sum of the first 𝑛 integers Prove using mathematical induction: Basis step: Prove 𝑃 1 Inductive step: Prove that if 𝑃 𝑘 is true, then 𝑃 𝑘+1 is true September 12, 2017 Cinda Heeren / Geoffrey Tien

Breaking down sums What does really look like? What about ? Sum of the first 𝑛 integers What does really look like? What about ? Substitute this with our assumption from the inductive hypothesis September 12, 2017 Cinda Heeren / Geoffrey Tien

A numeric example Inductive step Sum of the first 𝑛 integers Inductive hypothesis: Assume that 𝑃 𝑘 is true, i.e. If 𝑃 𝑘+1 is true, Substitute by I.H. Multiply by 2 2 𝑘+1 common factor QED September 12, 2017 Cinda Heeren / Geoffrey Tien

Readings for this lesson Epp Chapter 5.2 – 5.3 Next class: Koffman: Chapters P.7, 10.1 – 10.2, 10.4 Try it! Prove using mathematical induction, the following properties: is divisible by 8 Hint: an integer 𝑥 is divisible by 8 if 𝑥=8∙𝑚 Lion induction example from http://www.science4all.org/article/proof-by-induction/ September 12, 2017 Cinda Heeren / Geoffrey Tien