CS 3343: Analysis of Algorithms Review for final 11/6/2018

Final Exam Closed book exam Coverage: the whole semester Cheat sheet: you are allowed one letter-size sheet, both sides Monday, May 6, 10:30am – 1:00pm Basic calculator (no graphing) allowed 11/6/2018

Final Exam: Study Tips Study tips: Re-make your previous cheat sheets Study each lecture Study the homework and homework solutions Study the midterm exams Re-make your previous cheat sheets 11/6/2018

Topics covered (1) By reversed chronological order: Graph algorithms Representations MST (Prim’s, Kruskal’s) Shortest path (Dijkstra’s) Running time analysis with different implementations 11/6/2018

Topics covered (2) Dynamic programming Greedy algorithm LCS Restaurant location problem Shortest path problem on a grid Other problems How to define recurrence solution, and use dynamic programming to solve it Greedy algorithm Unit-profit restaurant location problem Fractional knapsack problem Prim’s, Kruskal’s, and Dijkstra’s are also examples of greedy algorithms How to prove that certain greedy choices are optimal 11/6/2018

Topics covered (3) Hash tables Binary heap and priority queue Division-based hash functions Multiplication-based hash functions A universal hash function family Binary heap and priority queue Heapify, buildheap, insert, exatractMax, changeKey Running time 11/6/2018

Topics covered (4) Sorting algorithms Order statistics Insertion sort Merge sort Quick sort Heap sort Linear time sorting: counting sort, radix sort Stability of sorting algorithms Worst-case and expected running time analysis Memory requirement of sorting algorithms Order statistics Rand-Select Worst-case Linear-time selection Running time analysis 11/6/2018

Topics covered (5) Analysis Analyzing non-recursive algorithms Order of growth Asymptotic notation, basic definition Limit method L’ Hopital’s rule Stirling’s formula Best case, worst case, average case Analyzing non-recursive algorithms Arithmetic series Geometric series Analyzing recursive algorithms Defining recurrence Solving recurrence Recursion tree (iteration) method Substitution method Master theorem 11/6/2018

Review for finals In chronological order Only the more important concepts Very likely to appear in your final Does not mean to be exclusive 11/6/2018

Asymptotic notations O: Big-Oh Ω: Big-Omega Θ: Theta o: Small-oh ω: Small-omega Intuitively: O is like  o is like <  is like   is like >  is like = 11/6/2018

Big-Oh Math: Engineering: O(g(n)) = {f(n):  positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n)  n>n0} Or: lim n→∞ g(n)/f(n) > 0 (if the limit exists.) Engineering: g(n) grows at least as faster as f(n) g(n) is an asymptotic upper bound of f(n) Intuitively it is like f(n) ≤ g(n) 11/6/2018

Big-Oh Claim: f(n) = 3n2 + 10n + 5  O(n2) Proof: 3n2 + 10n + 5  10n2 + 10n + 10  3 x 10 n2 for n  1 Therefore, Let c = 30 and n0 = 1 f(n)  c n2,  n ≥ n0 11/6/2018

Big-Omega Math: Engineering: Ω(g(n)) = {f(n):  positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n)  n>n0} Or: lim n→∞ f(n)/g(n) > 0 (if the limit exists.) Engineering: f(n) grows at least as faster as g(n) g(n) is an asymptotic lower bound of f(n) Intuitively it is like g(n) ≤ f(n) 11/6/2018

Big-Omega f(n) = n2 / 10 = Ω(n) Proof: f(n) = n2 / 10, g(n) = n g(n) = n ≤ n2 / 10 = f(n) when n ≥ 10 Therefore, c = 1 and n0 = 10 11/6/2018

Theta Math: Engineering: Θ(1) means constant time. Θ(g(n)) = {f(n):  positive constants c1, c2, and n0 such that c1 g(n)  f(n)  c2 g(n)  n  n0  n>n0} Or: lim n→∞ f(n)/g(n) = c > 0 and c < ∞ Or: f(n) = O(g(n)) and f(n) = Ω(g(n)) Engineering: f(n) grows in the same order as g(n) g(n) is an asymptotic tight bound of f(n) Intuitively it is like f(n) = g(n) Θ(1) means constant time. 11/6/2018

Theta Claim: f(n) = 2n2 + n = Θ (n2) Proof: We just need to find the three constants c1, c2, and n0 such that c1n2 ≤ 2n2+n ≤ c2n2 for all n > n0 A simple solution is c1 = 2, c2 = 3, and n0 = 1 11/6/2018

Using limits to compare orders of growth lim f(n) / g(n) = c > 0 ∞ f(n)  o(g(n)) f(n)  O(g(n)) f(n)  Θ (g(n)) n→∞ f(n)  Ω(g(n)) f(n)  ω (g(n)) 11/6/2018

Therefore, 2n  o(3n), and 3n  ω(2n) Compare 2n and 3n lim 2n / 3n = lim(2/3)n = 0 Therefore, 2n  o(3n), and 3n  ω(2n) n→∞ n→∞ 11/6/2018

L’ Hopital’s rule lim f(n) / g(n) = lim f(n)’ / g(n)’ n→∞ n→∞ If both lim f(n) and lim g(n) goes to ∞ n→∞ n→∞ 11/6/2018

∞ Compare n0.5 and log n lim n0.5 / log n = ? (n0.5)’ = 0.5 n-0.5 lim (n-0.5 / 1/n) = lim(n0.5) = Therefore, log n  o(n0.5) n→∞ ∞ 11/6/2018

Stirling’s formula (constant) 11/6/2018

Compare 2n and n! Therefore, 2n = o(n!) 11/6/2018

More advanced dominance ranking 11/6/2018

General plan for analyzing time efficiency of a non-recursive algorithm Decide parameter (input size) Identify most executed line (basic operation) worst-case = average-case? T(n) = i ti T(n) = Θ (f(n)) 11/6/2018

Analysis of insertion Sort Statement cost time__ InsertionSort(A, n) { for j = 2 to n { c1 n key = A[j] c2 (n-1) i = j - 1; c3 (n-1) while (i > 0) and (A[i] > key) { c4 S A[i+1] = A[i] c5 (S-(n-1)) i = i - 1 c6 (S-(n-1)) } 0 A[i+1] = key c7 (n-1) } 0 } 11/6/2018

Inner loop stops when A[i] <= key, or i = 0 Best case Inner loop stops when A[i] <= key, or i = 0 1 i j Key sorted Array already sorted 11/6/2018

Inner loop stops when A[i] <= key Worst case Inner loop stops when A[i] <= key 1 i j Key sorted Array originally in reverse order 11/6/2018

Inner loop stops when A[i] <= key Average case Inner loop stops when A[i] <= key 1 i j Key sorted Array in random order 11/6/2018

Find the order of growth for sums How to find out the actual order of growth? Remember some formulas Learn how to guess and prove 11/6/2018

Closed form, or explicit formula Arithmetic series An arithmetic series is a sequence of numbers such that the difference of any two successive members of the sequence is a constant. e.g.: 1, 2, 3, 4, 5 or 10, 12, 14, 16, 18, 20 In general: Recursive definition Closed form, or explicit formula Or: 11/6/2018

Sum of arithmetic series If a1, a2, …, an is an arithmetic series, then 11/6/2018

Closed form, or explicit formula Geometric series A geometric series is a sequence of numbers such that the ratio between any two successive members of the sequence is a constant. e.g.: 1, 2, 4, 8, 16, 32 or 10, 20, 40, 80, 160 or 1, ½, ¼, 1/8, 1/16 In general: Recursive definition Closed form, or explicit formula Or: 11/6/2018

Sum of geometric series if r < 1 if r > 1 if r = 1 11/6/2018

Remember them, or remember where to find them! Important formulas Remember them, or remember where to find them! 11/6/2018

Sum manipulation rules Example: 11/6/2018

Recursive algorithms General idea: Divide and Conquer Divide a large problem into smaller ones By a constant ratio By a constant or some variable Solve each smaller one recursively or explicitly Combine the solutions of smaller ones to form a solution for the original problem Divide and Conquer 11/6/2018

How to analyze the time-efficiency of a recursive algorithm? Express the running time on input of size n as a function of the running time on smaller problems 11/6/2018

Analyzing merge sort T(n) MERGE-SORT A[1 . . n] Θ(1) 2T(n/2) f(n) MERGE-SORT A[1 . . n] If n = 1, done. Recursively sort A[ 1 . . n/2 ] and A[ n/2+1 . . n ] . “Merge” the 2 sorted lists Sloppiness: Should be T( n/2 ) + T( n/2 ) , but it turns out not to matter asymptotically. 11/6/2018

Analyzing merge sort T(n) = 2 T(n/2) + f(n) +Θ(1) Divide: Trivial. Conquer: Recursively sort 2 subarrays. Combine: Merge two sorted subarrays T(n) = 2 T(n/2) + f(n) +Θ(1) # subproblems Work dividing and Combining subproblem size What is the time for the base case? What is f(n)? What is the growth order of T(n)? Constant 11/6/2018

Solving recurrence Running time of many algorithms can be expressed in one of the following two recursive forms or Challenge: how to solve the recurrence to get a closed form, e.g. T(n) = Θ (n2) or T(n) = Θ(nlgn), or at least some bound such as T(n) = O(n2)? 11/6/2018

Solving recurrence Recurrence tree (iteration) method - Good for guessing an answer Substitution method - Generic method, rigid, but may be hard Master method - Easy to learn, useful in limited cases only - Some tricks may help in other cases 11/6/2018

The master method The master method applies to recurrences of the form T(n) = a T(n/b) + f (n) , where a ³ 1, b > 1, and f is asymptotically positive. Divide the problem into a subproblems, each of size n/b Conquer the subproblems by solving them recursively. Combine subproblem solutions Divide + combine takes f(n) time. 11/6/2018

e.g.: merge sort: T(n) = 2 T(n/2) + Θ(n) Master theorem T(n) = a T(n/b) + f (n) Key: compare f(n) with nlogba CASE 1: f (n) = O(nlogba – e)  T(n) = Q(nlogba) . CASE 2: f (n) = Q(nlogba)  T(n) = Q(nlogba log n) . CASE 3: f (n) = W(nlogba + e) and a f (n/b) £ c f (n)  T(n) = Q( f (n)) . e.g.: merge sort: T(n) = 2 T(n/2) + Θ(n) a = 2, b = 2  nlogba = n  CASE 2  T(n) = Θ(n log n) . 11/6/2018

Case 1 Compare f (n) with nlogba: f (n) = O(nlogba – e) for some constant e > 0. : f (n) grows polynomially slower than nlogba (by an ne factor). Solution: T(n) = Q(nlogba) i.e., aT(n/b) dominates e.g. T(n) = 2T(n/2) + 1 T(n) = 4 T(n/2) + n T(n) = 2T(n/2) + log n T(n) = 8T(n/2) + n2 11/6/2018

Case 3 Compare f (n) with nlogba: f (n) = W (nlogba + e) for some constant e > 0. : f (n) grows polynomially faster than nlogba (by an ne factor). Solution: T(n) = Q(f(n)) i.e., f(n) dominates e.g. T(n) = T(n/2) + n T(n) = 2 T(n/2) + n2 T(n) = 4T(n/2) + n3 T(n) = 8T(n/2) + n4 11/6/2018

Case 2 Compare f (n) with nlogba: f (n) = Q (nlogba). : f (n) and nlogba grow at similar rate. Solution: T(n) = Q(nlogba log n) e.g. T(n) = T(n/2) + 1 T(n) = 2 T(n/2) + n T(n) = 4T(n/2) + n2 T(n) = 8T(n/2) + n3 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. … Q(1) 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. h = log n dn/4 dn/4 dn/4 dn/4 … Q(1) 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. h = log n dn/4 dn/4 dn/4 dn/4 … Q(1) 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. h = log n dn/4 dn/4 dn/4 dn/4 … Q(1) 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. h = log n dn/4 dn/4 dn/4 dn/4 dn … … Q(1) 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. h = log n dn/4 dn/4 dn/4 dn/4 dn … … Q(1) #leaves = n Q(n) 11/6/2018

Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant. h = log n dn/4 dn/4 dn/4 dn/4 dn … … Q(1) #leaves = n Q(n) Total Q(n log n) 11/6/2018

Substitution method The most general method to solve a recurrence (prove O and  separately): Guess the form of the solution: (e.g. using recursion trees, or expansion) Verify by induction (inductive step). 11/6/2018

Proof by substitution Recurrence: T(n) = 2T(n/2) + n. Guess: T(n) = O(n log n). (eg. by recurrence tree method) To prove, have to show T(n) ≤ c n log n for some c > 0 and for all n > n0 Proof by induction: assume it is true for T(n/2), prove that it is also true for T(n). This means: Fact: T(n) = 2T(n/2) + n Assumption: T(n/2)≤ cn/2 log (n/2) Need to Prove: T(n)≤ c n log (n) 11/6/2018

Proof Fact: T(n) = 2T(n/2) + n Assumption: T(n/2)≤ cn/2 log (n/2) Need to Prove: T(n)≤ c n log (n) Proof: Substitute T(n/2) into the recurrence function => T(n) = 2 T(n/2) + n ≤ cn log (n/2) + n => T(n) ≤ c n log n - c n + n => T(n) ≤ c n log n (if we choose c ≥ 1). 11/6/2018

Proof by substitution Recurrence: T(n) = 2T(n/2) + n. Guess: T(n) = Ω(n log n). To prove, have to show T(n) ≥ c n log n for some c > 0 and for all n > n0 Proof by induction: assume it is true for T(n/2), prove that it is also true for T(n). This means: Fact: Assumption: Need to Prove: T(n) ≥ c n log (n) T(n) = 2T(n/2) + n T(n/2) ≥ cn/2 log (n/2) 11/6/2018

Proof Fact: T(n) = 2T(n/2) + n Assumption: T(n/2) ≥ cn/2 log (n/2) Need to Prove: T(n) ≥ c n log (n) Proof: Substitute T(n/2) into the recurrence function => T(n) = 2 T(n/2) + n ≥ cn log (n/2) + n => T(n) ≥ c n log n - c n + n => T(n) ≥ c n log n (if we choose c ≤ 1). 11/6/2018

Quick sort Quicksort an n-element array: Divide: Partition the array into two subarrays around a pivot x such that elements in lower subarray £ x £ elements in upper subarray. Conquer: Recursively sort the two subarrays. Combine: Trivial. £ x x ≥ x Key: Linear-time partitioning subroutine. 11/6/2018

Partition All the action takes place in the partition() function £ x x Rearranges the subarray in place End result: two subarrays All values in first subarray  all values in second Returns the index of the “pivot” element separating the two subarrays p q r £ x x ≥ x 11/6/2018

Partition Code What is the running time of partition()? Partition(A, p, r) x = A[p]; // pivot is the first element i = p; j = r + 1; while (TRUE) { repeat i++; until A[i] > x or i >= j; j--; until A[j] < x or j < i; if (i < j) Swap (A[i], A[j]); else break; } swap (A[p], A[j]); return j; What is the running time of partition()? partition() runs in O(n) time 11/6/2018

p r 6 10 5 8 13 3 2 11 x = 6 i j 6 10 5 8 13 3 2 11 i j 6 2 5 8 13 3 10 11 i j 6 2 5 8 13 3 10 11 i j 6 2 5 3 13 8 10 11 i j 6 2 5 3 13 8 10 11 j i p q r 3 2 5 6 13 8 10 11 11/6/2018

6 10 5 8 11 3 2 13 3 2 5 6 11 8 10 13 2 3 5 6 10 8 11 13 2 3 5 6 8 10 11 13 2 3 5 6 8 10 11 13 11/6/2018

Quicksort Runtimes Best case runtime Tbest(n)  O(n log n) Worst case runtime Tworst(n)  O(n2) Worse than mergesort? Why is it called quicksort then? Its average runtime Tavg(n)  O(n log n ) Better even, the expected runtime of randomized quicksort is O(n log n) 11/6/2018

Randomized quicksort Randomly choose an element as pivot Every time need to do a partition, throw a die to decide which element to use as the pivot Each element has 1/n probability to be selected Partition(A, p, r) d = random(); // a random number between 0 and 1 index = p + floor((r-p+1) * d); // p<=index<=r swap(A[p], A[index]); x = A[p]; i = p; j = r + 1; while (TRUE) { … } 11/6/2018

Running time of randomized quicksort T(0) + T(n–1) + dn if 0 : n–1 split, T(1) + T(n–2) + dn if 1 : n–2 split, M T(n–1) + T(0) + dn if n–1 : 0 split, T(n) = The expected running time is an average of all cases Expectation 11/6/2018

Heaps In practice, heaps are usually implemented as arrays: 16 14 10 8 7 9 3 2 4 1 16 14 10 8 7 9 3 2 4 1 11/6/2018

Heaps To represent a complete binary tree as an array: The root node is A[1] Node i is A[i] The parent of node i is A[i/2] (note: integer divide) The left child of node i is A[2i] The right child of node i is A[2i + 1] 16 14 10 8 7 9 3 2 4 1 A = 16 14 10 8 7 9 3 2 4 1 = 11/6/2018

The Heap Property Heaps also satisfy the heap property: A[Parent(i)]  A[i] for all nodes i > 1 In other words, the value of a node is at most the value of its parent The value of a node should be greater than or equal to both its left and right children And all of its descendents Where is the largest element in a heap stored? 11/6/2018

Heap Operations: Heapify() Heapify(A, i) { // precondition: subtrees rooted at l and r are heaps l = Left(i); r = Right(i); if (l <= heap_size(A) && A[l] > A[i]) largest = l; else largest = i; if (r <= heap_size(A) && A[r] > A[largest]) largest = r; if (largest != i) { Swap(A, i, largest); Heapify(A, largest); } } // postcondition: subtree rooted at i is a heap Among A[l], A[i], A[r], which one is largest? If violation, fix it. 11/6/2018

Heapify() Example 16 4 10 14 7 9 3 2 8 1 A = 16 4 10 14 7 9 3 2 8 1 11/6/2018

Heapify() Example 16 4 10 14 7 9 3 2 8 1 A = 16 4 10 14 7 9 3 2 8 1 11/6/2018

Heapify() Example 16 4 10 14 7 9 3 2 8 1 A = 16 4 10 14 7 9 3 2 8 1 11/6/2018

Heapify() Example 16 14 10 4 7 9 3 2 8 1 A = 16 14 10 4 7 9 3 2 8 1 11/6/2018

Heapify() Example 16 14 10 4 7 9 3 2 8 1 A = 16 14 10 4 7 9 3 2 8 1 11/6/2018

Heapify() Example 16 14 10 8 7 9 3 2 4 1 A = 16 14 10 8 7 9 3 2 4 1 11/6/2018

Heapify() Example 16 14 10 8 7 9 3 2 4 1 A = 16 14 10 8 7 9 3 2 4 1 11/6/2018

Analyzing Heapify(): Formal T(n)  T(2n/3) + (1) By case 2 of the Master Theorem, T(n) = O(lg n) Thus, Heapify() takes logarithmic time 11/6/2018

Heap Operations: BuildHeap() We can build a heap in a bottom-up manner by running Heapify() on successive subarrays Fact: for array of length n, all elements in range A[n/2 + 1 .. n] are heaps (Why?) So: Walk backwards through the array from n/2 to 1, calling Heapify() on each node. Order of processing guarantees that the children of node i are heaps when i is processed 11/6/2018

BuildHeap() // given an unsorted array A, make A a heap BuildHeap(A) { heap_size(A) = length(A); for (i = length[A]/2 downto 1) Heapify(A, i); } 11/6/2018

BuildHeap() Example Work through example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} 4 1 3 2 16 9 10 14 8 7 11/6/2018

4 1 3 2 16 9 10 14 8 7 11/6/2018

4 1 3 14 16 9 10 2 8 7 11/6/2018

4 1 10 14 16 9 3 2 8 7 11/6/2018

4 16 10 14 7 9 3 2 8 1 11/6/2018

16 14 10 8 7 9 3 2 4 1 11/6/2018

Analyzing BuildHeap(): Tight To Heapify() a subtree takes O(h) time where h is the height of the subtree h = O(lg m), m = # nodes in subtree The height of most subtrees is small Fact: an n-element heap has at most n/2h+1 nodes of height h CLR 7.3 uses this fact to prove that BuildHeap() takes O(n) time 11/6/2018

Heapsort Example Work through example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} 4 1 3 2 16 9 10 14 8 7 A = 4 1 3 2 16 9 10 14 8 7 11/6/2018

Heapsort Example First: build a heap 16 14 10 8 7 9 3 2 4 1 A = 16 14 11/6/2018

Heapsort Example Swap last and first 1 14 10 8 7 9 3 2 4 16 A = 1 14 11/6/2018

Heapsort Example Last element sorted 1 14 10 8 7 9 3 2 4 16 A = 1 14 11/6/2018

Heapsort Example Restore heap on remaining unsorted elements 14 8 10 4 7 9 3 2 1 16 Heapify A = 14 8 10 4 7 9 3 2 1 16 11/6/2018

Heapsort Example Repeat: swap new last and first 1 8 10 4 7 9 3 2 14 16 A = 1 8 10 4 7 9 3 2 14 16 11/6/2018

Heapsort Example Restore heap 10 8 9 4 7 1 3 2 14 16 A = 10 8 9 4 7 1 11/6/2018

Heapsort Example Repeat 9 8 3 4 7 1 2 10 14 16 A = 9 8 3 4 7 1 2 10 14 11/6/2018

Heapsort Example Repeat 8 7 3 4 2 1 9 10 14 16 A = 8 7 3 4 2 1 9 10 14 11/6/2018

Heapsort Example Repeat 1 2 3 4 7 8 9 10 14 16 A = 1 2 3 4 7 8 9 10 14 11/6/2018

Analyzing Heapsort The call to BuildHeap() takes O(n) time Each of the n - 1 calls to Heapify() takes O(lg n) time Thus the total time taken by HeapSort() = O(n) + (n - 1) O(lg n) = O(n) + O(n lg n) = O(n lg n) 11/6/2018

HeapExtractMax Example 16 14 10 8 7 9 3 2 4 1 A = 16 14 10 8 7 9 3 2 4 1 11/6/2018

HeapExtractMax Example Swap first and last, then remove last 1 14 10 8 7 9 3 2 4 16 A = 1 14 10 8 7 9 3 2 4 16 11/6/2018

HeapExtractMax Example Heapify 14 8 10 4 7 9 3 2 1 16 A = 14 8 10 4 7 9 3 2 1 16 11/6/2018

HeapChangeKey Example Increase key 16 14 10 8 7 9 3 2 4 1 A = 16 14 10 8 7 9 3 2 4 1 11/6/2018

HeapChangeKey Example Increase key 16 14 10 15 7 9 3 2 4 1 A = 16 14 10 15 7 9 3 2 4 1 11/6/2018

HeapChangeKey Example Increase key 16 15 10 14 7 9 3 2 4 1 A = 16 15 10 14 7 9 3 2 4 1 11/6/2018

HeapInsert Example HeapInsert(A, 17) 16 14 10 8 7 9 3 2 4 1 A = 16 14 11/6/2018

HeapInsert Example HeapInsert(A, 17) -∞ -∞ -∞ makes it a valid heap 16 14 10 8 7 9 3 2 4 1 -∞ -∞ makes it a valid heap A = 16 14 10 8 7 9 3 2 4 1 -∞ 11/6/2018

HeapInsert Example HeapInsert(A, 17) Now call changeKey 16 14 10 8 7 9 3 2 4 1 17 Now call changeKey A = 16 14 10 8 7 9 3 2 4 1 17 11/6/2018

HeapInsert Example HeapInsert(A, 17) 17 16 10 8 14 9 3 2 4 1 7 A = 17 11/6/2018

HeapExtractMax: Θ(log n) HeapChangeKey: Θ(log n) HeapInsert: Θ(log n) Heapify: Θ(log n) BuildHeap: Θ(n) HeapSort: Θ(nlog n) HeapMaximum: Θ(1) HeapExtractMax: Θ(log n) HeapChangeKey: Θ(log n) HeapInsert: Θ(log n) 11/6/2018

Counting sort for i  1 to k do C[i]  0 for j  1 to n 1. for i  1 to k do C[i]  0 for j  1 to n do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}| for i  2 to k do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key £ i}| for j  n downto 1 do B[C[A[ j]]]  A[ j] C[A[ j]]  C[A[ j]] – 1 Initialize 2. Count 3. Compute running sum 4. Re-arrange 11/6/2018

Counting sort A: 4 1 3 4 3 C: 1 2 2 B: C': 1 1 3 5 for i  2 to k 2 2 B: C': 1 1 3 5 3. for i  2 to k do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key £ i}| 11/6/2018

Loop 4: re-arrange A: 4 1 3 4 3 C: 1 1 3 5 B: 3 C': 1 1 3 5 2 3 4 5 1 2 3 4 A: 4 1 3 4 3 C: 1 1 3 5 B: 3 C': 1 1 3 5 4. for j  n downto 1 do B[C[A[ j]]]  A[ j] C[A[ j]]  C[A[ j]] – 1 11/6/2018

Analysis Q(k) Q(n) Q(k) Q(n) Q(n + k) 1. for i  1 to k do C[i]  0 2. for j  1 to n do C[A[ j]]  C[A[ j]] + 1 Q(n) 3. for i  2 to k do C[i]  C[i] + C[i–1] Q(k) 4. for j  n downto 1 do B[C[A[ j]]]  A[ j] C[A[ j]]  C[A[ j]] – 1 Q(n) Q(n + k) 11/6/2018

What other algorithms have this property? Stable sorting Counting sort is a stable sort: it preserves the input order among equal elements. A: 4 1 3 B: Why this is important? What other algorithms have this property? 11/6/2018

Radix sort Similar to sorting the address books Treat each digit as a key Start from the least significant bit Most significant Least significant 198099109123518183599 340199540380128115295 384700101594539614696 382408360201039258538 614386507628681328936 11/6/2018

Time complexity Sort each of the d digits by counting sort Total cost: d (n + k) k = 10 Total cost: Θ(dn) Partition the d digits into groups of 3 Total cost: (n+103)d/3 We work with binaries rather than decimals Partition a binary number into groups of r bits Total cost: (n+2r)d/r Choose r = log n Total cost: dn / log n Compare with dn log n Catch: faster than quicksort only when n is very large 11/6/2018

Randomized selection algorithm RAND-SELECT(A, p, q, i) ⊳ i th smallest of A[ p . . q] if p = q & i > 1 then error! r  RAND-PARTITION(A, p, q) k  r – p + 1 ⊳ k = rank(A[r]) if i = k then return A[ r] if i < k then return RAND-SELECT( A, p, r – 1, i ) else return RAND-SELECT( A, r + 1, q, i – k ) £ A[r] ³ A[r] r p q k 11/6/2018

Select the 6 – 4 = 2nd smallest recursively. Example Select the i = 6th smallest: 7 10 5 8 11 3 2 13 i = 6 pivot 3 2 5 7 11 8 10 13 Partition: k = 4 Select the 6 – 4 = 2nd smallest recursively. 11/6/2018

Complete example: select the 6th smallest element. i = 6 7 10 5 8 11 3 2 13 3 2 5 7 11 8 10 13 k = 4 i = 6 – 4 = 2 k = 3 10 8 11 13 i = 2 < k Note: here we always used first element as pivot to do the partition (instead of rand-partition). k = 2 8 10 i = 2 = k 10 11/6/2018

Intuition for analysis (All our analyses today assume that all elements are distinct.) Lucky: T(n) = T(9n/10) + Q(n) = Q(n) CASE 3 Unlucky: T(n) = T(n – 1) + Q(n) = Q(n2) arithmetic series Worse than sorting! 11/6/2018

Running time of randomized selection T(max(0, n–1)) + n if 0 : n–1 split, T(max(1, n–2)) + n if 1 : n–2 split, M T(max(n–1, 0)) + n if n–1 : 0 split, T(n) ≤ For upper bound, assume ith element always falls in larger side of partition The expected running time is an average of all cases Expectation 11/6/2018

Worst-case linear-time selection if i = k then return x elseif i < k then recursively SELECT the i th smallest element in the lower part else recursively SELECT the (i–k)th smallest element in the upper part SELECT(i, n) Divide the n elements into groups of 5. Find the median of each 5-element group by rote. Recursively SELECT the median x of the ën/5û group medians to be the pivot. Partition around the pivot x. Let k = rank(x). Same as RAND-SELECT 11/6/2018

Developing the recurrence T(n) if i = k then return x elseif i < k then recursively SELECT the i th smallest element in the lower part else recursively SELECT the (i–k)th smallest element in the upper part SELECT(i, n) Divide the n elements into groups of 5. Find the median of each 5-element group by rote. Recursively SELECT the median x of the ën/5û group medians to be the pivot. Partition around the pivot x. Let k = rank(x). Q(n) T(n/5) Q(n) T(7n/10+3) 11/6/2018

Solving the recurrence Assumption: T(k) £ ck for all k < n if n ≥ 60 if c ≥ 20 and n ≥ 60 11/6/2018

Hash tables Problem: collision |U| >> K & |U| >> m U (universe of keys) h(k1) k1 h(k4) k4 K (actual keys) k5 collision h(k2) = h(k5) k2 h(k3) k3 m - 1 Problem: collision 11/6/2018

Chaining Chaining puts elements that hash to the same slot in a linked list: T —— U (universe of keys) k1 k4 —— —— k1 —— k4 K (actual keys) k5 —— k7 k5 k2 k7 —— —— k3 k2 k8 k3 —— k6 k8 k6 —— —— 11/6/2018

Hashing with Chaining Chained-Hash-Insert (T, x) Insert x at the head of list T[h(key[x])]. Worst-case complexity – O(1). Chained-Hash-Delete (T, x) Delete x from the list T[h(key[x])]. Worst-case complexity – proportional to length of list with singly-linked lists. O(1) with doubly-linked lists. Chained-Hash-Search (T, k) Search an element with key k in list T[h(k)]. Worst-case complexity – proportional to length of list. 11/6/2018

Analysis of Chaining Assume simple uniform hashing: each key in table is equally likely to be hashed to any slot Given n keys and m slots in the table, the load factor  = n/m = average # keys per slot Average cost of an unsuccessful search for a key is (1+) (Theorem 11.1) Average cost of a successful search is (2 + /2) = (1 + ) (Theorem 11.2) If the number of keys n is proportional to the number of slots in the table,  = n/m = O(1) The expected cost of searching is constant if  is constant 11/6/2018

Hash Functions: The Division Method h(k) = k mod m In words: hash k into a table with m slots using the slot given by the remainder of k divided by m Example: m = 31 and k = 78 => h(k) = 16. Advantage: fast Disadvantage: value of m is critical Bad if keys bear relation to m Or if hash does not depend on all bits of k Pick m = prime number not too close to power of 2 (or 10) 11/6/2018

Hash Functions: The Multiplication Method For a constant A, 0 < A < 1: h(k) = m (kA mod 1) =  m (kA - kA)  Advantage: Value of m is not critical Disadvantage: relatively slower Choose m = 2P, for easier implementation Choose A not too close to 0 or 1 Knuth: Good choice for A = (5 - 1)/2 Example: m = 1024, k = 123, A  0.6180339887… h(k) = 1024(123 · 0.6180339887 mod 1) = 1024 · 0.018169...  = 18. Fractional part of kA 11/6/2018

A Universal Hash Function Choose a prime number p that is larger than all possible keys Choose table size m ≥ n Randomly choose two integers a, b, such that 1  a  p -1, and 0  b  p -1 ha,b(k) = ((ak+b) mod p) mod m Example: p = 17, m = 6 h3,4 (8) = ((3*8 + 4) % 17) % 6 = 11 % 6 = 5 With a random pair of parameters a, b, the chance of a collision between x and y is at most 1/m Expected search time for any input is (1) 11/6/2018

Elements of dynamic programming Optimal sub-structures Optimal solutions to the original problem contains optimal solutions to sub-problems Overlapping sub-problems Some sub-problems appear in many solutions 11/6/2018

Two steps to dynamic programming Formulate the solution as a recurrence relation of solutions to subproblems. Specify an order to solve the subproblems so you always have what you need. 11/6/2018

Optimal subpaths Claim: if a path startgoal is optimal, any sub-path, startx, or xgoal, or xy, where x, y is on the optimal path, is also the shortest. Proof by contradiction If the subpath between x and y is not the shortest, we can replace it with the shorter one, which will reduce the total length of the new path => the optimal path from start to goal is not the shortest => contradiction! Hence, the subpath xy must be the shortest among all paths from x to y start goal x y a b c b’ a + b + c is shortest b’ < b a + b’ + c < a + b + c 11/6/2018

Dynamic programming illustration 3 9 1 2 3 12 13 15 5 3 3 3 3 3 2 5 2 5 6 8 13 15 2 3 3 9 3 2 4 2 3 7 9 11 13 16 6 2 3 7 4 3 6 3 3 13 11 14 17 20 4 6 3 1 3 1 2 3 2 17 17 17 18 20 G F(i-1, j) + dist(i-1, j, i, j) F(i, j) = min F(i, j-1) + dist(i, j-1, i, j) 11/6/2018

Trace back 3 9 1 2 3 12 13 15 5 3 3 3 3 3 2 5 2 5 6 8 13 15 2 3 3 9 3 2 4 2 3 7 9 11 13 16 6 2 3 7 4 3 6 3 3 13 11 14 17 20 4 6 3 1 3 1 2 3 2 17 17 17 18 20 11/6/2018

Longest Common Subsequence Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both. “a” not “the” x: A B C D y: BCBA = LCS(x, y) functional notation, but not a function 11/6/2018

Optimal substructure Notice that the LCS problem has optimal substructure: parts of the final solution are solutions of subproblems. If z = LCS(x, y), then any prefix of z is an LCS of a prefix of x and a prefix of y. Subproblems: “find LCS of pairs of prefixes of x and y” i m x z n y j 11/6/2018

Finding length of LCS m x n y Let c[i, j] be the length of LCS(x[1..i], y[1..j]) => c[m, n] is the length of LCS(x, y) If x[m] = y[n] c[m, n] = c[m-1, n-1] + 1 If x[m] != y[n] c[m, n] = max { c[m-1, n], c[m, n-1] } 11/6/2018

DP Algorithm c[i–1, j–1] + 1 if x[i] = y[j], Key: find out the correct order to solve the sub-problems Total number of sub-problems: m * n c[i, j] = c[i–1, j–1] + 1 if x[i] = y[j], max{c[i–1, j], c[i, j–1]} otherwise. j n C(i, j) i m 11/6/2018

LCS Example (0) ABCB BDCAB X = ABCB; m = |X| = 4 j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 B 2 3 C 4 B X = ABCB; m = |X| = 4 Y = BDCAB; n = |Y| = 5 Allocate array c[5,6] 11/6/2018

LCS Example (1) ABCB BDCAB for i = 1 to m c[i,0] = 0 j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 B 2 3 C 4 B for i = 1 to m c[i,0] = 0 for j = 1 to n c[0,j] = 0 11/6/2018

LCS Example (2) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 B 2 A 1 B 2 3 C 4 B if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] ) 11/6/2018

LCS Example (3) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 B 2 A 1 B 2 3 C 4 B if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] ) 11/6/2018

LCS Example (4) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 B A 1 1 B 2 3 C 4 B if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] ) 11/6/2018

LCS Example (5) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1 A 1 1 1 B 2 3 C 4 B if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] ) 11/6/2018

LCS Example (6) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1 A 1 1 1 B 2 1 3 C 4 B if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] ) 11/6/2018

LCS Example (7) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1 A 1 1 1 B 2 1 1 1 1 3 C 4 B if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] ) 11/6/2018

LCS Example (8) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1 A 1 1 1 B 2 1 1 1 1 2 3 C 4 B if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] ) 11/6/2018

LCS Example (14) 3 ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2 if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] ) 11/6/2018

LCS Algorithm Running Time LCS algorithm calculates the values of each entry of the array c[m,n] So what is the running time? O(m*n) since each c[i,j] is calculated in constant time, and there are m*n elements in the array 11/6/2018

How to find actual LCS For example, here The algorithm just found the length of LCS, but not LCS itself. How to find the actual LCS? For each c[i,j] we know how it was acquired: A match happens only when the first equation is taken So we can start from c[m,n] and go backwards, remember x[i] whenever c[i,j] = c[i-1, j-1]+1. 2 2 For example, here c[i,j] = c[i-1,j-1] +1 = 2+1=3 2 3 11/6/2018

Finding LCS 3 Time for trace back: O(m+n). j 0 1 2 3 4 5 i Y[j] B D C X[i] A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2 Time for trace back: O(m+n). 11/6/2018

Finding LCS (2) 3 LCS (reversed order): B C B B C B j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2 LCS (reversed order): B C B B C B (this string turned out to be a palindrome) LCS (straight order): 11/6/2018

LCS as a longest path problem D C A B A 1 B 1 1 1 C B 1 1 11/6/2018

LCS as a longest path problem D C A B A 1 1 1 1 1 B 1 1 1 1 2 1 C 1 1 2 2 2 B 1 1 1 1 1 2 3 11/6/2018

Restaurant location problem 1 You work in the fast food business Your company plans to open up new restaurants in Texas along I-35 Towns along the highway called t1, t2, …, tn Restaurants at ti has estimated annual profit pi No two restaurants can be located within 10 miles of each other due to some regulation Your boss wants to maximize the total profit You want a big bonus 10 mile 11/6/2018

A DP algorithm Suppose you’ve already found the optimal solution It will either include tn or not include tn Case 1: tn not included in optimal solution Best solution same as best solution for t1 , …, tn-1 Case 2: tn included in optimal solution Best solution is pn + best solution for t1 , …, tj , where j < n is the largest index so that dist(tj, tn) ≥ 10 11/6/2018

Recurrence formulation Let S(i) be the total profit of the optimal solution when the first i towns are considered (not necessarily selected) S(n) is the optimal solution to the complete problem S(n-1) S(j) + pn j < n & dist (tj, tn) ≥ 10 S(n) = max S(i-1) S(j) + pi j < i & dist (tj, ti) ≥ 10 S(i) = max Generalize Number of sub-problems: n. Boundary condition: S(0) = 0. Dependency: i i-1 j S 11/6/2018

Example S(i-1) S(j) + pi j < i & dist (tj, ti) ≥ 10 S(i) = max Distance (mi) 100 5 2 2 6 6 3 6 10 7 dummy 7 3 4 12 Profit (100k) 6 7 9 8 3 3 2 4 12 5 S(i) 6 7 9 9 10 12 12 14 26 26 Optimal: 26 S(i-1) S(j) + pi j < i & dist (tj, ti) ≥ 10 S(i) = max Natural greedy 1: 6 + 3 + 4 + 12 = 25 Natural greedy 2: 12 + 9 + 3 = 24 11/6/2018

Complexity Time: (nk), where k is the maximum number of towns that are within 10 miles to the left of any town In the worst case, (n2) Can be improved to (n) with some preprocessing tricks Memory: Θ(n) 11/6/2018

Knapsack problem Each item has a value and a weight Objective: maximize value Constraint: knapsack has a weight limitation Three versions: 0-1 knapsack problem: take each item or leave it Fractional knapsack problem: items are divisible Unbounded knapsack problem: unlimited supplies of each item. Which one is easiest to solve? We study the 0-1 problem today. 11/6/2018

Formal definition (0-1 problem) Knapsack has weight limit W Items labeled 1, 2, …, n (arbitrarily) Items have weights w1, w2, …, wn Assume all weights are integers For practical reason, only consider wi < W Items have values v1, v2, …, vn Objective: find a subset of items, S, such that iS wi  W and iS vi is maximal among all such (feasible) subsets 11/6/2018

A DP algorithm Suppose you’ve find the optimal solution S Case 1: item n is included Case 2: item n is not included Total weight limit: W Total weight limit: W wn wn Find an optimal solution using items 1, 2, …, n-1 with weight limit W - wn Find an optimal solution using items 1, 2, …, n-1 with weight limit W 11/6/2018

Recursive formulation Let V[i, w] be the optimal total value when items 1, 2, …, i are considered for a knapsack with weight limit w => V[n, W] is the optimal solution V[n, W] = max V[n-1, W-wn] + vn V[n-1, W] Generalize V[i, w] = max V[i-1, w-wi] + vi item i is taken V[i-1, w] item i not taken V[i-1, w] if wi > w item i not taken Boundary condition: V[i, 0] = 0, V[0, w] = 0. Number of sub-problems = ? 11/6/2018

Example n = 6 (# of items) W = 10 (weight limit) Items (weight, value): 2 2 4 3 3 3 5 6 2 4 6 9 11/6/2018

w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 2 2 4 3 wi 3 3 3 V[i-1, w-wi] V[i-1, w] 4 5 5 6 6 V[i, w] 5 2 4 6 6 9 V[i-1, w-wi] + vi item i is taken V[i-1, w] item i not taken max V[i, w] = V[i-1, w] if wi > w item i not taken 11/6/2018

w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 4 3 5 6 9 2 2 2 2 2 2 2 2 2 3 5 2 2 3 5 5 5 5 6 8 3 5 2 3 5 6 8 6 8 9 11 2 3 3 6 9 4 6 7 10 12 13 4 7 10 13 15 9 4 4 6 7 10 13 V[i-1, w-wi] + vi item i is taken V[i-1, w] item i not taken max V[i-1, w] if wi > w item i not taken V[i, w] = 11/6/2018

w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 4 3 5 6 9 2 2 2 2 2 2 2 2 2 3 5 2 2 3 5 5 5 5 6 8 3 5 2 3 5 6 8 6 8 9 11 2 3 3 6 9 4 7 10 12 13 4 6 7 10 13 9 4 4 6 7 10 13 15 Optimal value: 15 Item: 6, 5, 1 Weight: 6 + 2 + 2 = 10 Value: 9 + 4 + 2 = 15 11/6/2018

Time complexity Θ (nW) Polynomial? Pseudo-polynomial Works well if W is small Consider following items (weight, value): (10, 5), (15, 6), (20, 5), (18, 6) Weight limit 35 Optimal solution: item 2, 4 (value = 12). Iterate: 2^4 = 16 subsets Dynamic programming: fill up a 4 x 35 = 140 table entries What’s the problem? Many entries are unused: no such weight combination Top-down may be better 11/6/2018

Longest increasing subsequence Given a sequence of numbers 1 2 5 3 2 9 4 9 3 5 6 8 Find a longest subsequence that is non-decreasing E.g. 1 2 5 9 It has to be a subsequence of the original list It has to in sorted order => It is a subsequence of the sorted list Original list: 1 2 5 3 2 9 4 9 3 5 6 8 LCS: Sorted: 1 2 2 3 3 4 5 5 6 8 9 9 1 2 3 4 5 6 8 11/6/2018

Events scheduling problem Time A list of events to schedule (or shows to see) ei has start time si and finishing time fi Indexed such that fi < fj if i < j Each event has a value vi Schedule to make the largest value You can attend only one event at any time Very similar to the new restaurant location problem Sort events according to their finish time Consider: if the last event is included or not 11/6/2018

Events scheduling problem f9 s8 f8 s7 f7 e8 e3 e4 e5 e7 e9 e1 e2 Time V(i) is the optimal value that can be achieved when the first i events are considered V(n) = V(n-1) en not selected max { V(j) + vn en selected j < n and fj < sn 11/6/2018

Coin change problem Given some denomination of coins (e.g., 2, 5, 7, 10), decide if it is possible to make change for a value (e.g, 13), or minimize the number of coins Version 1: Unlimited number of coins for each denomination Unbounded knapsack problem Version 2: Use each denomination at most once 0-1 Knapsack problem 11/6/2018

Use DP algorithm to solve new problems Directly map a new problem to a known problem Modify an algorithm for a similar task Design your own Think about the problem recursively Optimal solution to a larger problem can be computed from the optimal solution of one or more subproblems These sub-problems can be solved in certain manageable order Works nicely for naturally ordered data such as strings, trees, some special graphs Trickier for general graphs The text book has some very good exercises. 11/6/2018

Unit-profit restaurant location problem Now the objective is to maximize the number of new restaurants (subject to the distance constraint) In other words, we assume that each restaurant makes the same profit, no matter where it is opened 10 mile 11/6/2018

A DP Algorithm Exactly as before, but pi = 1 for all i S(i-1) S(j) + pi j < i & dist (tj, ti) ≥ 10 S(i) = max S(i-1) S(j) + 1 j < i & dist (tj, ti) ≥ 10 S(i) = max 11/6/2018

Greedy algorithm for restaurant location problem select t1 d = 0; for (i = 2 to n) d = d + dist(ti, ti-1); if (d >= min_dist) select ti end 5 2 2 6 6 3 6 10 7 d 5 7 9 15 6 9 15 10 7 11/6/2018

Complexity Time: Θ(n) Memory: Θ(n) to store the input Θ(1) for greedy selection 11/6/2018

Optimal substructure Claim 1: if A = [m1, m2, …, mk] is the optimal solution to the restaurant location problem for a set of towns [t1, …, tn] m1 < m2 < … < mk are indices of the selected towns Then B = [m2, m3, …, mk] is the optimal solution to the sub-problem [tj, …, tn], where tj is the first town that are at least 10 miles to the right of tm1 Proof by contradiction: suppose B is not the optimal solution to the sub-problem, which means there is a better solution B’ to the sub-problem A’ = mi || B’ gives a better solution than A = mi || B => A is not optimal => contradiction => B is optimal B m1 A m2 mk m1 B’ (imaginary) A’ 11/6/2018

Greedy choice property Claim 2: for the uniform-profit restaurant location problem, there is an optimal solution that chooses t1 Proof by contradiction: suppose that no optimal solution can be obtained by choosing t1 Say the first town chosen by the optimal solution S is ti, i > 1 Replace ti with t1 will not violate the distance constraint, and the total profit remains the same => S’ is an optimal solution Contradiction Therefore claim 2 is valid S S’ 11/6/2018

Fractional knapsack problem Each item has a value and a weight Objective: maximize value Constraint: knapsack has a weight limitation 0-1 knapsack problem: take each item or leave it Fractional knapsack problem: items are divisible Unbounded knapsack problem: unlimited supplies of each item. Which one is easiest to solve? We can solve the fractional knapsack problem using greedy algorithm 11/6/2018

Greedy algorithm for fractional knapsack problem Compute value/weight ratio for each item Sort items by their value/weight ratio into decreasing order Call the remaining item with the highest ratio the most valuable item (MVI) Iteratively: If the weight limit can not be reached by adding MVI Select MVI Otherwise select MVI partially until weight limit 11/6/2018

Example Weight limit: 10 9 6 4 2 5 3 1 Value ($) Weight (LB) item 1.5 1.2 1 0.75 $ / LB 11/6/2018

Example Weight limit: 10 Take item 5 Take item 6 Take 2 LB of item 4 Weight (LB) Value ($) $ / LB 5 2 4 6 9 1.5 1.2 1 3 0.75 11/6/2018

Why is greedy algorithm for fractional knapsack problem valid? Claim: the optimal solution must contain the MVI as much as possible (either up to the weight limit or until MVI is exhausted) Proof by contradiction: suppose that the optimal solution does not use all available MVI (i.e., there is still w (w < W) pounds of MVI left while we choose other items) We can replace w pounds of less valuable items by MVI The total weight is the same, but with value higher than the “optimal” Contradiction w w 11/6/2018

Graphs 1 2 4 3 A graph G = (V, E) V = set of vertices E = set of edges = subset of V  V Thus |E| = O(|V|2) 1 Vertices: {1, 2, 3, 4} Edges: {(1, 2), (2, 3), (1, 3), (4, 3)} 2 4 3 11/6/2018

Graphs: Adjacency Matrix Example: A 1 2 3 4 1 2 4 3 How much storage does the adjacency matrix require? A: O(V2) 11/6/2018

Graphs: Adjacency List Adjacency list: for each vertex v  V, store a list of vertices adjacent to v Example: Adj[1] = {2,3} Adj[2] = {3} Adj[3] = {} Adj[4] = {3} Variation: can also keep a list of edges coming into vertex 1 2 4 3 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Kruskal’s algorithm: example c-d: 3 b-f: 5 b-a: 6 f-e: 7 b-d: 8 f-g: 9 d-e: 10 a-f: 12 b-c: 14 e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Time complexity Depending on implementation Pseudocode: sort all edges according to weights T = {}. tree(v) = v for all v. for each edge (u, v) if tree(u) != tree(v) T = T U (u, v); union (tree(u), tree(v)) Θ(m log m) = Θ(m log n) m edges Avg time spent per edge Naïve: Θ (n) Better: Θ (log n) using set union Overall time complexity Naïve: Θ(nm) Better implementation: Θ(m log n) 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d a b c d e f g h ∞ ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ChangeKey c b a d e f g h 0 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ExctractMin h b a d e f g ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ChangeKey d b a h e f g 3 14 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ExctractMin b g a h e f 14 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d Changekey b e a h g f 8 10 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ExtractMin e f a h g 10 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d Changekey f e a h g 5 10 6 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ExtractMin a e g h 6 10 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d Changekey a e g h 6 7 9 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ExtractMin e h g 7 ∞ 9 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ExtractMin g h 9 ∞ 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d Changekey g h 9 15 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d ExtractMin h 15 11/6/2018

Prim’s algorithm: example 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d 11/6/2018

Complete Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; T = {}; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) T = T U (u, v); ChangeKey(v, w(u,v)); Overall running time: Θ(m log n) Cost per ChangeKey n vertices Θ(n) times Θ(n2) times? Θ(m) times How often is ExtractMin() called? How often is ChangeKey() called? 11/6/2018

Summary Kruskal’s algorithm Prim’s algorithm Θ(m log n) Possibly Θ(m + n log n) with counting sort Prim’s algorithm With priority queue : Θ(m log n) Assume graph represented by adj list With distance array : Θ(n^2) Adj list or adj matrix For sparse graphs priority queue wins For dense graphs distance array may be better 11/6/2018

b h f e d a i c g a b c d e f g h i ∞ 14 7 5 Dijkstra’s algorithm 9 14 6 9 1 5 3 5 e 6 d 8 a i 4 7 2 1 c 7 g a b c d e f g h i ∞ 14 7 5 Dijkstra’s algorithm 11/6/2018

b h f e d a i c g a b c d e f g h i 11 7 5 ∞ Dijkstra’s algorithm 9 14 6 9 1 5 3 5 e 6 d 8 a i 4 7 11 2 1 c 7 g a b c d e f g h i 11 7 5 ∞ Dijkstra’s algorithm 11/6/2018

b h f e d a i c g a b c d e f g h i 9 11 7 5 ∞ Dijkstra’s algorithm 9 14 11 b 9 14 h 7 f 7 6 9 1 5 3 5 e 6 d 8 a i 4 11 7 2 1 9 c 7 g a b c d e f g h i 9 11 7 5 ∞ Dijkstra’s algorithm 11/6/2018

14 11 b 9 14 h 12 7 f 7 6 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g a b c d e f g h i 9 11 7 5 12 ∞ 17 Dijkstra’s algorithm 11/6/2018

14 11 b 9 20 14 h 12 7 f 7 6 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g a b c d e f g h i 9 11 7 5 12 ∞ 20 17 Dijkstra’s algorithm 11/6/2018

14 11 b 9 20 14 h 12 19 7 f 7 6 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g a b c d e f g h i 9 11 7 5 12 ∞ 19 17 Dijkstra’s algorithm 11/6/2018

14 11 b 9 20 14 h 12 19 7 f 7 6 18 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g 18 a b c d e f g h i 9 11 7 5 12 18 17 Dijkstra’s algorithm 11/6/2018

14 11 b 9 20 14 h 12 19 7 f 7 6 18 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g 18 a b c d e f g h i 9 11 7 5 12 18 17 Dijkstra’s algorithm 11/6/2018

14 11 b 9 20 14 h 12 19 7 f 7 6 18 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g 18 a b c d e f g h i 9 11 7 5 12 18 17 Dijkstra’s algorithm 11/6/2018

Prim’s Algorithm Overall running time: Θ(m log n) MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; T = {}; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) T = T U (u, v); ChangeKey(v, w(u,v)); Overall running time: Θ(m log n) Cost per ChangeKey 11/6/2018

Running time of Dijkstra’s algorithm is the same as Prim’s algorithm Dijkstra(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; T = {}; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and key[u]+w(u,v) < key[v]) T = T U (u, v); ChangeKey(v, key[u]+w(u,v)); Overall running time: Θ(m log n) Cost per ChangeKey Running time of Dijkstra’s algorithm is the same as Prim’s algorithm 11/6/2018

Good luck with your final! 11/6/2018