1. CS 3343: Analysis of Algorithms
Review for final
2/21/2019

2. Final Exam Closed book exam Coverage: the whole semester
Cheat sheet: you are allowed one letter-size sheet, both sides
Monday, May 5, 9:45 – 12:15pm
Basic calculator (no graphing) allowed
No cell phones!
2/21/2019

3. Final Exam: Study Tips Study tips: Re-make your previous cheat sheets
Study each lecture
Study the homework and homework solutions
Study the midterm exams
Re-make your previous cheat sheets
2/21/2019

4. Topics covered (1) By reversed chronological order: Graph algorithms
Representations
MST (Prim’s, Kruskal’s)
Shortest path (Dijkstra’s)
Running time analysis with different implementations
Greedy algorithm
Unit-profit restaurant location problem
Fractional knapsack problem
Prim’s and Kruskal’s are also examples of greedy algorithms
How to show that certain greedy choices are optimal
2/21/2019

5. Topics covered (2) Dynamic programming Binary heap and priority queue
LCS
Restaurant location problem
Shortest path problem on a grid
Other problems
How to define recurrence solution, and use dynamic programming to solve it
Binary heap and priority queue
Heapify, buildheap, insert, exatractMax, changeKey
Running time
2/21/2019

6. Topics covered (3) Order statistics Sorting algorithms Rand-Select
Worst-case Linear-time selection
Running time analysis
Sorting algorithms
Insertion sort
Merge sort
Quick sort
Heap sort
Linear time sorting: counting sort, radix sort
Stability of sorting algorithms
Worst-case and expected running time analysis
Memory requirement of sorting algorithms
2/21/2019

7. Topics covered (4) Analysis Analyzing non-recursive algorithms
Order of growth
Asymptotic notation, basic definition
Limit method
L’ Hopital’s rule
Stirling’s formula
Best case, worst case, average case
Analyzing non-recursive algorithms
Arithmetic series
Geometric series
Analyzing recursive algorithms
Defining recurrence
Solving recurrence
Recursion tree (iteration) method
Substitution method
Master theorem
2/21/2019

8. Review for finals In chronological order
Only the more important concepts
Very likely to appear in your final
Does not mean to be exclusive
2/21/2019

9. Asymptotic notations O: Big-Oh Ω: Big-Omega Θ: Theta o: Small-oh
ω: Small-omega
Intuitively:
O is like 
o is like <
 is like 
 is like >
 is like =
2/21/2019

10. Big-Oh Math: Engineering:
O(g(n)) = {f(n):  positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n)  n>n0}
Or: lim n→∞ g(n)/f(n) > 0 (if the limit exists.)
Engineering:
g(n) grows at least as faster as f(n)
g(n) is an asymptotic upper bound of f(n)
Intuitively it is like f(n) ≤ g(n)
2/21/2019

11. Big-Oh Claim: f(n) = 3n2 + 10n + 5  O(n2) Proof:
3n2 + 10n + 5  3n2 + 10n2 + 5n2 when n >  18 n2 when n > 1
Therefore,
Let c = 18 and n0 = 1
We have f(n)  c n2,  n > n0
By definition, f(n)  O(n2)
2/21/2019

12. Big-Omega Math: Engineering:
Ω(g(n)) = {f(n):  positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n)  n>n0}
Or: lim n→∞ f(n)/g(n) > 0 (if the limit exists.)
Engineering:
f(n) grows at least as faster as g(n)
g(n) is an asymptotic lower bound of f(n)
Intuitively it is like g(n) ≤ f(n)
2/21/2019

13. Big-Omega f(n) = n2 / 10 = Ω(n) Proof: f(n) = n2 / 10, g(n) = n
g(n) = n ≤ n2 / 10 = f(n) when n > 10
Therefore, c = 1 and n0 = 10
2/21/2019

14. Theta Math: Engineering: Θ(1) means constant time.
Θ(g(n)) = {f(n):  positive constants c1, c2, and n0 such that c1 g(n)  f(n)  c2 g(n)  n  n0  n>n0}
Or: lim n→∞ f(n)/g(n) = c > 0 and c < ∞
Or: f(n) = O(g(n)) and f(n) = Ω(g(n))
Engineering:
f(n) grows in the same order as g(n)
g(n) is an asymptotic tight bound of f(n)
Intuitively it is like f(n) = g(n)
Θ(1) means constant time.
2/21/2019

15. Theta Claim: f(n) = 2n2 + n = Θ (n2) Proof:
We just need to find three constants c1, c2, and n0 such that
c1n2 ≤ 2n2+n ≤ c2n2 for all n > n0
A simple solution is c1 = 2, c2 = 3, and n0 = 1
2/21/2019

16. Using limits to compare orders of growth
lim f(n) / g(n) = c > 0 ∞
f(n)  o(g(n))
f(n)  O(g(n))
f(n)  Θ (g(n))
n→∞
f(n)  Ω(g(n))
f(n)  ω (g(n))
2/21/2019

17. Therefore, 2n  o(3n), and 3n  ω(2n)
Compare 2n and 3n
lim 2n / 3n = lim(2/3)n = 0
Therefore, 2n  o(3n), and 3n  ω(2n)
n→∞
n→∞
2/21/2019

18. L’ Hopital’s rule lim f(n) / g(n) = lim f(n)’ / g(n)’ n→∞ n→∞
If both lim f(n) and lim g(n) goes to ∞
n→∞
n→∞
2/21/2019

19. ∞ Compare n0.5 and log n lim n0.5 / log n = ? (n0.5)’ = 0.5 n-0.5
lim (n-0.5 / 1/n) = lim(n0.5) =
Therefore, log n  o(n0.5)
n→∞ ∞
2/21/2019

20. Stirling’s formula
(constant)
2/21/2019

21. Compare 2n and n!
Therefore, 2n = o(n!)
2/21/2019

22. More advanced dominance ranking
2/21/2019

23. General plan for analyzing time efficiency of a non-recursive algorithm
Decide parameter (input size)
Identify most executed line (basic operation)
worst-case = average-case?
T(n) = i ti
T(n) = Θ (f(n))
2/21/2019

24. Analysis of insertion Sort
Statement cost time__
InsertionSort(A, n) {
for j = 2 to n { c1 n
key = A[j] c2 (n-1)
i = j - 1; c3 (n-1)
while (i > 0) and (A[i] > key) { c4 S
A[i+1] = A[i] c5 (S-(n-1))
i = i c6 (S-(n-1)) }
A[i+1] = key c7 (n-1) } }
2/21/2019

25. Inner loop stops when A[i] <= key, or i = 0
Best case
Inner loop stops when A[i] <= key, or i = 0 1 i j
Key
sorted
Array already sorted
2/21/2019

26. Inner loop stops when A[i] <= key
Worst case
Inner loop stops when A[i] <= key 1 i j
Key
sorted
Array originally in reverse order
2/21/2019

27. Inner loop stops when A[i] <= key
Average case
Inner loop stops when A[i] <= key 1 i j
Key
sorted
Array in random order
2/21/2019

28. Find the order of growth for sums
How to find out the actual order of growth?
Remember some formulas
Learn how to guess and prove
2/21/2019

29. Closed form, or explicit formula
Arithmetic series
An arithmetic series is a sequence of numbers such that the difference of any two successive members of the sequence is a constant.
e.g.: 1, 2, 3, 4, 5
or 10, 12, 14, 16, 18, 20
In general:
Recursive definition
Closed form, or explicit formula
Or:
2/21/2019

30. Sum of arithmetic series
If a1, a2, …, an is an arithmetic series, then
2/21/2019

31. Closed form, or explicit formula
Geometric series
A geometric series is a sequence of numbers such that the ratio between any two successive members of the sequence is a constant.
e.g.: 1, 2, 4, 8, 16, 32
or 10, 20, 40, 80, 160
or 1, ½, ¼, 1/8, 1/16
In general:
Recursive definition
Closed form, or explicit formula
Or:
2/21/2019

32. Sum of geometric series
if r < 1
if r > 1
if r = 1
2/21/2019

33. Important formulas
2/21/2019

34. Sum manipulation rules
Example:
2/21/2019

35. Recursive algorithms General idea: Divide and Conquer
Divide a large problem into smaller ones
By a constant ratio
By a constant or some variable
Solve each smaller one recursively or explicitly
Combine the solutions of smaller ones to form a solution for the original problem
Divide and Conquer
2/21/2019

36. How to analyze the time-efficiency of a recursive algorithm?
Express the running time on input of size n as a function of the running time on smaller problems
2/21/2019

37. Analyzing merge sort T(n) MERGE-SORT A[1 . . n] Θ(1) 2T(n/2)
f(n)
MERGE-SORT A[1 . . n]
If n = 1, done.
Recursively sort A[ n/2 ] and A[ n/2 n ] .
“Merge” the 2 sorted lists
Sloppiness: Should be T( n/2 ) + T( n/2 ) , but it turns out not to matter asymptotically.
2/21/2019

38. Analyzing merge sort T(n) = 2 T(n/2) + f(n) +Θ(1) Divide: Trivial.
Conquer: Recursively sort 2 subarrays.
Combine: Merge two sorted subarrays
T(n) = 2 T(n/2) + f(n) +Θ(1)
# subproblems
Work dividing and
Combining
subproblem size
What is the time for the base case?
What is f(n)?
What is the growth order of T(n)?
Constant
2/21/2019

39. Solving recurrence
Running time of many algorithms can be expressed in one of the following two recursive forms or
Challenge: how to solve the recurrence to get a closed form, e.g. T(n) = Θ (n2) or T(n) = Θ(nlgn), or at least some bound such as T(n) = O(n2)?
2/21/2019

40. Solving recurrence Recurrence tree (iteration) method
- Good for guessing an answer
Substitution method
- Generic method, rigid, but may be hard
Master method
- Easy to learn, useful in limited cases only
- Some tricks may help in other cases
2/21/2019

41. The master method The master method applies to recurrences of the form
T(n) = a T(n/b) + f (n) ,
where a ³ 1, b > 1, and f is asymptotically positive.
Divide the problem into a subproblems, each of size n/b
Conquer the subproblems by solving them recursively.
Combine subproblem solutions
Divide + combine takes f(n) time.
2/21/2019

42. e.g.: merge sort: T(n) = 2 T(n/2) + Θ(n)
Master theorem
T(n) = a T(n/b) + f (n)
Key: compare f(n) with nlogba
CASE 1: f (n) = O(nlogba – e)  T(n) = Q(nlogba) .
CASE 2: f (n) = Q(nlogba)  T(n) = Q(nlogba log n) .
CASE 3: f (n) = W(nlogba + e) and a f (n/b) £ c f (n)
 T(n) = Q( f (n)) .
e.g.: merge sort: T(n) = 2 T(n/2) + Θ(n)
a = 2, b = 2  nlogba = n
 CASE 2  T(n) = Θ(n log n) .
2/21/2019

43. Case 1 Compare f (n) with nlogba:
f (n) = O(nlogba – e) for some constant e > 0.
: f (n) grows polynomially slower than nlogba (by an ne factor).
Solution: T(n) = Q(nlogba) i.e., aT(n/b) dominates
e.g. T(n) = 2T(n/2) + 1
T(n) = 4 T(n/2) + n
T(n) = 2T(n/2) + log n
T(n) = 8T(n/2) + n2
2/21/2019

44. Case 3 Compare f (n) with nlogba:
f (n) = W (nlogba + e) for some constant e > 0.
: f (n) grows polynomially faster than nlogba (by an ne factor).
Solution: T(n) = Q(f(n)) i.e., f(n) dominates
e.g. T(n) = T(n/2) + n
T(n) = 2 T(n/2) + n2
T(n) = 4T(n/2) + n3
T(n) = 8T(n/2) + n4
2/21/2019

45. Case 2 Compare f (n) with nlogba: f (n) = Q (nlogba).
: f (n) and nlogba grow at similar rate.
Solution: T(n) = Q(nlogba log n)
e.g. T(n) = T(n/2) + 1
T(n) = 2 T(n/2) + n
T(n) = 4T(n/2) + n2
T(n) = 8T(n/2) + n3
2/21/2019

46. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
2/21/2019

47. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
2/21/2019

48. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
2/21/2019

49. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
2/21/2019

50. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.…
Q(1)
2/21/2019

51. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
h = log n
dn/4
dn/4
dn/4
dn/4 …
Q(1)
2/21/2019

52. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
h = log n
dn/4
dn/4
dn/4
dn/4 …
Q(1)
2/21/2019

53. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
h = log n
dn/4
dn/4
dn/4
dn/4 …
Q(1)
2/21/2019

54. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
h = log n
dn/4
dn/4
dn/4
dn/4 dn … …
Q(1)
2/21/2019

55. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
h = log n
dn/4
dn/4
dn/4
dn/4 dn … …
Q(1)
#leaves = n
Q(n)
2/21/2019

56. Recursion tree Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
h = log n
dn/4
dn/4
dn/4
dn/4 dn … …
Q(1)
#leaves = n
Q(n)
Total Q(n log n)
2/21/2019

57. Substitution method
The most general method to solve a recurrence (prove O and  separately):
Guess the form of the solution:
(e.g. using recursion trees, or expansion)
Verify by induction (inductive step).
2/21/2019

58. Proof by substitution Recurrence: T(n) = 2T(n/2) + n.
Guess: T(n) = O(n log n). (eg. by recurrence tree method)
To prove, have to show T(n) ≤ c n log n for some c > 0 and for all n > n0
Proof by induction: assume it is true for T(n/2), prove that it is also true for T(n). This means:
Fact: T(n) = 2T(n/2) + n
Assumption: T(n/2)≤ cn/2 log (n/2)
Need to Prove: T(n)≤ c n log (n)
2/21/2019

59. Proof Fact: T(n) = 2T(n/2) + n Assumption: T(n/2)≤ cn/2 log (n/2)
Need to Prove: T(n)≤ c n log (n)
Proof:
Substitute T(n/2) into the recurrence function
=> T(n) = 2 T(n/2) + n ≤ cn log (n/2) + n
=> T(n) ≤ c n log n - c n + n
=> T(n) ≤ c n log n (if we choose c ≥ 1).
2/21/2019

60. Proof by substitution Recurrence: T(n) = 2T(n/2) + n.
Guess: T(n) = Ω(n log n).
To prove, have to show T(n) ≥ c n log n for some c > 0 and for all n > n0
Proof by induction: assume it is true for T(n/2), prove that it is also true for T(n). This means:
Fact:
Assumption:
Need to Prove: T(n) ≥ c n log (n)
T(n) = 2T(n/2) + n
T(n/2) ≥ cn/2 log (n/2)
2/21/2019

61. Proof Fact: T(n) = 2T(n/2) + n Assumption: T(n/2) ≥ cn/2 log (n/2)
Need to Prove: T(n) ≥ c n log (n)
Proof:
Substitute T(n/2) into the recurrence function
=> T(n) = 2 T(n/2) + n ≥ cn log (n/2) + n
=> T(n) ≥ c n log n - c n + n
=> T(n) ≥ c n log n (if we choose c ≤ 1).
2/21/2019

62. Quick sort Quicksort an n-element array:
Divide: Partition the array into two subarrays around a pivot x such that elements in lower subarray £ x £ elements in upper subarray.
Conquer: Recursively sort the two subarrays.
Combine: Trivial.
£ x x
≥ x
Key: Linear-time partitioning subroutine.
2/21/2019

63. Partition All the action takes place in the partition() function £ x x
Rearranges the subarray in place
End result: two subarrays
All values in first subarray  all values in second
Returns the index of the “pivot” element separating the two subarrays p q r
£ x x
≥ x
2/21/2019

64. Partition Code What is the running time of partition()?
Partition(A, p, r)
x = A[p]; // pivot is the first element
i = p;
j = r + 1;
while (TRUE) {
repeat
i++;
until A[i] > x or i >= j;
j--;
until A[j] < x or j < i;
if (i < j)
Swap (A[i], A[j]);
else
break; }
swap (A[p], A[j]);
return j;
What is the running time of partition()?
partition() runs in O(n) time
2/21/2019

65. p r 6 10 5 8 13 3 2 11
x = 6 i j 6 10 5 8 13 3 2 11 i j 6 2 5 8 13 3 10 11 i j 6 2 5 8 13 3 10 11 i j 6 2 5 3 13 8 10 11 i j 6 2 5 3 13 8 10 11 j i p q r 3 2 5 6 13 8 10 11
2/21/2019

66. 6 10 5 8 11 3 2 13 3 2 5 6 11 8 10 13 2 3 5 6 10 8 11 13 2 3 5 6 8 10 11 13 2 3 5 6 8 10 11 13
2/21/2019

67. Quicksort Runtimes Best case runtime Tbest(n)  O(n log n)
Worst case runtime Tworst(n)  O(n2)
Worse than mergesort? Why is it called quicksort then?
Its average runtime Tavg(n)  O(n log n )
Better even, the expected runtime of randomized quicksort is O(n log n)
2/21/2019

68. Randomized quicksort Randomly choose an element as pivot
Every time need to do a partition, throw a die to decide which element to use as the pivot
Each element has 1/n probability to be selected
Partition(A, p, r)
d = random(); // a random number between 0 and 1
index = p + floor((r-p+1) * d); // p<=index<=r
swap(A[p], A[index]);
x = A[p];
i = p;
j = r + 1;
while (TRUE) { … }
2/21/2019

69. Running time of randomized quicksort
T(0) + T(n–1) + dn if 0 : n–1 split,
T(1) + T(n–2) + dn if 1 : n–2 split, M
T(n–1) + T(0) + dn if n–1 : 0 split,
T(n) =
The expected running time is an average of all cases
Expectation
2/21/2019

70. Heaps In practice, heaps are usually implemented as arrays: 16 14 10 87 9 3 2 4 1 16 14 10 8 7 9 3 2 4 1
2/21/2019

71. Heaps To represent a complete binary tree as an array:
The root node is A[1]
Node i is A[i]
The parent of node i is A[i/2] (note: integer divide)
The left child of node i is A[2i]
The right child of node i is A[2i + 1] 16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1 =
2/21/2019

72. The Heap Property Heaps also satisfy the heap property:
A[Parent(i)]  A[i] for all nodes i > 1
In other words, the value of a node is at most the value of its parent
The value of a node should be greater than or equal to both its left and right children
And all of its descendents
Where is the largest element in a heap stored?
2/21/2019

73. Heap Operations: Heapify()
Heapify(A, i)
{ // precondition: subtrees rooted at l and r are heaps
l = Left(i); r = Right(i);
if (l <= heap_size(A) && A[l] > A[i])
largest = l;
else
largest = i;
if (r <= heap_size(A) && A[r] > A[largest])
largest = r;
if (largest != i) {
Swap(A, i, largest);
Heapify(A, largest); }
} // postcondition: subtree rooted at i is a heap
Among A[l], A[i], A[r],
which one is largest?
If violation, fix it.
2/21/2019

74. Heapify() Example 16 4 10 14 7 9 3 2 8 1
A = 16 4 10 14 7 9 3 2 8 1
2/21/2019

75. Heapify() Example 16 4 10 14 7 9 3 2 8 1
A = 16 4 10 14 7 9 3 2 8 1
2/21/2019

76. Heapify() Example 16 4 10 14 7 9 3 2 8 1
A = 16 4 10 14 7 9 3 2 8 1
2/21/2019

77. Heapify() Example 16 14 10 4 7 9 3 2 8 1
A = 16 14 10 4 7 9 3 2 8 1
2/21/2019

78. Heapify() Example 16 14 10 4 7 9 3 2 8 1
A = 16 14 10 4 7 9 3 2 8 1
2/21/2019

79. Heapify() Example 16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1
2/21/2019

80. Heapify() Example 16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1
2/21/2019

81. Analyzing Heapify(): Formal
T(n)  T(2n/3) + (1)
By case 2 of the Master Theorem,
T(n) = O(lg n)
Thus, Heapify() takes logarithmic time
2/21/2019

82. Heap Operations: BuildHeap()
We can build a heap in a bottom-up manner by running Heapify() on successive subarrays
Fact: for array of length n, all elements in range A[n/2 n] are heaps (Why?)
So:
Walk backwards through the array from n/2 to 1, calling Heapify() on each node.
Order of processing guarantees that the children of node i are heaps when i is processed
2/21/2019

83. BuildHeap() // given an unsorted array A, make A a heap BuildHeap(A) {
heap_size(A) = length(A);
for (i = length[A]/2 downto 1)
Heapify(A, i); }
2/21/2019

84. BuildHeap() Example
Work through example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} 4 1 3 2 16 9 10 14 8 7
2/21/2019

85. 4 1 3 2 16 9 10 14 8 7
2/21/2019

86. 4 1 3 14 16 9 10 2 8 7
2/21/2019

87. 4 1 10 14 16 9 3 2 8 7
2/21/2019

88. 4 16 10 14 7 9 3 2 8 1
2/21/2019

89. 16 14 10 8 7 9 3 2 4 1
2/21/2019

90. Analyzing BuildHeap(): Tight
To Heapify() a subtree takes O(h) time where h is the height of the subtree
h = O(lg m), m = # nodes in subtree
The height of most subtrees is small
Fact: an n-element heap has at most n/2h+1 nodes of height h
CLR 7.3 uses this fact to prove that BuildHeap() takes O(n) time
2/21/2019

91. Heapsort Example
Work through example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} 4 1 3 2 16 9 10 14 8 7
A = 4 1 3 2 16 9 10 14 8 7
2/21/2019

92. Heapsort Example First: build a heap 16 14 10 8 7 9 3 2 4 1 A = 16 14
2/21/2019

93. Heapsort Example Swap last and first 1 14 10 8 7 9 3 2 4 16 A = 1 14
2/21/2019

94. Heapsort Example Last element sorted 1 14 10 8 7 9 3 2 4 16 A = 1 14
2/21/2019

95. Heapsort Example Restore heap on remaining unsorted elements 14 8 10 47 9 3 2 1 16
Heapify
A = 14 8 10 4 7 9 3 2 1 16
2/21/2019

96. Heapsort Example Repeat: swap new last and first 1 8 10 4 7 9 3 2 1416
A = 1 8 10 4 7 9 3 2 14 16
2/21/2019

97. Heapsort Example Restore heap 10 8 9 4 7 1 3 2 14 16 A = 10 8 9 4 7 1
2/21/2019

98. Heapsort Example Repeat 9 8 3 4 7 1 2 10 14 16 A = 9 8 3 4 7 1 2 10 14
2/21/2019

99. Heapsort Example Repeat 8 7 3 4 2 1 9 10 14 16 A = 8 7 3 4 2 1 9 10 14
2/21/2019

100. Heapsort Example Repeat 1 2 3 4 7 8 9 10 14 16 A = 1 2 3 4 7 8 9 10 14
2/21/2019

101. Analyzing Heapsort The call to BuildHeap() takes O(n) time
Each of the n - 1 calls to Heapify() takes O(lg n) time
Thus the total time taken by HeapSort() = O(n) + (n - 1) O(lg n) = O(n) + O(n lg n) = O(n lg n)
2/21/2019

102. HeapExtractMax Example16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1
2/21/2019

103. HeapExtractMax Example
Swap first and last, then remove last 1 14 10 8 7 9 3 2 4 16
A = 1 14 10 8 7 9 3 2 4 16
2/21/2019

104. HeapExtractMax Example
Heapify 14 8 10 4 7 9 3 2 1 16
A = 14 8 10 4 7 9 3 2 1 16
2/21/2019

105. HeapChangeKey Example
Increase key 16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1
2/21/2019

106. HeapChangeKey Example
Increase key 16 14 10 15 7 9 3 2 4 1
A = 16 14 10 15 7 9 3 2 4 1
2/21/2019

107. HeapChangeKey Example
Increase key 16 15 10 14 7 9 3 2 4 1
A = 16 15 10 14 7 9 3 2 4 1
2/21/2019

108. HeapInsert Example HeapInsert(A, 17) 16 14 10 8 7 9 3 2 4 1 A = 16 14
2/21/2019

109. HeapInsert Example HeapInsert(A, 17) -∞ -∞ -∞ makes it a valid heap 1614 10 8 7 9 3 2 4 1 -∞
-∞ makes it a valid heap
A = 16 14 10 8 7 9 3 2 4 1 -∞
2/21/2019

110. HeapInsert Example HeapInsert(A, 17) Now call changeKey 16 14 10 8 7 93 2 4 1 17
Now call changeKey
A = 16 14 10 8 7 9 3 2 4 1 17
2/21/2019

111. HeapInsert Example HeapInsert(A, 17) 17 16 10 8 14 9 3 2 4 1 7 A = 17
2/21/2019

112. HeapExtractMax: Θ(log n) HeapChangeKey: Θ(log n) HeapInsert: Θ(log n)
Heapify: Θ(log n)
BuildHeap: Θ(n)
HeapSort: Θ(nlog n)
HeapMaximum: Θ(1)
HeapExtractMax: Θ(log n)
HeapChangeKey: Θ(log n)
HeapInsert: Θ(log n)
2/21/2019

113. Counting sort for i  1 to k do C[i]  0 for j  1 to n1.
for i  1 to k
do C[i]  0
for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key £ i}|
for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
Initialize 2.
Count 3.
Compute running sum 4.
Re-arrange
2/21/2019

114. Counting sort A: 4 1 3 4 3 C: 1 2 2 B: C': 1 1 3 5 for i  2 to k2 2 B:
C': 1 1 3 5 3.
for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key £ i}|
2/21/2019

115. Loop 4: re-arrange A: 4 1 3 4 3 C: 1 1 3 5 B: 3 C': 1 1 3 52 3 4 5 1 2 3 4 A: 4 1 3 4 3 C: 1 1 3 5 B: 3
C': 1 1 3 5 4.
for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
2/21/2019

116. Analysis Q(k) Q(n) Q(k) Q(n) Q(n + k) 1. for i  1 to k do C[i]  0 2.
for j  1 to n
do C[A[ j]]  C[A[ j]] + 1
Q(n) 3.
for i  2 to k
do C[i]  C[i] + C[i–1]
Q(k) 4.
for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
Q(n)
Q(n + k)
2/21/2019

117. What other algorithms have this property?
Stable sorting
Counting sort is a stable sort: it preserves the input order among equal elements. A: 4 1 3 B:
Why this is important?
What other algorithms have this property?
2/21/2019

118. Radix sort Similar to sorting the address books
Treat each digit as a key
Start from the least significant bit
Most significant
Least significant
2/21/2019

119. Time complexity Sort each of the d digits by counting sort
Total cost: d (n + k)
k = 10
Total cost: Θ(dn)
Partition the d digits into groups of 3
Total cost: (n+103)d/3
We work with binaries rather than decimals
Partition a binary number into groups of r bits
Total cost: (n+2r)d/r
Choose r = log n
Total cost: dn / log n
Compare with dn log n
Catch: faster than quicksort only when n is very large
2/21/2019

120. Randomized selection algorithm
RAND-SELECT(A, p, q, i) ⊳ i th smallest of A[ p . . q]
if p = q & i > 1 then error!
r  RAND-PARTITION(A, p, q)
k  r – p + 1 ⊳ k = rank(A[r])
if i = k then return A[ r]
if i < k
then return RAND-SELECT( A, p, r – 1, i )
else return RAND-SELECT( A, r + 1, q, i – k )
£ A[r]
³ A[r] r p q k
2/21/2019

121. Select the 6 – 4 = 2nd smallest recursively.
Example
Select the i = 6th smallest: 7 10 5 8 11 3 2 13
i = 6
pivot 3 2 5 7 11 8 10 13
Partition:
k = 4
Select the 6 – 4 = 2nd smallest recursively.
2/21/2019

122. Complete example: select the 6th smallest element.
i = 6 7 10 5 8 11 3 2 13 3 2 5 7 11 8 10 13
k = 4
i = 6 – 4 = 2
k = 3 10 8 11 13
i = 2 < k
Note: here we always used first element as pivot to do the partition (instead of rand-partition).
k = 2 8 10
i = 2 = k 10
2/21/2019

123. Intuition for analysis
(All our analyses today assume that all elements are distinct.)
Lucky:
T(n) = T(9n/10) + Q(n)
= Q(n)
CASE 3
Unlucky:
T(n) = T(n – 1) + Q(n)
= Q(n2)
arithmetic series
Worse than sorting!
2/21/2019

124. Running time of randomized selection
T(max(0, n–1)) + n if 0 : n–1 split,
T(max(1, n–2)) + n if 1 : n–2 split, M
T(max(n–1, 0)) + n if n–1 : 0 split,
T(n) ≤
For upper bound, assume ith element always falls in larger side of partition
The expected running time is an average of all cases
Expectation
2/21/2019

125. Worst-case linear-time selection
if i = k then return x
elseif i < k
then recursively SELECT the i th smallest element in the lower part
else recursively SELECT the (i–k)th smallest element in the upper part
SELECT(i, n)
Divide the n elements into groups of 5. Find the median of each 5-element group by rote.
Recursively SELECT the median x of the ën/5û group medians to be the pivot.
Partition around the pivot x. Let k = rank(x).
Same as RAND-SELECT
2/21/2019

126. Developing the recurrence
T(n)
if i = k then return x
elseif i < k
then recursively SELECT the i th smallest element in the lower part
else recursively SELECT the (i–k)th smallest element in the upper part
SELECT(i, n)
Divide the n elements into groups of 5. Find the median of each 5-element group by rote.
Recursively SELECT the median x of the ën/5û group medians to be the pivot.
Partition around the pivot x. Let k = rank(x).
Q(n)
T(n/5)
Q(n)
T(7n/10+3)
2/21/2019

127. Solving the recurrence
Assumption: T(k) £ ck for all k < n
if n ≥ 60
if c ≥ 20 and n ≥ 60
2/21/2019

128. Elements of dynamic programming
Optimal sub-structures
Optimal solutions to the original problem contains optimal solutions to sub-problems
Overlapping sub-problems
Some sub-problems appear in many solutions
2/21/2019

129. Two steps to dynamic programming
Formulate the solution as a recurrence relation of solutions to subproblems.
Specify an order to solve the subproblems so you always have what you need.
2/21/2019

130. Optimal subpaths
Claim: if a path startgoal is optimal, any sub-path, startx, or xgoal, or xy, where x, y is on the optimal path, is also the shortest.
Proof by contradiction
If the subpath between x and y is not the shortest, we can replace it with the shorter one, which will reduce the total length of the new path => the optimal path from start to goal is not the shortest => contradiction!
Hence, the subpath xy must be the shortest among all paths from x to y
start
goal x y a b c b’
a + b + c is shortest
b’ < b
a + b’ + c < a + b + c
2/21/2019

131. Dynamic programming illustration3 9 1 2 3 12 13 15 5 3 3 3 3 3 2 5 2 5 6 8 13 15 2 3 3 9 3 2 4 2 3 7 9 11 13 16 6 2 3 7 4 3 6 3 3 13 11 14 17 20 4 6 3 1 3 1 2 3 2 17 17 17 18 20 G
F(i-1, j) + dist(i-1, j, i, j)
F(i, j) = min
F(i, j-1) + dist(i, j-1, i, j)
2/21/2019

132. Trace back 3 9 1 2 3 12 13 15 5 3 3 3 3 3 2 5 2 5 6 8 13 15 2 3 3 9 3 2 4 2 3 7 9 11 13 16 6 2 3 7 4 3 6 3 3 13 11 14 17 20 4 6 3 1 3 1 2 3 2 17 17 17 18 20
2/21/2019

133. Longest Common Subsequence
Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both.
“a” not “the” x: A B C D y:
BCBA = LCS(x, y)
functional notation, but not a function
2/21/2019

134. Optimal substructure
Notice that the LCS problem has optimal substructure: parts of the final solution are solutions of subproblems.
If z = LCS(x, y), then any prefix of z is an LCS of a prefix of x and a prefix of y.
Subproblems: “find LCS of pairs of prefixes of x and y” i m x z n y j
2/21/2019

135. Finding length of LCS m x n y
Let c[i, j] be the length of LCS(x[1..i], y[1..j])
=> c[m, n] is the length of LCS(x, y)
If x[m] = y[n]
c[m, n] = c[m-1, n-1] + 1
If x[m] != y[n]
c[m, n] = max { c[m-1, n], c[m, n-1] }
2/21/2019

136. DP Algorithm c[i–1, j–1] + 1 if x[i] = y[j],
Key: find out the correct order to solve the sub-problems
Total number of sub-problems: m * n
c[i, j] =
c[i–1, j–1] + 1 if x[i] = y[j],
max{c[i–1, j], c[i, j–1]} otherwise. j n
C(i, j) i m
2/21/2019

137. LCS Example (0) ABCB BDCAB X = ABCB; m = |X| = 4j i
Y[j] B D C A B
X[i] A 1 B 2 3 C 4 B
X = ABCB; m = |X| = 4
Y = BDCAB; n = |Y| = 5
Allocate array c[5,6]
2/21/2019

138. LCS Example (1) ABCB BDCAB for i = 1 to m c[i,0] = 0j i
Y[j] B D C A B
X[i] A 1 B 2 3 C 4 B
for i = 1 to m c[i,0] = 0
for j = 1 to n c[0,j] = 0
2/21/2019

139. LCS Example (2) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 B 2A 1 B 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
2/21/2019

140. LCS Example (3) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 B 2A 1 B 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
2/21/2019

141. LCS Example (4) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 BA 1 1 B 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
2/21/2019

142. LCS Example (5) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
2/21/2019

143. LCS Example (6) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 1 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
2/21/2019

144. LCS Example (7) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 1 1 1 1 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
2/21/2019

145. LCS Example (8) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 1 1 1 1 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
2/21/2019

146. LCS Example (14) 3 ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
2/21/2019

147. LCS Algorithm Running Time
LCS algorithm calculates the values of each entry of the array c[m,n]
So what is the running time?
O(m*n)
since each c[i,j] is calculated in constant time, and there are m*n elements in the array
2/21/2019

148. How to find actual LCS For example, here
The algorithm just found the length of LCS, but not LCS itself.
How to find the actual LCS?
For each c[i,j] we know how it was acquired:
A match happens only when the first equation is taken
So we can start from c[m,n] and go backwards, remember x[i] whenever c[i,j] = c[i-1, j-1]+1. 2 2
For example, here
c[i,j] = c[i-1,j-1] +1 = 2+1=3 2 3
2/21/2019

149. Finding LCS 3 Time for trace back: O(m+n). j 0 1 2 3 4 5 i Y[j] B D C
X[i] A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2
Time for trace back: O(m+n).
2/21/2019

150. Finding LCS (2) 3 LCS (reversed order): B C B B C Bj i
Y[j] B D C A B
X[i] A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2
LCS (reversed order): B C B
B C B
(this string turned out to be a palindrome)
LCS (straight order):
2/21/2019

151. LCS as a longest path problemD C A B A 1 B 1 1 1 C B 1 1
2/21/2019

152. LCS as a longest path problemD C A B A 1 1 1 1 1 B 1 1 1 1 2 1 C 1 1 2 2 2 B 1 1 1 1 1 2 3
2/21/2019

153. Restaurant location problem 1
You work in the fast food business
Your company plans to open up new restaurants in Texas along I-35
Towns along the highway called t1, t2, …, tn
Restaurants at ti has estimated annual profit pi
No two restaurants can be located within 10 miles of each other due to some regulation
Your boss wants to maximize the total profit
You want a big bonus
10 mile
2/21/2019

154. A DP algorithm Suppose you’ve already found the optimal solution
It will either include tn or not include tn
Case 1: tn not included in optimal solution
Best solution same as best solution for t1 , …, tn-1
Case 2: tn included in optimal solution
Best solution is pn + best solution for t1 , …, tj , where j < n is the largest index so that dist(tj, tn) ≥ 10
2/21/2019

155. Recurrence formulation
Let S(i) be the total profit of the optimal solution when the first i towns are considered (not necessarily selected)
S(n) is the optimal solution to the complete problem
S(n-1)
S(j) + pn j < n & dist (tj, tn) ≥ 10
S(n) = max
S(i-1)
S(j) + pi j < i & dist (tj, ti) ≥ 10
S(i) = max
Generalize
Number of sub-problems: n. Boundary condition: S(0) = 0.
Dependency: i
i-1 j S
2/21/2019

156. Example S(i-1) S(j) + pi j < i & dist (tj, ti) ≥ 10 S(i) = max
Distance (mi)
100 5 2 2 6 6 3 6 10 7
dummy 7 3 4 12
Profit (100k) 6 7 9 8 3 3 2 4 12 5
S(i) 6 7 9 9 10 12 12 14 26 26
Optimal: 26
S(i-1)
S(j) + pi j < i & dist (tj, ti) ≥ 10
S(i) = max
Natural greedy 1: = 25
Natural greedy 2: = 24
2/21/2019

157. Complexity
Time: (nk), where k is the maximum number of towns that are within 10 miles to the left of any town
In the worst case, (n2)
Can be improved to (n) with some preprocessing tricks
Memory: Θ(n)
2/21/2019

158. Knapsack problem Each item has a value and a weight
Objective: maximize value
Constraint: knapsack has a weight limitation
Three versions:
0-1 knapsack problem: take each item or leave it
Fractional knapsack problem: items are divisible
Unbounded knapsack problem: unlimited supplies of each item.
Which one is easiest to solve?
We study the 0-1 problem today.
2/21/2019

159. Formal definition (0-1 problem)
Knapsack has weight limit W
Items labeled 1, 2, …, n (arbitrarily)
Items have weights w1, w2, …, wn
Assume all weights are integers
For practical reason, only consider wi < W
Items have values v1, v2, …, vn
Objective: find a subset of items, S, such that iS wi  W and iS vi is maximal among all such (feasible) subsets
2/21/2019

160. A DP algorithm Suppose you’ve find the optimal solution S
Case 1: item n is included
Case 2: item n is not included
Total weight limit: W
Total weight limit: W wn wn
Find an optimal solution using items 1, 2, …, n-1 with weight limit W - wn
Find an optimal solution using items 1, 2, …, n-1 with weight limit W
2/21/2019

161. Recursive formulation
Let V[i, w] be the optimal total value when items 1, 2, …, i are considered for a knapsack with weight limit w
=> V[n, W] is the optimal solution
V[n, W] = max
V[n-1, W-wn] + vn
V[n-1, W]
Generalize
V[i, w] = max
V[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
V[i-1, w] if wi > w item i not taken
Boundary condition: V[i, 0] = 0, V[0, w] = 0. Number of sub-problems = ?
2/21/2019

162. Example n = 6 (# of items) W = 10 (weight limit)
Items (weight, value):
2/21/2019

163. w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 2 2 4 3 wi 3 3 3
V[i-1, w-wi]
V[i-1, w] 4 5 5 6 6
V[i, w] 5 2 4 6 6 9
V[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
max
V[i, w] =
V[i-1, w] if wi > w item i not taken
2/21/2019

164. w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 4 3 5 6 9 2 2 2 2 2 2 2 2 2 3 5 2 2 3 5 5 5 5 6 8 3 5 2 3 5 6 8 6 8 9 11 2 3 3 6 9 4 6 7 10 12 13 4 7 10 13 15 9 4 4 6 7 10 13
V[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
max
V[i-1, w] if wi > w item i not taken
V[i, w] =
2/21/2019

165. w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 4 3 5 6 9 2 2 2 2 2 2 2 2 2 3 5 2 2 3 5 5 5 5 6 8 3 5 2 3 5 6 8 6 8 9 11 2 3 3 6 9 4 7 10 12 13 4 6 7 10 13 9 4 4 6 7 10 13 15
Optimal value: 15
Item: 6, 5, 1
Weight: = 10
Value: = 15
2/21/2019

166. Time complexity Θ (nW) Polynomial?
Pseudo-polynomial
Works well if W is small
Consider following items (weight, value):
(10, 5), (15, 6), (20, 5), (18, 6)
Weight limit 35
Optimal solution: item 2, 4 (value = 12). Iterate: 2^4 = 16 subsets
Dynamic programming: fill up a 4 x 35 = 140 table entries
What’s the problem?
Many entries are unused: no such weight combination
Top-down may be better
2/21/2019

167. Longest increasing subsequence
Given a sequence of numbers
Find a longest subsequence that is non-decreasing
E.g
It has to be a subsequence of the original list
It has to in sorted order
=> It is a subsequence of the sorted list
Original list:
LCS:
Sorted:
2/21/2019

168. Events scheduling problem
Time
A list of events to schedule (or shows to see)
ei has start time si and finishing time fi
Indexed such that fi < fj if i < j
Each event has a value vi
Schedule to make the largest value
You can attend only one event at any time
Very similar to the new restaurant location problem
Sort events according to their finish time
Consider: if the last event is included or not
2/21/2019

169. Events scheduling problemf9 s8 f8 s7 f7 e8 e3 e4 e5 e7 e9 e1 e2
Time
V(i) is the optimal value that can be achieved when the first i events are considered
V(n) =
V(n-1)
en not selected
max {
V(j) + vn
en selected
j < n and fj < sn
2/21/2019

170. Coin change problem
Given some denomination of coins (e.g., 2, 5, 7, 10), decide if it is possible to make change for a value (e.g, 13), or minimize the number of coins
Version 1: Unlimited number of coins for each denomination
Unbounded knapsack problem
Version 2: Use each denomination at most once
0-1 Knapsack problem
2/21/2019

171. Use DP algorithm to solve new problems
Directly map a new problem to a known problem
Modify an algorithm for a similar task
Design your own
Think about the problem recursively
Optimal solution to a larger problem can be computed from the optimal solution of one or more subproblems
These sub-problems can be solved in certain manageable order
Works nicely for naturally ordered data such as strings, trees, some special graphs
Trickier for general graphs
The text book has some very good exercises.
2/21/2019

172. Unit-profit restaurant location problem
Now the objective is to maximize the number of new restaurants (subject to the distance constraint)
In other words, we assume that each restaurant makes the same profit, no matter where it is opened
10 mile
2/21/2019

173. A DP Algorithm Exactly as before, but pi = 1 for all i S(i-1)
S(j) + pi j < i & dist (tj, ti) ≥ 10
S(i) = max
S(i-1)
S(j) + 1 j < i & dist (tj, ti) ≥ 10
S(i) = max
2/21/2019

174. Greedy algorithm for restaurant location problem
select t1
d = 0;
for (i = 2 to n)
d = d + dist(ti, ti-1);
if (d >= min_dist)
select ti
end 5 2 2 6 6 3 6 10 7 d 5 7 9 15 6 9 15 10 7
2/21/2019

175. Complexity Time: Θ(n) Memory: Θ(n) to store the input
Θ(1) for greedy selection
2/21/2019

176. Optimal substructure
Claim 1: if A = [m1, m2, …, mk] is the optimal solution to the restaurant location problem for a set of towns [t1, …, tn]
m1 < m2 < … < mk are indices of the selected towns
Then B = [m2, m3, …, mk] is the optimal solution to the sub-problem [tj, …, tn], where tj is the first town that are at least 10 miles to the right of tm1
Proof by contradiction: suppose B is not the optimal solution to the sub-problem, which means there is a better solution B’ to the sub-problem
A’ = mi || B’ gives a better solution than A = mi || B => A is not optimal => contradiction => B is optimal B m1 A m2 mk m1
B’ (imaginary) A’
2/21/2019

177. Greedy choice property
Claim 2: for the uniform-profit restaurant location problem, there is an optimal solution that chooses t1
Proof by contradiction: suppose that no optimal solution can be obtained by choosing t1
Say the first town chosen by the optimal solution S is ti, i > 1
Replace ti with t1 will not violate the distance constraint, and the total profit remains the same => S’ is an optimal solution
Contradiction
Therefore claim 2 is valid S S’
2/21/2019

178. Fractional knapsack problem
Each item has a value and a weight
Objective: maximize value
Constraint: knapsack has a weight limitation
0-1 knapsack problem: take each item or leave it
Fractional knapsack problem: items are divisible
Unbounded knapsack problem: unlimited supplies of each item.
Which one is easiest to solve?
We can solve the fractional knapsack problem using greedy algorithm
2/21/2019

179. Greedy algorithm for fractional knapsack problem
Compute value/weight ratio for each item
Sort items by their value/weight ratio into decreasing order
Call the remaining item with the highest ratio the most valuable item (MVI)
Iteratively:
If the weight limit can not be reached by adding MVI
Select MVI
Otherwise select MVI partially until weight limit
2/21/2019

180. Example Weight limit: 10 9 6 4 2 5 3 1 Value ($) Weight (LB) item 1.5
1.2 1
0.75
$ / LB
2/21/2019

181. Example Weight limit: 10 Take item 5 Take item 6 Take 2 LB of item 4
Weight (LB)
Value ($)
$ / LB 5 2 4 6 9
1.5
1.2 1 3
0.75
2/21/2019

182. Why is greedy algorithm for fractional knapsack problem valid?
Claim: the optimal solution must contain the MVI as much as possible (either up to the weight limit or until MVI is exhausted)
Proof by contradiction: suppose that the optimal solution does not use all available MVI (i.e., there is still w (w < W) pounds of MVI left while we choose other items)
We can replace w pounds of less valuable items by MVI
The total weight is the same, but with value higher than the “optimal”
Contradiction w w
2/21/2019

183. Graphs 1 2 4 3 A graph G = (V, E) V = set of vertices
E = set of edges = subset of V  V
Thus |E| = O(|V|2) 1
Vertices: {1, 2, 3, 4}
Edges: {(1, 2), (2, 3), (1, 3), (4, 3)} 2 4 3
2/21/2019

184. Graphs: Adjacency Matrix
Example: A 1 2 3 4 1 2 4 3
How much storage does the adjacency matrix require?
A: O(V2)
2/21/2019

185. Graphs: Adjacency List
Adjacency list: for each vertex v  V, store a list of vertices adjacent to v
Example:
Adj[1] = {2,3}
Adj[2] = {3}
Adj[3] = {}
Adj[4] = {3}
Variation: can also keep a list of edges coming into vertex 1 2 4 3
2/21/2019

186. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

187. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

188. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

189. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

190. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

191. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

192. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

193. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

194. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

195. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

196. Kruskal’s algorithm: example
c-d: 3
b-f: 5
b-a: 6
f-e: 7
b-d: 8
f-g: 9
d-e: 10
a-f: 12
b-c: 14
e-h: 15 a 6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

197. Time complexity Depending on implementation Pseudocode:
sort all edges according to weights
T = {}. tree(v) = v for all v.
for each edge (u, v)
if tree(u) != tree(v)
T = T U (u, v);
union (tree(u), tree(v))
Θ(m log m)
= Θ(m log n)
m edges
Avg time spent per edge
Naïve: Θ (n)
Better: Θ (log n)
using set union
Overall time complexity
Naïve: Θ(nm)
Better implementation: Θ(m log n)
2/21/2019

198. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d a b c d e f g h ∞ ∞
2/21/2019

199. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ChangeKey c b a d e f g h 0 ∞
2/21/2019

200. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ExctractMin h b a d e f g ∞
2/21/2019

201. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ChangeKey d b a h e f g 3 14 ∞
2/21/2019

202. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ExctractMin b g a h e f
14 ∞
2/21/2019

203. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
Changekey b e a h g f 8 10 ∞
2/21/2019

204. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ExtractMin e f a h g 10 ∞
2/21/2019

205. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
Changekey f e a h g 5 10 6 ∞
2/21/2019

206. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ExtractMin a e g h 6 10 ∞
2/21/2019

207. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
Changekey a e g h 6 7 9 ∞
2/21/2019

208. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ExtractMin e h g 7 ∞ 9
2/21/2019

209. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ExtractMin g h 9 ∞
2/21/2019

210. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
Changekey g h 9 15
2/21/2019

211. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
ExtractMin h 15
2/21/2019

212. Prim’s algorithm: example6 12 5 9 b f g 14 7 15 8 c e h 3 10 d
2/21/2019

213. Complete Prim’s Algorithm
MST-Prim(G, w, r)
Q = V[G];
for each u  Q
key[u] = ;
key[r] = 0;
T = {};
while (Q not empty)
u = ExtractMin(Q);
for each v  Adj[u]
if (v  Q and w(u,v) < key[v])
T = T U (u, v);
ChangeKey(v, w(u,v));
Overall running time: Θ(m log n)
Cost per ChangeKey
n vertices
Θ(n) times
Θ(n2) times?
Θ(m) times
How often is ExtractMin() called?
How often is ChangeKey() called?
2/21/2019

214. Summary Kruskal’s algorithm Prim’s algorithm Θ(m log n)
Possibly Θ(m + n log n) with counting sort
Prim’s algorithm
With priority queue : Θ(m log n)
Assume graph represented by adj list
With distance array : Θ(n^2)
Adj list or adj matrix
For sparse graphs priority queue wins
For dense graphs distance array may be better
2/21/2019

215. b h f e d a i c g a b c d e f g h i ∞ 14 7 5 Dijkstra’s algorithm 9 146 9 1 5 3 5 e 6 d 8 a i 4 7 2 1 c 7 g a b c d e f g h i ∞ 14 7 5
Dijkstra’s algorithm
2/21/2019

216. b h f e d a i c g a b c d e f g h i 11 7 5 ∞ Dijkstra’s algorithm 9 146 9 1 5 3 5 e 6 d 8 a i 4 7 11 2 1 c 7 g a b c d e f g h i 11 7 5 ∞
Dijkstra’s algorithm
2/21/2019

217. b h f e d a i c g a b c d e f g h i 9 11 7 5 ∞ Dijkstra’s algorithm 914 11 b 9 14 h 7 f 7 6 9 1 5 3 5 e 6 d 8 a i 4 11 7 2 1 9 c 7 g a b c d e f g h i 9 11 7 5 ∞
Dijkstra’s algorithm
2/21/2019

218. 14 11 b 9 14 h 12 7 f 7 6 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g a b c d e f g h i 9 11 7 5 12 ∞ 17
Dijkstra’s algorithm
2/21/2019

219. 14 11 b 9 20 14 h 12 7 f 7 6 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g a b c d e f g h i 9 11 7 5 12 ∞ 20 17
Dijkstra’s algorithm
2/21/2019

220. 14 11 b 9 20 14 h 12 19 7 f 7 6 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g a b c d e f g h i 9 11 7 5 12 ∞ 19 17
Dijkstra’s algorithm
2/21/2019

221. 14 11 b 9 20 14 h 12 19 7 f 7 6 18 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g 18 a b c d e f g h i 9 11 7 5 12 18 17
Dijkstra’s algorithm
2/21/2019

222. 14 11 b 9 20 14 h 12 19 7 f 7 6 18 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g 18 a b c d e f g h i 9 11 7 5 12 18 17
Dijkstra’s algorithm
2/21/2019

223. 14 11 b 9 20 14 h 12 19 7 f 7 6 18 9 1 5 3 5 e 6 d 8 a i 17 4 11 7 2 1 9 c 7 g 18 a b c d e f g h i 9 11 7 5 12 18 17
Dijkstra’s algorithm
2/21/2019

224. Prim’s Algorithm Overall running time: Θ(m log n) MST-Prim(G, w, r)
Q = V[G];
for each u  Q
key[u] = ;
key[r] = 0;
T = {};
while (Q not empty)
u = ExtractMin(Q);
for each v  Adj[u]
if (v  Q and w(u,v) < key[v])
T = T U (u, v);
ChangeKey(v, w(u,v));
Overall running time: Θ(m log n)
Cost per ChangeKey
2/21/2019

225. Running time of Dijkstra’s algorithm is the same as Prim’s algorithm
Dijkstra(G, w, r)
Q = V[G];
for each u  Q
key[u] = ;
key[r] = 0;
T = {};
while (Q not empty)
u = ExtractMin(Q);
for each v  Adj[u]
if (v  Q and key[u]+w(u,v) < key[v])
T = T U (u, v);
ChangeKey(v, key[u]+w(u,v));
Overall running time: Θ(m log n)
Cost per ChangeKey
Running time of Dijkstra’s algorithm is the same as Prim’s algorithm
2/21/2019

226. Good luck with your final!
2/21/2019