Algorithm Analysis Fall 2017 CS 4306/03 Graph Algorithms (Ch 22.1, Ch 23.1) Graph representation Minimum spanning trees Greedy Algorithms (Ch 23.1) Optimal substructure Greedy choice MST Algorithms (Ch 23.2) Prim’s MST algorithm Kruskal’s MST algorithm Dr. Donghyun Kim Department of Computer Science College of Computing and Software Engineering Kennesaw State University Marietta, GA, USA This slide is courtesy of Prof. Charles E. Leiserson at Massachusetts Institute of Technology.

Graphs (review) Definition. A directed graph (digraph) G = (V, E) is an ordered pair consisting of a set V of vertices (singular: vertex), a set E  V  V of edges. In an undirected graph G = (V, E), the edge set E consists of unordered pairs of vertices. In either case, we have | E | = O(V 2). Moreover, if G is connected, then | E |  | V | – 1, which implies that lg | E | = (lg V). (Review CLRS, Appendix B.) Fall 2016 CS 4306 Algorithm Analysis

Adjacency-matrix representation The adjacency matrix of a graph G = (V, E), where V = {1, 2, …, n}, is the matrix A[1 . . n, 1 . . n] given by 1 if (i, j)  E, 0 if (i, j)  E. A[i, j] = Fall 2016 CS 4306 Algorithm Analysis

Adjacency-matrix representation The adjacency matrix of a graph G = (V, E), where V = {1, 2, …, n}, is the matrix A[1 . . n, 1 . . n] given by 1 if (i, j)  E, 0 if (i, j)  E. A[i, j] = A 1 2 3 4 (V 2) storage  dense representation. Fall 2016 CS 4306 Algorithm Analysis

Adjacency-list representation An adjacency list of a vertex v  V is the list Adj[v] of vertices adjacent to v. Adj[1] = {2, 3} Adj[2] = {3} Adj[3] = {} Adj[4] = {3} Fall 2016 CS 4306 Algorithm Analysis

Adjacency-list representation An adjacency list of a vertex v  V is the list Adj[v] of vertices adjacent to v. Adj[1] = {2, 3} Adj[2] = {3} Adj[3] = {} Adj[4] = {3} For undirected graphs, | Adj[v] | = degree(v). For digraphs, | Adj[v] | = out-degree(v). Fall 2016 CS 4306 Algorithm Analysis

Adjacency-list representation An adjacency list of a vertex v  V is the list Adj[v] of vertices adjacent to v. Adj[1] = {2, 3} Adj[2] = {3} Adj[3] = {} Adj[4] = {3} For undirected graphs, | Adj[v] | = degree(v). For digraphs, | Adj[v] | = out-degree(v). Handshaking Lemma: vV = 2 |E| for undirected graphs  adjacency lists use (V + E) storage — a sparse representation (for either type of graph). Fall 2016 CS 4306 Algorithm Analysis

Minimum spanning trees Input: A connected, undirected graph G = (V, E) with weight function w : E  R. For simplicity, assume that all edge weights are distinct. (CLRS covers the general case.) Fall 2016 CS 4306 Algorithm Analysis

Minimum spanning trees Input: A connected, undirected graph G = (V, E) with weight function w : E  R. For simplicity, assume that all edge weights are distinct. (CLRS covers the general case.) Output: A spanning tree T — a tree that connects all vertices — of minimum weight:  w(u, v) . (u,v)T w(T )  Fall 2016 CS 4306 Algorithm Analysis

Example of MST 6 12 5 9 14 7 15 8 3 10 Fall 2016 CS 4306 Algorithm Analysis

Example of MST 6 12 5 9 14 7 15 8 3 10 Fall 2016 CS 4306 Algorithm Analysis

Optimal substructure MST T: (Other edges of G are not shown.) Fall 2016 CS 4306 Algorithm Analysis

Optimal substructure MST T: u v Remove any edge (u, v)  T. (Other edges of G are not shown.) u v Remove any edge (u, v)  T. Fall 2016 CS 4306 Algorithm Analysis

Optimal substructure MST T: u v Remove any edge (u, v)  T. (Other edges of G are not shown.) u v Remove any edge (u, v)  T. Fall 2016 CS 4306 Algorithm Analysis

Optimal substructure MST T: u T T1 v (Other edges of G are not shown.) u T 2 T1 v Remove any edge (u, v)  T. Then, T is partitioned into two subtrees T1 and T2. Fall 2016 CS 4306 Algorithm Analysis

Optimal substructure MST T: u T T1 v (Other edges of G are not shown.) u T 2 T1 v Remove any edge (u, v)  T. Then, T is partitioned into two subtrees T1 and T2. Theorem. The subtree T1 is an MST of G1 = (V1, E1), the subgraph of G induced by the vertices of T1: V1 = vertices of T1, E1 = { (x, y)  E : x, y  V1 }. Similarly for T2. Fall 2016 CS 4306 Algorithm Analysis

Proof of optimal substructure Proof. Cut and paste: w(T) = w(u, v) + w(T1) + w(T2). If T1 were a lower-weight spanning tree than T1 for G1, then T  = {(u, v)}  T1  T2 would be a lower-weight spanning tree than T for G. Fall 2016 CS 4306 Algorithm Analysis

Proof of optimal substructure Proof. Cut and paste: w(T) = w(u, v) + w(T1) + w(T2). If T1 were a lower-weight spanning tree than T1 for G1, then T  = {(u, v)}  T1  T2 would be a lower-weight spanning tree than T for G. Do we also have overlapping subproblems? •Yes. Fall 2016 CS 4306 Algorithm Analysis

Proof of optimal substructure Proof. Cut and paste: w(T) = w(u, v) + w(T1) + w(T2). If T1 were a lower-weight spanning tree than T1 for G1, then T  = {(u, v)}  T1  T2 would be a lower-weight spanning tree than T for G. Do we also have overlapping subproblems? •Yes. Great, then dynamic programming may work! •Yes, but MST exhibits another powerful property which leads to an even more efficient algorithm. Fall 2016 CS 4306 Algorithm Analysis

Greedy-choice property A locally optimal choice is globally optimal. Hallmark for “greedy” algorithms Greedy-choice property A locally optimal choice is globally optimal. Fall 2016 CS 4306 Algorithm Analysis

Hallmark for “greedy” algorithms Greedy-choice property A locally optimal choice is globally optimal. Theorem. Let T be the MST of G = (V, E), and let A  V. Suppose that (u, v)  E is the least-weight edge connecting A to V – A. Then, (u, v)  T. Fall 2016 CS 4306 Algorithm Analysis

Proof of theorem Proof. Suppose (u, v)  T. Cut and paste. T: v u  A  V – A (u, v) = least-weight edge connecting A to V – A Fall 2016 CS 4306 Algorithm Analysis

Proof of theorem Proof. Suppose (u, v)  T. Cut and paste. T: v u  A  V – A (u, v) = least-weight edge connecting A to V – A Consider the unique simple path from u to v in T. Fall 2016 CS 4306 Algorithm Analysis

Proof of theorem Proof. Suppose (u, v)  T. Cut and paste. T: v u  A  V – A (u, v) = least-weight edge connecting A to V – A Consider the unique simple path from u to v in T. Swap (u, v) with the first edge on this path that connects a vertex in A to a vertex in V – A. Fall 2016 CS 4306 Algorithm Analysis

Proof of theorem Proof. Suppose (u, v)  T. Cut and paste. T : v u  V – A (u, v) = least-weight edge connecting A to V – A Consider the unique simple path from u to v in T. Swap (u, v) with the first edge on this path that connects a vertex in A to a vertex in V – A. A lighter-weight spanning tree than T results. Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A  6 12 5  9  7 15 10   14  8   3  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A  6 12 5  9  7 15 10   14  8   3  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A  6 12 5  9    14  7 10 15 15 8  3  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A  6 12 5 9     14  7 10 15 15 8  3  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm  V – A  5 6 12 9  5  9  14  7 10 15 15 8  3   Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm  V – A  5 6 12 9   9  14  7 10 15 15 8  3   Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm  V – A  5 6 12 9   9  14  7 15 15 8  3 10   Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A 6 12 5 9   9  14  7 15 15 8  3 10  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A 6 12 5 9   9  14  7 15 15 8  3 10  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A 6 12 5 9   9  14  7 10 15 15 8  3  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A 6 12 5 9   9  14  7 10 15 15 8  3  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A 6 12 5 9    14  7 10 15 15 8  3  Fall 2016 CS 4306 Algorithm Analysis

Example of Prim’s algorithm   A  V – A 6 12 5 9    14  7 10 15 8  3  Fall 2016 CS 4306 Algorithm Analysis

Prim’s algorithm IDEA: Maintain V – A as a priority queue Q. Key each vertex in Q with the weight of the least-weight edge connecting it to a vertex in A={v | vV-{s}-Q}. Q  V key[v]   for all v  V key[s]  0 for some arbitrary s  V. while Q   do u  EXTRACT-MIN(Q) for each v  Adj[u] do if v  Q and w(u, v) < key[v] // item in Q with minimum key, initially s then key[v]  w(u, v) [v]  u ⊳ DECREASE-KEY At the end, {(v, [v])} forms the MST. Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim Q  V key[v]   for all v  V key[s]  0 for some arbitrary s  V while Q   do u  EXTRACT-MIN(Q) for each v  Adj[u] do if v  Q and w(u, v) < key[v] then key[v]  w(u, v) [v]  u Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim Q  V key[v]   for all v  V (V) key[s]  0 for some arbitrary s  V while Q   do u  EXTRACT-MIN(Q) for each v  Adj[u] do if v  Q and w(u, v) < key[v] then key[v]  w(u, v) [v]  u Q  V (V) total Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim Q  V key[v]   for all v  V (V) key[s]  0 for some arbitrary s  V while Q   do u  EXTRACT-MIN(Q) for each v  Adj[u] do if v  Q and w(u, v) < key[v] then key[v]  w(u, v) [v]  u Q  V (V) total |V | times Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim key[v]   for all v  V key[s]  0 for some arbitrary s  V while Q   do u  EXTRACT-MIN(Q) for each v  Adj[u] Q  V (V) total |V | times degree(u) times do if v  Q and w(u, v) < key[v] then key[v]  w(u, v) [v]  u Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim key[v]   for all v  V key[s]  0 for some arbitrary s  V while Q   do u  EXTRACT-MIN(Q) for each v  Adj[u] Q  V (V) total |V | times degree(u) times do if v  Q and w(u, v) < key[v] then key[v]  w(u, v) [v]  u Handshaking Lemma  (E) implicit DECREASE-KEY’s. Fall 2016 CS 4306 Algorithm Analysis

Time = (V)·TEXTRACT-MIN + (E)·TDECREASE-KEY Analysis of Prim key[v]   for all v  V key[s]  0 for some arbitrary s  V while Q   do u  EXTRACT-MIN(Q) for each v  Adj[u] Q  V (V) total |V | times degree(u) times do if v  Q and w(u, v) < key[v] then key[v]  w(u, v) [v]  u Handshaking Lemma  (E) implicit DECREASE-KEY’s. Time = (V)·TEXTRACT-MIN + (E)·TDECREASE-KEY Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim (continued) Time = (V)·TEXTRACT-MIN + (E)·TDECREASE-KEY Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim (continued) Time = (V)·TEXTRACT-MIN + (E)·TDECREASE-KEY Q Total TEXTRACT-MIN TDECREASE-KEY Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim (continued) Time = (V)·TEXTRACT-MIN + (E)·TDECREASE-KEY TEXTRACT-MIN TDECREASE-KEY O(V) O(1) Q Total array O(V2) Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim (continued) Time = (V)·TEXTRACT-MIN + (E)·TDECREASE-KEY TEXTRACT-MIN TDECREASE-KEY O(V) O(1) O(lg V) O(lg V) Q Total array binary heap O(V2) O(E lg V) Fall 2016 CS 4306 Algorithm Analysis

Analysis of Prim (continued) Time = (V)·TEXTRACT-MIN + (E)·TDECREASE-KEY Q Total TEXTRACT-MIN TDECREASE-KEY array binary heap Fibonacci heap O(V) O(1) O(V2) O(E lg V) O(E + V lg V) worst case O(lg V) O(lg V) amortized O(lg V) O(1) amortized Fall 2016 CS 4306 Algorithm Analysis

Kruskal’s algorithm Kruskal’s algorithm (see CLRS): Uses the disjoint-set data structure (Lecture 10). Running time = O(E lg V). Best to date: Karger, Klein, and Tarjan [1993]. Randomized algorithm. O(V + E) expected time. Fall 2016 CS 4306 Algorithm Analysis

Kruskal’s algorithm Main Idea Use a priority queue using the edge weights as keys Each node starts off as it’s own tree While the queue is not empty, if the edge retrieved connects two trees, connect them, if not, discard it Once the queue is empty, you are left with the minimum spanning tree Fall 2016 CS 4306 Algorithm Analysis

Kruskal's Minimum Spanning Tree Algorithm 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 2, 5, 6, 6, 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 2, 5, 6, 6, 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 MIN 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 5, 6, 6, 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 5, 6, 6, 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 MIN 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 6, 6, 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 6, 6, 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 MIN 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 6, 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 6, 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 MIN 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 9, 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) MIN 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 11, 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 MIN 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 14, 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 MIN 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 14, 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 MIN 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 16, 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 MIN 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 18, 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 18, 19, 20, 23, 30, 44 ) 2 23 3 MIN 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Kruskal's Minimum Spanning Tree Algorithm PQ = ( 19, 20, 23, 30, 44 ) 2 23 3 9 1 18 14 2 6 6 4 30 19 11 14 5 5 6 20 16 8 7 44

Problem Set (for Exam 2) Exercises 22.1-1, 22.1-2 * Your answer must be well-organized; starts from given facts and assumptions, follow your logics line by line, and should be connected to the conclusion to support your answer. Fall 2017 CS 4306 Algorithm Analysis

Problem Set (for Exam 3) You should be able to run each MST algorithm for a given input graph. Exercises 23.2-1, 23.2-4, 23.2-5, 23.2-8 Problem 23-1 * Your answer must be well-organized; starts from given facts and assumptions, follow your logics line by line, and should be connected to the conclusion to support your answer. Fall 2017 CS 4306 Algorithm Analysis