Chapter 7: Greedy Algorithms Chapter 7: Greedy Algorithms 7.4 Finding the Shortest Path Dijkstra’s Algorithm pp. 295-301

Finding Shortest Paths Finding Shortest Paths Many real world situations can be represented using Graphs. Many real world applications and problems require finding the shortest path in a Graph. Consider the following “real world” example...

“Real World” Example 1 Ninja’s often find themselves in precarious situations where they must escape. Like when they are being chased by a Pirate disguised as Johnny Depp. A good Ninja always knows the shortest path to the escape route

Problem 1 Location of Ninja being chased by Pirate P I L F O A N U J D Problem 1 Location of Ninja being chased by Pirate 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5 The only way out crazy Pirate maze.

Initially, the distance is set to infinite and all the paths are null. A   B   C   D   E   F   G   I   J   L   N   O   P   T   U   Keeps track of the shortest distance to each vertex and the path to get there. Initially, the distance is set to infinite and all the paths are null. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Initialization: Set the start vertex to zero. A   B   C   D   E   F 0  G   I   J   L   N   O   P   T   U   Initialization: Set the start vertex to zero. Represents that you can get to F with zero cost. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A   B   C   D   E   F 0  G   I   J   L 1 F N 14 F O   P   T   U   Step 1: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. Where can you go from F. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 14 L B   C   D 13 L E   F 0  G   I 3 L J   L 1 F N 14 F O 7 L P   T   U   Step 2: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. Where can you go from F and L. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 14 L B   C   D 13 L E   F 0  G   I 3 L J   L 1 F N 14 F O 7 L P 4 I T   U   Step 3: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. Where can you go from F, L, and I 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 14 L B   C   D 13 L E   F 0  G   I 3 L J   L 1 F N 14 F O 6 P P 4 I T   U 8 P Step 4: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. Where can you go from F, L, I and P 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 14 L B   C   D 13 L E   F 0  G   I 3 L J   L 1 F N 14 F O 6 P P 4 I T   U 7 O Step 5: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. Where can you go from F, L, I, P, and O. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 14 L B   C   D 13 L E   F 0  G   I 3 L J 8 U L 1 F N 14 F O 6 P P 4 I T 9 U U 7 O Step 6: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 14 L B   C   D 11 J E   F 0  G   I 3 L J 8 U L 1 F N 14 F O 6 P P 4 I T 9 U U 7 O Step 7: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 14 L B   C   D 10 T E   F 0  G 12 T I 3 L J 8 U L 1 F N 14 F O 6 P P 4 I T 9 U U 7 O Step 8: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 12 D B 15 D C 17 D D 10 T E   F 0  G 12 T I 3 L J 8 U L 1 F N 14 F O 6 P P 4 I T 9 U U 7 O Step 9: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 12 D B 15 D C 17 D D 10 T E   F 0  G 12 T I 3 L J 8 U L 1 F N 14 F O 6 P P 4 I T 9 U U 7 O Step 10: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 12 D B 15 D C 17 D D 10 T E   F 0  G 12 T I 3 L J 8 U L 1 F N 13 A O 6 P P 4 I T 9 U U 7 O Step 11: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 12 D B 15 D C 15 N D 10 T E   F 0  G 12 T I 3 L J 8 U L 1 F N 13 A O 6 P P 4 I T 9 U U 7 O Step 12: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 12 D B 15 D C 15 N D 10 T E 16 C F 0  G 12 T I 3 L J 8 U L 1 F N 13 A O 6 P P 4 I T 9 U U 7 O Step 13: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 12 D B 15 D C 15 N D 10 T E 16 C F 0  G 12 T I 3 L J 8 U L 1 F N 13 A O 6 P P 4 I T 9 U U 7 O Step 14: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

Iterate over v’s adjacency list, and update the distances. A 12 D B 15 D C 15 N D 10 T E 16 C F 0  G 12 T I 3 L J 8 U L 1 F N 13 A O 6 P P 4 I T 9 U U 7 O Step 15: Find the unmarked vertex with minimum distance (v). Mark it as visited Iterate over v’s adjacency list, and update the distances. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

A 12 D B 15 D C 15 N D 10 T E 16 C F 0  G 12 T I 3 L J 8 U L 1 F N 13 A O 6 P P 4 I T 9 U U 7 O Final Step: This data structure can be used to compute the shortest path from F to all other vertices. 1 2 1 P I L F 14 2 4 6 13 O A 1 N 4 12 1 5 2 2 U J D C 1 3 7 2 4 1 3 5 4 1 T G B E 3 4 5

A B C D E F G I J L N O P T U L 14   13   3  1F 14 7    I 14   13   3L  14 7 4   P 14   13    14 6 4I  8 O 14   13    14 6P  7 U 14   13   8 14 9 7O J 14   11   8U 14 9 T 14   10  12 14 9U D 12 15 17 10T  12 14 G 12 15 17  12T 14 A 12D 15 17  13 N 15 15  13A C 15 15N 16 B 15D 16 B 16C P I L F O A N U J D C T G B E 1 2 4 6 12 13 14 7 5 3

“Real World” Lesson Learned Just do the Flip Out Dance to escape from the crazy Ninja maze with minimum damage.

Serious Lessons Learned Dijkstra’s algorithm is greedy. It always chooses the vertex with minimum distance from the source vertex. Optimization after each step Dijkstra’s algorithm updates the distances Shorter paths are discovered as minimum distances get updated.

Worst Case Finding the shortest path from a Start Vertex to a End Vertex could require marking every vertex. The Crazy Pirate Maze example showed us this fact. However, could we have stopped the algorithm after we marked the End Vertex as visited?

Example 2 F  0  P I L F O A N U J D C T G B E 1 9 1 5 2 4 9 13 1 4 12 1 5 2 5 U J D C 1 3 1 2 4 4 3 5 4 1 T G B E 3 4 5

Example 2 F  0  L  1 F N  5 F P I L F O A N U J D C T G B E 1 9 1 4 9 13 O A 1 N 4 12 1 5 2 5 U J D C 1 3 1 2 4 4 3 5 4 1 T G B E 3 4 5

Example 2 F  0  L  1 F N  5 F I  10 L O  10 L D  13 L A  14 L 9 1 P I L F 5 2 4 9 13 O A 1 N 4 12 1 5 2 5 U J D C 1 3 1 2 4 4 3 5 4 1 T G B E 3 4 5

Example 2 F  0  L  1 F N  5 F A  6 N I  10 L O  10 L C  10 N D  13 L 1 9 1 P I L F 5 2 4 9 13 O A 1 N 4 12 1 5 2 5 U J D C 1 3 1 2 4 4 3 5 4 1 T G B E 3 4 5

Example 2 F  0  L  1 F N  5 F A  6 N D  8 A I  10 L O  10 L C  10 N 1 9 1 P I L F 5 2 4 9 13 O A 1 N 4 12 1 5 2 5 U J D C 1 3 1 2 4 4 3 5 4 1 T G B E 3 4 5

Example 2 F  0  L  1 F N  5 F A  6 N D  8 L C  9 D I  10 L O  10 L J  11 D G  11 D T  12 D B  13 D 1 9 1 P I L F 5 2 4 9 13 O A 1 N 4 12 1 5 2 5 U J D C 1 3 1 2 4 4 3 5 4 1 T G B E 3 4 5

Question: Is 9 the minimum distance from F to C? F  0  L  1 F N  5 F A  6 N D  8 L C  9 D I  10 L I  10 L J  11 D G  11 D E  11 C T  12 D B  13 D 1 9 1 P I L F 5 2 4 9 13 O A 1 N 4 12 1 5 2 5 U J D C 1 3 1 2 4 4 3 5 4 1 T G B E 3 4 5

Question: How do we know there isn’t a shorter path? F  0  L  1 F N  5 F A  6 N D  8 L C  9 D I  10 L I  10 L J  11 D G  11 D E  11 C T  12 D B  13 D 1 9 1 P I L F 5 2 4 9 13 O A 1 N 4 12 1 5 2 5 U J D C 1 3 1 2 4 4 3 5 4 1 T G B E 3 4 5

Technicality The textbook’s version of Dijkstra’s algorithm stops when all vertices are marked. which computes the shortest distance from a Start Vertex to all the other vertices. But, if you are only interested in the shortest path from A to B, you can stop the algorithm as soon as B is marked.

Worst Case Nonetheless, there are A to B problems that require all vertices to be marked. Thus, Dijkstra’s Algorithm takes O(N) steps. Now, in the worst case, how much work does one step take?

Worst Case Step Dijkstra’s table will have N entries, one for each vertex. You have to pick the unmarked entry with minimum distance. Given an array of N numbers, how long does it take to find the minimum? What would be the overall worst case running time of Dijkstra’s algorithm consider it has to find a minimum N times? A B C D E F G I J L N O P T U 14   13  3  2  1 14 6 4I  8 Find minimum

Worst Case Step N Steps Find minimum for each step O(N2) so far... Worst Case Step N Steps Find minimum for each step O(N2) so far... A B C D E F G I J L N O P T U 14   13  3  2  1 14 6 4I  8 Find minimum

Worst Case Step Worse yet, once we find the closest vertex, we must update all the distances. It is possible that the closest vertex could be adjacent to all vertices. Thus, there could be up to N updates per step. Does this impact the Big-O?

Worst Case Example A  0 B  440 C  900 D  900 E  900 F  900 F E A Worst Case Example A  0 B  440 C  900 D  900 E  900 F  900 F E 900 900 A 900 D 900 440 B C

Worst Case Example A  0 B  440 C  630 D  880 E  880 F  880 F E A Worst Case Example A  0 B  440 C  630 D  880 E  880 F  880 F E 900 900 440 A 900 D 440 440 900 440 B C 210

Worst Case Example A  0 B  440 C  630 D  670 E  730 F  730 F E A Worst Case Example A  0 B  440 C  630 D  670 E  730 F  730 F E 100 900 100 900 440 A 900 D 440 440 900 440 40 B C 210

Worst Case Example A  0 B  440 C  630 D  670 E  680 F  690 F E A Worst Case Example A  0 B  440 C  630 D  670 E  680 F  690 F E 100 10 900 100 900 20 440 A 900 D 440 440 900 440 40 B C 210

Worst Case Example A  0 B  440 C  630 D  670 E  680 F  685 F E A Worst Case Example A  0 B  440 C  630 D  670 E  680 F  685 5 F E 100 10 900 100 900 20 440 A 900 D 440 440 900 440 40 B C 210

Worst Case Example The Problem: An update requires finding a vertex in the table. Find is O(N) Each of the N2 updates required N operations. F A B C 900 E 440 210 100 40 20 10 5 A  0 B  440 C  630 D  670 E  680 F  685 Step 1: N-1 updates Step 2: N-2 updates Step 3: N-3 updates ... Step i: N-i updates Step N-1: 1 update Step N: 0 updates D Worst case on number of updates:

Worst Case Example Note: Worst Case Example Note: The worst case can only arise in a complete graph with N(N-1)/2 edges. The number of updates is really O(M), where M is the number of edges, which is bounded by O(N2) F A B C 900 E 440 210 100 40 20 10 5 A  0 B  440 C  630 D  670 E  680 F  685 Step 1: N-1 updates Step 2: N-2 updates Step 3: N-3 updates ... Step i: N-i updates Step N-1: 1 update Step N: 0 updates D Worst case on number of updates:

Worst Case Algorithm N * (N + (N-1)*N) = O(N3) Worst Case Algorithm while there are no un-marked vertices { find the vertex v with minimum weight mark it as visited for all of v’s edges { update the minimum distance and path to each vertex adjacent to v } N vertices * Find min: O(N) N-1 edges * 1 update: requires a find: O(N) N * (N + (N-1)*N) = O(N3)

Data Structures to the Rescue Is there a data structure we can use instead of an array. Goals: Find a vertex in constant time Update a distance in constant time Find minimum as efficiently as possible

Sorting & Binary Search: Sort the array by Distance Sorting & Binary Search: Sort the array by Distance while there are no un-marked vertices { find the vertex v with minimum weight mark it as visited for all of v’s edges { update the minimum distance and path to each vertex adjacent to v } re-sort the vertices by distance } N vertices * Find min: O(1) N-1 edges * 1 update: requires a find: O(N) Sort: O(N log N) N * (1 + (N-1)*N + N*log N) = O(N3)

Sorting & Binary Search: Sort the array by Vertex Label Sorting & Binary Search: Sort the array by Vertex Label while there are no un-marked vertices { find the vertex v with minimum weight mark it as visited for all of v’s edges { update the minimum distance and path to each vertex adjacent to v } N vertices * Find min: O(N) N-1 edges * 1 update: find: O(log N) N * (N + (N-1)*log N) = O(N2 log N)

Hash Table: Constant time vertex lookup Hash Table: Constant time vertex lookup while there are no un-marked vertices { find the vertex v with minimum weight mark it as visited for all of v’s edges { update the minimum distance and path to each vertex adjacent to v } N vertices * Find min: O(N) N-1 edges * 1 update: find: O(1) N * (N + (N-1)*1) = O(N2)

N * log N + M) = O(N log N + M) Binary Heap with Hash Index: Constant time vertex lookup Find min: O(log N) Updates: O(1) while there are no un-marked vertices { find the vertex v with minimum weight mark it as visited for all of v’s edges { update the minimum distance and path to each vertex adjacent to v } N vertices * Find min: O(log N) N-1 edges * 1 update: find: O(1) N * log N + M) = O(N log N + M)

Is there a difference?

3 2 8 P I L F 4 2 4 1 2 O A 1 N 4 12 6 9 6 2 U J D C 8 3 2 2 4 8 7 2 5 8 T G B E 5 4 1