1. Parallel Graph Algorithms

2. Graph Algorithms Minimum Spanning Tree (Prim’s Algorithm)
Single-Source Shortest Path (Dijkstra’s Algorithm)
All-Pairs Shortest Paths (Dijkstra’s and Floyd’s Algorithm)

3. Adjacency Matrix
An adjacency matrix represent the edges of a graph

4. Adjacency Matrix
Example 1 2 3 4

5. Prim’s Algorithm for Minimum Spanning Tree
Prim_MST(V, E, A, r) { VT = {r}; d[r] = 0; for v in (V – VT) d[v] = Ar,v; while (VT != V) { Find a vertex v such that d[u] = min(d[v] for all v in (V – VT)); VT = VT + {u}; for v in (V – VT) { d[v] = min(d[v], Au,v); }
V – set of vertices
VT – set of vertices in the MST
E – set of edges
A – adjacency matrix
r – root node
Complexity = O(n2)

6. Root is node b (Prim’s) Initialize3
Initialize 1 f 3 b 5 c 5 1 1 2 d e 4
Since d[3] = 1, add the edge b to d and consider node d next

7. Next consider node d (Prim’s)a 3
Take Minimums except for b and d 1 f 3 b 5 c 5 1 1 2 d e 4
Since d[0] = 1, add the edge b to a and consider node a next

8. Next consider node a (Prim’s)3 1 f 3 b 5 c 5 1 1 2 d e 4
Since d[2] = 2, add the edge d to c and consider node c next

9. Next consider node c (Prim’s)a 3 1 f 3 b 5 c 5 1 1 2 d e 4
Since d[4] = 1, add the edge c to e and consider node e next

10. Next consider node e (Prim’s)a 3 1 f 3 b 5 c 5 1 1 2 d e 4
Since d[5] = 3, add the edge a to f and consider node f next

11. Next consider node f (Prim’s)a 3 1 f 3 b 5 c 5 1 1 2 d e 4
VT= V so stop

12. Parallelizing Prim’s Algorithm
We can’t just simply execute the while loop in parallel because the d[] array changes with each selection of a vertex
We have to update values in d[] from all processors after each iteration
Suppose we have n vertices in the graph and p processors

13. Parallelizing Prim’s Algorithm
Partition and adjacency matrix and the distance array (d) across processors
d[ ] n A 1 2
p-1

14. Parallelizing Prim’s Algorithm
Each processor computes the next vertex from among its vertices
A reduction is done on the distance array (d) to find the minimum
The result is broadcast out to all the processors

15. Prim’s Algorithm (Parallel)
V – set of vertices
VT – set of vertices in the MST
Vi – Vertices assigned to processor i
E – set of edges
A – adjacency matrix
r – root node
Prim_MST(V, E, A, r) { ... // Initialize d as before begin parallel region while (VT != Vi) { Find a vertex v such that d[u] = min(d[v] for all v in (Vi – VT)); VT = VT + {u}; for v in (Vi – VT) d[v] = min(d[v], Au,v); MPI_Reduce (d, ..., MPI_MIN, ...) MPI_Bcast (d, ...) } end parallel region

16. Prim’s Algorithm (Parallel)
Complexity of Parallel algorithm:
Each reduction and broadcast takes log p time, but we have to do up to n of them.
Communication
Computation

17. Dijkstra’s Algorithm for Single-Source Shortest Path
Given a source node, what is the shortest distance to each other node
The minimum spanning tree gives is this information

18. Dijkstra’s Algorithm Complexity = O(n2)
Dijkstra_SP(V, E, A, s) { VT = {s}; d[s] = 0; for v in (V – VT) l[v] = As,v; while (VT != V) { Find a vertex v such that l[u] = min(l[v] for all v in (V – VT)); VT = VT + {u}; l[v] = min(l[v], l[u] + Au,v); }
V – set of vertices
VT – set of vertices in the MST
E – set of edges
A – adjacency matrix
s – source node
Complexity = O(n2)
This is the only thing different

19. Source Node is node b (Dijkstra’s)3
Initialize 1 f 3 b 5 c 5 1 1 2 d e 4
Since l[3] = 1, consider node d next

20. Next consider node d (Dijkstra’s)3 1 f 3 b 5 c 5 1 1 2 d e 4
Since l[0] = 1, consider node a next

21. Next consider node a (Dijkstra’s)3 1 f 3 b 5 c 5 1 1 2 d e 4
Since l[2] = 3, consider node c next

22. Next consider node c (Dijkstra’s)3 1 f 3 b 5 c 5 1 1 2 d e 4
Since l[4] = 4, consider node e next

23. Next consider node e (Dijkstra’s)3 1 f 3 b 5 c 5 1 1 2 d e 4
Since d[5] = 4, add the edge a to f and consider node f next

24. Next consider node f (Dijkstra’s)3 1 f 3 b 5 c 5 1 1 2 d e 4
VT= V so stop

25. Parallelizing Dijkstra’s Algorithm
Since Dijkstra’s Algorithm and Prim’s Algorithm are essentially the same, we can parallelize them the same way:
Complexity of Parallel algorithm:
If we have n processors, this becomes:
Communication
Computation

26. All Pairs Shortest Path
Dijkstra’s Algorithm gives us the shortest path from a particular node to all the others
For All Paris Shortest Path, we want to find the shortest path between all pairs of vertices
We can apply Dijkstra’s Algorithm to every pair of vertices
Complexity = O(n3)

27. All Pairs using Dijkstra’s Algorithm
Dijkstra_APSP(V, E, A) { for s in V { VT = {s}; d[s] = 0; for all v in (V – VT) l[v] = As,v; while (VT != V) { Find a vertex v such that l[u] = min(l[v] for all v in (V – VT)); VT = VT + {u}; for v in (V – VT) l[v] = min(l[v], l[u] + Au,v); }
V – set of vertices
VT – set of vertices in the MST
E – set of edges
A – adjacency matrix
s – source node
Complexity = O(n3)

28. All Pairs Shortest Path
We can parallelize the outermost loop
Each processors assumes a different node vi and computes the shortest path to all nodes
No communication if needed
Complexity is O(n3/p)
If we have n processors, complexity is O(n2)
If we have n2 processors, we can use n processors for each vertex. Complexity becomes O(nlogn)

29. Floyd’s Algorithm for All Pairs Shortest Path
Floyd’s Algorithm works off of this observation:
Consider a subset of V:
Let be the weight of the shortest path from vi to vj that includes one of the vertices in
If vk is not in the shortest path from vi to vj, then
Otherwise, the shortest path is

30. Floyd’s Algorithm for All Pairs Shortest Path
This leads to the following recurrence:
We can implement this using iteration and not recursion

31. All Pairs using Floyd’s Algorithm
Floyd_APSP(V, E, A) {
d0i,j = Ai,j for all i,j
for k = 1 to n
for i = 1 to n
for j = 1 to n
d(k)i,j = min(d(k-1)i,j , d(k-1)i,k + d(k-1)k,j )
We don’t need n copies of the d matrix. We only need one.
In fact, we can do it with only one matrix
V – set of vertices
E – set of edges
A – adjacency matrix
Complexity = O(n3)

32. Partitioning of the d matrix…
We divide the d matrix into p blocks of size n/√p
Each processor is responsible for n2/√p elements of the d matrix

33. Partitioning of the d matrix
k column
j column
However, we have to send data between processors
k row
i row

34. Communication Pattern…

35. Analysis of Floyd’s Algorithm
Each processor has to send its block to all processors on the same row and column.
If we use a broadcast, then the time to communication is
The synchronization step requires
The time to compute the values for each processors is

36. Analysis of Floyd’s Algorithm
So the complexity for each step is:
And finally, the complexity for n steps (of the k loop) is:
Communication
Computation

37. A faster version of Floyd’s Algorithm
We can do a pipeline of values moving through the matrix.
The reason is because once processor pi, j computes the value of it can then send it to the processors pi, j-1 , pi, j+1 , pi+1, j , and pi-1, j

38. Consider the movement of the value computed by processor 4
Time t
t+1
t+2
t+3
t+4 1 2 3 4 5 6 7 8
Processors

39. Analysis of Floyd’s Algorithm with pipelining
The net complexity of the algorithm using pipelining is:
Communication
Computation

40. Questions