1. Mayank Bhatt, Jayasi Mehar
Topology-Aware Distributed Graph Processing for Tightly-Coupled Clusters
Mayank Bhatt, Jayasi Mehar
DPRG:

2. Our work explores the problem of graph partitioning, focused on reducing the communication cost on tightly coupled clusters

3. Why? Experimenting with cloud frameworks on HPC systems
Interest in supercomputing as a service
More big data jobs running on supercomputers

4. Tightly-Coupled Clusters
Supercomputers
Compute nodes embedded inside the network topology
Messages routed via compute nodes
Communication patterns can influence performance
“Hop count” is an approximate measure of cost of communication

5. Blue Waters Interconnect
3D Torus
Subset of nodes returned for running job
Static routing - number of hops between two nodes will remain constant

6. Graph Processing Systems
Lot of real world data is expressed in the form of graphs
Billion of vertices, trillions of edges, need to distribute
Algorithms - ex. Shortest path, PageRank
2 stages - Ingress and Processing

7. Types of Partitioning System of choice: PowerGraph Masters and Mirrors
Masters communicate with all mirrors
Our hypothesis: placing masters and mirrors close by should reduce communication cost
Vertex Cuts
Edge Cuts

8. Master mirror placement
Place replicas of a vertex first and then decide where to place the master
Place the master of each vertex first and then decide where to place the replica - Hashing M R M R M

9. Random Partitioning Fast ingress
Communication cost between master and mirrors can be high
Replication factor could be high M R R

10. Oblivious Partitioning
Slower ingress
Heuristic based partitioning
Leads to smaller replication factor than random
Starting point to optimize Master mirror communication M R

11. Grid Partitioning Intersecting constraint sets
Leads to a controlled replication factor
Master mirror communication not optimized M R

12. Topology Aware Variants
Make the partitioning step aware of the underlying network topology
Place masters and mirrors such that communication cost is minimized

13. Choosing a master
Pick master such that total number of hops are minimum
Geometric centroid
Edge degrees of each replica can be different
Weighted Centroid

14. Grid Centroid
Edges are placed using the Grid partitioning Strategy first
Load: number of masters on candidate
Number of edges on mirror
Number of hops between mirror and candidate

15. Restricted Oblivious

16. Restricted Oblivious Number of edges on candidate
Maximum number of edges on a node
Minimum number of edges on a node
Number of hops between candidate and master

17. Experiments Cluster size: 36 nodes Algorithm: Approximate diameter
Graph: Power-law, 20 million vertices

18. Tradeoff between runtime and ingress

19. Data intensive algorithms benefit more
Graph Algorithms
Data intensive algorithms benefit more

20. Improvements depend on type of graph
Graph Type
Improvements depend on type of graph

21. Network Data Transfer

22. Other System Optimizations
Controlling the frequency of data injection into network impacts runtime in certain algorithms
Smaller network buffers => flushed more frequently

23. Small computation and network data benefit from frequent flushing
Buffer Sizes
PageRank
Approximate Diameter
Small computation and network data benefit from frequent flushing

24. Decisions, decisions

25. DPRG: http://dprg.cs.uiuc.edu
Conclusions
Two new topology-aware algorithms for graph partitioning
No ‘one size fits all’ approach to graph partitioning
We propose a decision tree that can help decide which partitioning algorithm is best
System optimizations complement performance
DPRG:

26. Questions and Feedback?
DPRG: