Distributed Graph Algorithms Colorado State University Distributed Graph Algorithms Wade Lawrence, Roger Marquez, Michael Miller CS 455 Introduction to Distributed Systems Dr. Shrideep Pallickara

Why is this problem important? Large graphs common in real-world applications Social networks Navigation We want to be able to run calculations on these graphs Graphs can have hundreds of millions of vertices and edges Many graph problems are NP hard

Problem Characterization Execution environment Optimizing for: Storage, e.g. state of system Processor utilization, e.g. message passing In the context of: Data centers (renting machines) Wireless sensors (battery lifetime)

Trade-off Space Distribution of algorithms across a system: time consuming. needs to be repeated for every algorithm. Can use Frameworks: Simplified distribution. Imperfect. Many challenges introduced In DHTs: Local state vs lookup time Routing payloads: Shortest topological path Least congested path Some Examples of tradeoffs that occur in systems that implement DGAs.

Dominant Approaches Part 1 Frameworks: Hadoop, LightGraph, Pregel, and many more. Provide ability to distribute algorithms with ease PowerGraph High concurrency using distributed locking Good partitioning of power law graphs LightGraph Improved PowerGraph Less internode communication Better partitioning

Dominant Approaches Part 2 Algorithms: No dominant approach Depends on intended application Gallager, Humblet, Spira algorithm Algorithm developed to find Min. Spanning Tree Many algorithms built off of this algorithm New algorithms target weakness of intended system. Example: Some algorithms for a battery based system limit the number of messages being passed. Increases the longevity of the system.

Insights Gleaned No blanket solution: Many unique challenges for every system Many algorithms exist to produce results Not many handle specific challenges. Need to be very careful with choices of algorithm or framework Frameworks Provide benefits at costs No simplification is free Algorithms Choice dependent on system. Must be careful regarding this choice.

Future Problem Space Advances in technology Remove many of previous limitations Example: batteries are more powerful, last longer Processors have more computation power for similar energy costs as today. Not as much focus on what the system is limited to Focused on what it should do. Advancing past limitations fixes some problems However, it creates many more Too many choices complicates things

Future Trade-offs and Solutions Improved Networking Focus on faster, more reliable communication as well as reduced need for communication Battery life Not as large a concern as it was in the past Reducing importance of individual machines Distributed settings increase processing power rather than hardware improvements