Pregel: A System for Large-Scale Graph Processing Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik, James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkwoski Google, Inc. SIGMOD ’10 15 Mar 2013 Dong Chang

Outline  Introduction  Computation Model  Writing a Pregel Program  System Implementation  Experiments  Conclusion & Future Work 2

Outline  Introduction  Computation Model  Writing a Pregel Program  System Implementation  Experiments  Conclusion & Future Work 3

Introduction (1/2) 4

Introduction (2/2) 5  Many practical computing problems concern large graphs  MapReduce is ill-suited for graph processing –Many iterations are needed for parallel graph processing –Materializations of intermediate results at every MapReduce iteration harm performance Large graph data Web graph Transportation routes Citation relationships Social networks Graph algorithms PageRank Shortest path Connected components Clustering techniques

MapReduce Execution 6  Map invocations are distributed across multiple machine s by automatically partitioning the input data into a set o f M splits.  The input splits can be processed in parallel by different machines  Reduce invocations are distributed by partitioning the int ermediate key space into R pieces using a hash function: hash(key) mod R – R and the partitioning function are specified by the pr ogrammer.

MapReduce Execution 7 / 40

Data Flow  Input, final output are stored on a distributed file system – Scheduler tries to schedule map tasks “close” to physical storage location of input data  Intermediate results are stored on local file system of ma p and reduce workers  Output can be input to another map reduce task 8 / 40

MapReduce Execution 9 / 40

MapReduce Parallel Execution 10 / 40

Outline  Introduction  Computation Model  Writing a Pregel Program  System Implementation  Experiments  Conclusion & Future Work 11

Computation Model (1/3) 12 Input Output Supersteps (a sequence of iterations)

Computation Model (2/3) 13  “Think like a vertex”  Inspired by Valiant’s Bulk Synchronous Parallel model (1990) Source:

Computation Model (3/3) 14  Superstep: the vertices compute in parallel –Each vertex  Receives messages sent in the previous superstep  Executes the same user-defined function  Modifies its value or that of its outgoing edges  Sends messages to other vertices (to be received in the next superstep)  Mutates the topology of the graph  Votes to halt if it has no further work to do –Termination condition  All vertices are simultaneously inactive  There are no messages in transit

An Example 15 / 40

Example: SSSP – Parallel BFS in Pregel 16 0    

Example: SSSP – Parallel BFS in Pregel 17 0            

Example: SSSP – Parallel BFS in Pregel  

Example: SSSP – Parallel BFS in Pregel  

Example: SSSP – Parallel BFS in Pregel

Example: SSSP – Parallel BFS in Pregel

Example: SSSP – Parallel BFS in Pregel

Example: SSSP – Parallel BFS in Pregel

Example: SSSP – Parallel BFS in Pregel

Differences from MapReduce 25  Graph algorithms can be written as a series of chained MapReduce invocation  Pregel –Keeps vertices & edges on the machine that performs computation –Uses network transfers only for messages  MapReduce –Passes the entire state of the graph from one stage to the next –Needs to coordinate the steps of a chained MapReduce

Outline  Introduction  Computation Model  Writing a Pregel Program  System Implementation  Experiments  Conclusion & Future Work 26

C++ API 27  Writing a Pregel program –Subclassing the predefined Vertex class Override this! in msgs out msg

Example: Vertex Class for SSSP 28

Outline  Introduction  Computation Model  Writing a Pregel Program  System Implementation  Experiments  Conclusion & Future Work 29

MapReduce Coordination  Master data structures – Task status: (idle, in-progress, completed) – Idle tasks get scheduled as workers become available – When a map task completes, it sends the master the l ocation and sizes of its R intermediate files, one for ea ch reducer – Master pushes this info to reducers  Master pings workers periodically to detect failures 30 / 40

Mapreduce Failures  Map worker failure –Map tasks completed or in-progress at worker are reset to idle –Reduce workers are notified when task is rescheduled on another worker  Reduce worker failure –Only in-progress tasks are reset to idle  Master failure –MapReduce task is aborted and client is notified 31 / 40

System Architecture 32  Pregel system also uses the master/worker model –Master  Maintains worker  Recovers faults of workers  Provides Web-UI monitoring tool of job progress –Worker  Processes its task  Communicates with the other workers  Persistent data is stored as files on a distributed storage system (such as GFS or BigTable)  Temporary data is stored on local disk

Execution of a Pregel Program 33 1.Many copies of the program begin executing on a cluster of machines 2.The master assigns a partition of the input to each worker –Each worker loads the vertices and marks them as active 3.The master instructs each worker to perform a superstep –Each worker loops through its active vertices & computes for each vertex –Messages are sent asynchronously, but are delivered before the end of the superstep –This step is repeated as long as any vertices are active, or any messages are in transit 4.After the computation halts, the master may instruct each worker to save its portion of the graph

Fault Tolerance 34  Checkpointing –The master periodically instructs the workers to save the state of their partitions to persistent storage  e.g., Vertex values, edge values, incoming messages  Failure detection –Using regular “ping” messages  Recovery –The master reassigns graph partitions to the currently available workers –The workers all reload their partition state from most recent available checkpoint

Outline  Introduction  Computation Model  Writing a Pregel Program  System Implementation  Experiments  Conclusion & Future Work 35

Experiments 36  Environment –H/W: A cluster of 300 multicore commodity PCs –Data: binary trees, log-normal random graphs (general graphs)  Naïve SSSP implementation –The weight of all edges = 1 –No checkpointing

Experiments 37  SSSP – 1 billion vertex binary tree: varying # of worker tasks

Experiments 38  SSSP – binary trees: varying graph sizes on 800 worker tasks

Experiments 39  SSSP – Random graphs: varying graph sizes on 800 worker tasks

Outline  Introduction  Computation Model  Writing a Pregel Program  System Implementation  Experiments  Conclusion & Future Work 40

Conclusion & Future Work 41  Pregel is a scalable and fault-tolerant platform with an API that is sufficiently flexible to express arbitrary graph algorithms  Future work –Relaxing the synchronicity of the model  Not to wait for slower workers at inter-superstep barriers –Assigning vertices to machines to minimize inter-machine communication –Caring dense graphs in which most vertices send messages to most other vertices

Thank You!