Motivation Contemporary big data tools such as MapReduce and graph processing tools have fixed data abstraction and support a limited set of communication operations MPI contains abundant and highly-optimized collective communication operations but is limited on data abstractions To improve the expressiveness and performance in big data processing… We introduce Harp library, which provides data abstractions and related communication abstractions and transform map-reduce programming model to map-collecitve model.

Features Hadoop Plugin (on Hadoop 1.2.1 and Hadoop 2.2.0) Hierarchical data abstraction on arrays, key-values and graphs for easy programming expressiveness. Collective communication model to support various communication operations on the data abstractions. Caching with buffer management for memory allocation required from computation and communication BSP style parallelism Fault tolerance with check-pointing

Architecture MapReduce Applications Map-Collective Applications YARN MapReduce V2 Harp MapReduce Applications Map-Collective Applications Application Framework Resource Manager

Collective Communication Parallelism Model Shuffle M Collective Communication R Map-Collective Model MapReduce Model

Hierarchical Data Abstraction and Collective Communication Vertex Table Key-Value Partition Array Commutable Key-Values Vertices, Edges, Messages Double Array Int Array Long Array Array Partition < Array Type > Struct Object Vertex Partition Edge Partition Array Table <Array Type> Message Partition Key-Value Table Byte Array Message Table Edge Table Broadcast, Send, Gather Broadcast, Allgather, Allreduce, Regroup-(combine/reduce), Message-to-Vertex, Edge-to-Vertex Broadcast, Send Partition Basic Types

Performance on Madrid Cluster (8 nodes)