High Throughput Computing on P2P Networks Carlos Pérez Miguel

Overview High Throughput Computing Motivation All things distributed: Peer-to-peer – Non structured overlays – Structured overlays P2P Computing Cassandra HTC over Cassandra Eventual consistency Experiments Future Work Conclusions

High Throughput Computing Concept introduced by the Condor team in 1996 In contrast to HPC, it optimizes the execution of a set of applications Figure of merit: the number of computational tasks per time unit Tasks are independent Examples: Condor, Oracle Grid Engine (Kalimero), BOINC

Functioning N worker nodes One master node Users interact with the master node Master manages pending task and idle workers using a queuing system Task are (usually) executed in FIFO order

Motivations Limitations of this model – Master node may become a scalability bottleneck – Failures in the master affects the whole system Is it possible to distribute the capabilities of the master node among all sytem nodes? How? (which technology can help?)

All things distributed: peer- to-peer Distributed systems in which all nodes have the same role Nodes are interconnected defining an application-level virtual network An overlay network This overlay is used to locate other nodes and information inside them Two types of overlays: structured and non-structured

Non-structured overlays Nodes are interconnected randomly Searchs in the overlay are made by flooding Efficient search of popular contents Cannot guarantee that any system point is reachable Not efficient in terms of number of messages

Non-structured overlays (II)

Structured overlays Nodes interconnected using some kind of (regular) structure Each node has an unique ID of N bits, defining a 2 N keyspace This keyspace is divided among the nodes

Structured overlays (II) Each object in the system has an ID and a position in the key space A distance-based routing protocol is used This permits reaching any point with O(log n) messages

Distributed Hash Tables Provides a hash-like user API: Put (ID, Object) Get (ID) Fast access to distributed information Used to distribute file, communicate users, VoIP, Video Streaming

P2P Computing Must be seen by the user as a single resource pool User should be able to submit jobs from any node in the system System stores job’s information permitting progress even when the user is not connected A FIFO order should be guaranteed DHTs are suitable for this purpose

DHTs for P2P Computing Must provide scalability in adverse conditions Must provide persistency (using replication) Replicas are synchronized by consensus algorithms Load balancing algorithms are also needed

DHTs for P2P Computing (II) In 2007 Amazon presented Dynamo, a DHT P2P system with persistence, scalability, access in O(n) and eventual consistency From Dynamo, many alternatives have been proposed: Riak, Scalaris, Memcached,... Facebook proposed Cassandra in 2009 with the same Dynamo capabilities and Google's BigTable data model

Cassandra Developed by Facebook and Twitter since 2009 Has been released to the Apache Foundation Developed in Java with multilanguage client libraries Pros: Fault tolerant, decentralized, scalable, durable Cons: Eventual consistency

Cassandra’s Data Model DHTs store (key, value) pairs Cassandra store (key, (values..)) tuples across different tables The different tables are named ColumnFamilies or SuperColumnFamilies CF are 4-dimensional tables SCF are 5-dimensional tables

Column Families WaitingQueue ColumnFamily JobIDNameOwnerBinary 1Task1User1URL 2Task2User2URL 3Task3User1URL NTaskNUser3URL

SuperColumn Families Queues SuperColumn Family Waiting Job1Job2JobN Task1User1Task2User2TaskNUserN Running Job1Job2JobN Task1User1Task2User2TaskNUserN

HTC over Cassandra A batch queue system has been implemented over Cassandra’s data model This permits idle workers decide which task to run, in FIFO order Users can: – Submit jobs – Check jobs’ status – Retrieve jobs’ results The use of Cassandra as underlying data storage allows for disconnected operation

HTC over Cassandra (II) System stores Job information – Name – Owner – Binaries Users information Queues information The system is totally reconfigurable at run time, permitting the utilization of unlimited queues with different policies

Eventual Consistency All changes in any object reach all object replicas eventually CAP theorem implies that it is not possible to have these three properties at the same time: – Consistency – Availability – Partition tolerance Cassandra have selected availability and partition tolerance instead of consistency In a failure-free scenario, Cassandra provides low latency

Eventual Consistency (II) This scenario implies the impossibility of atomic operations in Cassandra In our HTC system, collisions may happen when several nodes try to execute the same task We have implemented some partial solutions that reduce the probability of a collision: – QUORUM consistency for all I/O operations – Extra queue where idle nodes compete for the waiting task Reduces the collision probability from 30% to 4%

Experiments We have performed some experiments to evaluate our system A 20 nodes cluster has been used for this purpose – Each node has a P4 processor with hyperthreading – 1.5 – 2 GB of RAM Each node represents one user in the system We have used a workload generator in order to generate a works list for each user

Metrics Bounded Slowdown: Waiting time for a job plus the running time System utilization Scheduling time: time used by idle nodes to schedule a waiting job Collisions detected

System Load

Bounded Slowdown

Scheduling Time

Collisions

Future Work Find a viable solution to the Eventual Consistency problem Develop a workflow system with MapReduce tasks Reputation systems in order to classify nodes behavior

Conclusions HTC over P2P is possible A prototype has been developed Some preliminary experiments have been done obtaining good performance levels

QUESTIONS?