Peer-to-Peer Distributed Shared Memory? Gabriel Antoniu, Luc Bougé, Mathieu Jan IRISA / INRIA & ENS Cachan/Bretagne France Dagstuhl seminar, October 2003

2 Why Are DSM Systems Interesting? Allow to share mutable data in a distributed environment Transparent access (local/remote) Transparent data localization (remote) (Lots of) consistency models and protocols Node 0Node 1 Migration ? Replication ?

3 What Do DSM Systems Usually Assume? Protocols: implicit hypotheses! Static configuration (number of nodes) Every node knows every node No node failure No dynamic node departure/arrival Homogeneous architecture (processors, OS) Designed for small-scale environment Typically clusters of workstations Node 0Node 1 Migration ? Replication ?

4 Challenge: Data Sharing on the Grid (1) Distributed numerical simulations (code coupling) Solid mechanics Thermodynamics Optics Dynamics Satellite design

5 Challenge: Data Sharing on the Grid (2) Challenge for DSM systems: get larger! Need to integrate new hypotheses Scalability Fault tolerance Dynamicity Heterogeneity Just the opposite of DSM systems!

6 Large-Scale Data Sharing: Peer-to-Peer (P2P) Systems Client Internet server Client Cache Proxy server Congestion zone Client/ Server server Client/ Server Congestion zone Client/server model Peer-to-peer model

7 Data Sharing at a Large Scale: Peer-to-Peer Systems Features: Excellent scalability: millions of nodes High volatility tolerance But: Sharing read-only data Few exceptions: Oceanstore, Ivy, etc. Question: What consistency models and protocols for a large scale, dynamic environment?

8 DSM Systems vs. P2P Systems DSMP2P Scale DynamicityNullHigh Resource homogeneity Homogeneous (clusters) Heterogeneous (Internet) Control and trust HighLow TopologyFlat Data typeMutableImmutable Typical applications Scientific computation File sharing and storage

9 Data Sharing: the Gap! DSM Small-scale Static Homogeneous P2P Large-scale Dynamic Heterogeneous 10 3 ?

10 Idea: Hybrid Approach DSM systems: consistency and transparent access P2P systems: scalability and high dynamicity DSMData Sharing ServiceP2P Scale DynamicityNullMediumHigh Resource homogeneity Homogeneous (clusters) Rather heterogeneous (clusters of clusters) Heterogeneous (Internet) Control and trust HighMediumLow TopologyFlatHierarchicalFlat Data typeMutable Immutable Typical applications Scientific computation Scientific computation and data storage File sharing and storage

11 Why Such a Service? Data sharing service for ASP environments Persistent data Transparent localization Consistency Automatic redistribution AGENT(s) S1 S2 S3S4 Client A, B, C Answer (C) S2 ! Request Op1(C, A, B) Server Op2(C, A, B) Data Sharing Service

12 A Data Sharing Service for the Grid Internet Persistence

13 A Data Sharing Service for the Grid Internet Data transfer ? Transparent data location

14 A Data Sharing Service for the Grid Internet Scalability Internet

15 A Data Sharing Service for the Grid Internet Volatility tolerance

16 JXTA: a Framework for P2P Services Open-source platform for programming P2P applications Peer Uniquely identified (ID) Address independent of physical location Multiple network access points (TCP, HTTP, etc) Peer group Peer Firewall Peer TCP/IP HTTP Peer ID Firewall

17 JuxMem: an Architecture Proposal juxmem group cluster A group cluster B group cluster C group data group Physical architecture Logical architecture

18 JuxMem API Alloc (size, attribs) Map (id, attribs) Put (id, value) Get (id) Lock (id) Unlock (id)

19 Managing Shared Data Blocks Allocate a memory block = create a data group Data blocks replicated on providers Data blocks identified by the ID of the peer group Transparent access for clients via data ID Consistency Current model: SC Assume data-race-free programs Simultaneous updates of replicas (logical multicast) Clients are not notified of updates Synchronization One lock per data block Other mechanisms: in progress

20 Handling Peer Volatility Provider volatility A manager per peer group Dynamic monitoring of available peers (cluster) Automatic replication of data blocks (data) Manager volatility Periodic exchange of heartbeats Dynamic replication of managers if needed juxmem group cluster A group cluster B group cluster C group data group

21 Implementation and Preliminary Evaluation Implementation JXTA service, 5000 Java code lines Experimental setup PentiumII: 450 Mhz and 256 MB of RAM FastEthernet 100 Mb/s Linux 2.4 Number of nodes: 20 Experiment Study provider volatility

22 Study: Provider Volatility (1) juxmem group cluster group data group Data size: one byte Replication degree = 3 Data manager not killed 1 client: 100 iterations lock-put-unlock 16 providers

23 Study: Provider Volatility (2) juxmem group cluster group data group 1 client: 100 iterations lock-put-unlock 16 providers Data size: one byte Replication degree = 3 Data manager not killed

24 Study: Provider Volatility (3) Internal locking during replication Guarantee consistency during replica creation Client is blocked juxmem group cluster group data group

25 Study: Provider Volatility (4) JXTA/Java Expensive underlying JXTA-level dynamic channel management Reconfiguration time  11 seconds Targeted volatility is weaker ( >> 80 seconds)

26 Summary A hierarchical architecture for a data sharing service for the grid Hybrid approach: DSM and P2P systems Transparent access to data blocks Persistent storage Mutable data SC memory model for DRF access Active support for peer volatility

27 Ongoing Work Studies Replication strategies for fault tolerance Consistency protocols in a dynamic environment Co-scheduling computation and data distribution Manage data-data affinity Integrate high-speed networks: Myrinet, SCI Goal: build a Grid Data Service GDS project: Extensive evaluation on realistic codes Actual execution: 100 nodes Simulation: 1,000-10,000 nodes

28 Questions?

29 Managing Memory Resources cluster group juxmem group Size: 8 MB Memory provided Provider advertisements: cluster group Cluster advertisements: juxmem group

30 Allocation: How Does It Work? 2 1 3a 3b MB?