Building Hierarchical Grid Storage Using the GFarm Global File System and the JuxMem Grid Data-Sharing Service Gabriel Antoniu, Lo ï c Cudennec, Majd Ghareeb INRIA/IRISA Rennes, France Osamu Tatebe University of Tsukuba Japan

2 Context: Grid Computing Target architecture: cluster federations (e.g. Grid’5000) Focus: large-scale data sharing Solid mechanics Thermodynamics Optics Dynamics Satellite design

3 Approaches for Data Management on Grids Use of data catalogs: Globus GridFTP, Replica Location Service, etc Logistical networking of data: IBP Buffers available across Internet Unified access to data: SRB From file-systems to tapes and databases Limitations No transparency => Increased complexity at large scale No consistency guarantees for replicated data

4 Towards Transparent Access to Data Desirable features Uniform access to distributed data via global identifiers Transparent data localization and transfer Consistency models and protocols for replicated data Examples of systems taking this approach On clusters Memory level: DSM systems (Ivy, TreadMarks, etc.) File level: NFS-like systems On grids Memory level: data sharing services JuxMem - INRIA Rennes, France File level: global file systems Gfarm - AIST/University of Tsukuba, Japan

Idea: a Collaborative Research of Memory and File-level Data Sharing Study possible interactions between The JuxMem grid data sharing service The Gfarm global file system Goal Enhance global data sharing functionality Improve performance and reliability Build a memory hierarchy for global data sharing by combining the memory level and the file system level Approach Enhance JuxMem with Persistent Storage using Gfarm Support The DISCUSS Sakura bilateral collaboration ( ) NEGST ( ) 5

6 JuxMem: a Grid Data-Sharing Service Generic grid data-sharing service Grid-scale: nodes Transparent data localization Data consistency Fault-tolerance JuxMem ~= DSM + P2P Implementation Multiple replication strategies Configurable consistency protocols Based on JXTA 2.0 ( Integrated into 2 grid programming models GridRPC (DIET, ENS Lyon) Component models (CCM & CCA) Cluster group A Juxmem group Cluster group C Cluster group B Data group D

7 JuxMem’s Data Group: a Fault-Tolerant, Self-Organizing Group GDG : Global Data Group LDG : Local Data Group LDG Client GDG LDG Data group D Data availability despite failures is ensured through replication and fault-tolerant building blocks Hierarchical self-organizing groups Cluster level: Local Data Group (LDG) Grid level: Global Data Group (GDG) Group membership Atomic multicast Consensus Failure detectors Adaptation layer Self-organizing group

8 JuxMem: Memory Model and API Memory model (currently): entry consistency Explicit association of data to locks Multiple Reader Single Writer (MRSW) juxmem_acquire, acquire_read, release Explicit lock acquire/release before/after access API Allocate memory for JuxMem data ptr = juxmem_malloc (size, #clusters, #replicas per cluster, &ID…) Map existing JuxMem data to local memory ptr = juxmem_mmap (ID), juxmem_unmap (ptr) Synchronization before/after data access juxmem_acquire(ptr), juxmem_acquire_read(ptr), juxmem_release(ptr) Read and write data: direct access through pointers! int n = *ptr; *ptr =…

Gfarm: a Global File System [CCGrid 2002] Commodity-based distributed file system that federates storage of each site It can be mounted from all cluster nodes and clients It provides scalable I/O performance wrt the number of parallel processes and users It avoids access concentration by automatic replica selection Gfarm File System /gfarm ggfjp aistgtrc file1file3 file2 file4 file1file2 File replica creation Global namespace mapping

GridFTP, samba, NFS server Compute & fs node Gfarm: a Global File System (2) Files can be shared among all nodes and clients Physically, it may be replicated and stored on any file system node Applications can access it regardless of its location File system nodes can be distributed GridFTP, samba, NFS server Gfarm metadata server Compute & fs node Client PC Note PC /gfarm metadata Gfarm file system … File A File B File C File A File B File C File B US Japan

11 Our Goal: Build a Memory Hierarchy for Global Data Sharing Approach Applications use JuxMem’s API (memory-level sharing) Applications DO NOT use Gfarm directly JuxMem uses Gfarm to enhance data persistence Without Gfarm, JuxMem supports some crashes of memory providers thanks to the self-organizing groups With Gfarm, persistence is further enhanced thanks to secondary storage How does it work? Basic principle: on each lock release, data can be flushed to Gfarm Flush frequency can be tuned to compromise efficiency/fault tolerance

12 Step 1: A Single Flush by One Provider Cluster #1Cluster #2 JuxMem Global Data Group (GDG) JuxMem Provider GDG Leader JuxMem Provider JuxMem Provider GFarm GFSD One particular JuxMem provider (GDG leader) flushes data to Gfarm Then, other Gfarm copies can be created using Gfarm’s gfrep command

13 Step 2: Parallel Flush by LDG Leaders Cluster #1Cluster #2 JuxMem Local Data Group (LDG #1) JuxMem Provider LDG #1 Leader JuxMem Provider GFarm GFSD LDG #2 Leader GFSD JuxMem Local Data Group (LDG #2) One particular JuxMem provider in each cluster (LDG leader) flushes data to Gfarm (parallel copy creation, one copy per cluster) The copies are registered as the same Gfarm file Then, extra Gfarm copies can be created using Gfarm’s gfrep command

14 Step 3: Parallel Flush by All Providers Cluster #1Cluster #2 JuxMem Global Data Group (GDG) JuxMem Provider GFarm GFSD JuxMem Provider GFSD JuxMem Provider GFSD JuxMem Provider GFSD All JuxMem providers in each cluster (LDG leader) flush data to Gfarm All copies are registered as the same Gfarm file Useful to create multiple copies of the Gfarm file per cluster No more replication using gfrep

15 Deployment issues Application deployment on large scale infrastructures Reserve resources Configure the nodes Manage dependencies between processes Start processes Monitor and clean up the nodes Mixed-deployment of GFarm and JuxMem Manage dependencies between processes of both applications Make the JuxMem provider able to act as a Gfarm client Approach: use a generic deployment tool: ADAGE (INRIA, Rennes, France) Design specific plugins for Gfarm and JuxMem

16 ADAGE: Automatic Deployment of Applications in a Grid Environment IRISA/INRIA Paris Research Group Deploy a same application on different kinds of resources from clusters to grids Support multi-middleware applications MPI+CORBA+JXTA+GFARM... Network topology description Latency and bandwidth hierarchy NAT, non-IP networks Firewalls, Asymmetric links Planner as plugin Round robin & Random Preliminary support for dynamic applications Some successes 29,000 JXTA peers on ~400 nodes 4003 components on 974 processors on 7 sites GFarm Application Description JuxMem Application Description Resource Description Generic Application Description Control Parameters Deployment Planning Deployment Plan Execution Application Configuration

17 Roadmap overview (1) Design of the common architecture : 2006 Discussions on possible interactions between JuxMem and Gfarm May 2006, Singapore (CCGRID 2006) June 2006, Paris (HPDC 2006 and NEGST workshop) October 2006: Gabriel Antoniu and Lo ï c Cudennec visited the Gfarm team First deployment tests of Gfarm on G5K Overall Gfarm/JuxMem design December 2006: Osamu Tatebe visited the JuxMem team Refinement of the Gfarm/JuxMem design Implementation of JuxMem on top of Gfarm : 2007 April 2007: Gabriel Antoniu and Lo ï c Cudennec visited the Gfarm team One JuxMem provider (GDG leader) flushes data to Gfarm after each critical section (step 1 done) Master internship: Majd Ghareeb December 2007: Osamu Tatebe visited the JuxMem team Commun paper at Euro-Par 2008

Read performance Gfarm: 69 MB/s Worst case: 39 MB/s Usual case: 100 MB/s

Write performance Gfarm: 42 MB/s Worst case: 28.5 MB/s Usual case: 89 MB/s

20 Roadmap (2) Design the Gfarm plugin for ADAGE (April 2007) Propose a specific application description language for GFarm Translate the specific description into a generic description Start processes with respect of the dependencies Transfer the Gfarm configuration files from: The Metadata Server to the Agents The Agents to their GFSD and Clients Deployment of JuxMem on top of Gfarm (May 2007) - first prototype running on G5K) ADAGE deploys Gfarm, then JuxMem (separate deployment) Limitations: the user still needs to indicate the Gfarm client hostname, the Gfarm configuration file location Design a meta-plugin for ADAGE that automatically deploys a mixed description of a Gfarm+JuxMem configuration (December 2007) Gfarm v1 and v2 Work in progress (2008) Fault-tolerant, distributed meta-data server: Gfarm on top of JuxMem Master internship: Andre Lage