National Institute of Advanced Industrial Science and Technology Gfarm v2: A Grid file system that supports high-performance distributed and parallel data computing Osamu Tatebe 1, Noriyuki Soda 2, Youhei Morita 3, Satoshi Matsuoka 4, Satoshi Sekiguchi 1 1 Grid Technology Research Center, AIST 2 SRA, Inc, 3 KEK, 4 Tokyo Institute of Technology / NII CHEP 04 Sep 27, 2004 Interlaken, Switzerland

National Institute of Advanced Industrial Science and Technology [Background] Petascale Data Intensive Computing Detector for ALICE experiment Detector for LHCb experiment High Energy Physics CERN LHC, KEK-B Belle ~MB/collision, 100 collisions/sec ~PB/year 2000 physicists, 35 countries Astronomical Data Analysis data analysis of the whole data TB~PB/year/telescope Subaru telescope 10 GB/night, 3 TB/year

National Institute of Advanced Industrial Science and Technology Petascale Data-intensive Computing Requirements Peta/Exabyte scale files, millions of millions of files Scalable computational power > 1TFLOPS, hopefully > 10TFLOPS Scalable parallel I/O throughput > 100GB/s, hopefully > 1TB/s within a system and between systems Efficiently global sharing with group-oriented authentication and access control Fault Tolerance / Dynamic re-configuration Resource Management and Scheduling System monitoring and administration Global Computing Environment

National Institute of Advanced Industrial Science and Technology Goal and feature of Grid Datafarm Goal Dependable data sharing among multiple organizations High-speed data access, High-performance data computing Grid Datafarm Gfarm File System – Global dependable virtual file system Federates scratch disks in PCs Parallel & distributed data computing Associates Computational Grid with Data GridFeatures Secured based on Grid Security Infrastructure Scalable depending on data size and usage scenarios Data location transparent data access Automatic and transparent replica selection for fault tolerance High-performance data access and computing by accessing multiple dispersed storages in parallel (file affinity scheduling)

National Institute of Advanced Industrial Science and Technology Grid Datafarm (1): Gfarm file system - World- wide virtual file system [CCGrid 2002] Transparent access to dispersed file data in a Grid POSIX I/O APIs Applications can access Gfarm file system without any modification as if it is mounted at /gfarm Automatic and transparent replica selection for fault tolerance and access-concentration avoidance Gfarm File System /gfarm ggfjp aistgtrc file1file3 file2 file4 file1file2 File replica creation Virtual Directory Tree mapping File system metadata

National Institute of Advanced Industrial Science and Technology Grid Datafarm (2): High-performance data access and computing support [CCGrid 2002] Do not separate Storage and CPU Parallel and distributed file I/O

National Institute of Advanced Industrial Science and Technology Scientific Application ATLAS Data Production Distribution kit (binary) Atlfast – fast simulation Input data stored in Gfarm file system not NFS G4sim – full simulation (Collaboration with ICEPP, KEK) Belle Monte-Carlo Production 30 TB data needs to be generated 3 M events (60 GB) / day is being generated using a 50-node PC cluster Simulation data will be generated distributedly in tens of universities and KEK (Collaboration with KEK, U-Tokyo)

National Institute of Advanced Industrial Science and Technology Gfarm TM v1 Open source development Gfarm TM version released on July 5, 2004 ( ) scp, GridFTP server、samba server,... Application Gfarm library Metadata server CPU... gfsd ＊ Existing applications can access Gfarm file system without any modification using LD_PRELOAD gfmdslapd Compute and file system nodes

National Institute of Advanced Industrial Science and Technology Problems of Gfarm TM v1 Functionality of file access File open in read-write mode*, file locking (* supported in version 1.0.4)Robustness Consistency between metadata and physical file at unexpected application crash at unexpected modification of physical filesSecurity Access control of filesystem metadata Access control of files by group File model of Gfarm file - group of files (collection, container) Flexibility of file grouping

National Institute of Advanced Industrial Science and Technology Design of Gfarm TM v2 Supports more than ten thousands of clients and file server nodes Provides scalable file I/O performance Gfarm v2 – towards *true* global virtual file system POSIX compliant - supports read-write mode, advisory file locking,... Robust, dependabe, and secure Can be substituted for NFS, AFS,...

National Institute of Advanced Industrial Science and Technology Related work (1) Lustre >1,000 clients Object (file) based management, placed in any OST No replica management, Writeback cache, Collaborative read cache (planned) GSSAPI, ACL, StorageTek SFS Kernel module

National Institute of Advanced Industrial Science and Technology Related work (2) Google File System >1,000 storage nodes Fixed-size chunk, placed in any chunkserver by default, three replicas User client library, no client and server cache not POSIX API, support for Google ’ s data processing needs [SOSP’03]

National Institute of Advanced Industrial Science and Technology Opening files in read-write mode (1) Semantics (the same as AFS) [without advisory file locking] Updated content is available only when opening the file after a writing process closes the file [with advisory file locking] Among processes that locks a file, up-to-date content is available in the locked region. This is not ensured when a process writes the same file without file locking.

National Institute of Advanced Industrial Science and Technology Opening file in read-write mode (2) /grid ggfjp file1file2 Process 1Process 2 fopen(“/grid/jp/file2”, “rw”)fopen(“/grid/jp/file2”, “r”) Metadata server FSN1 FSN2 file2 FSN1 File access file2 FSN2 File access fclose() Before closing, any file copy can be accessed Delete invalid file copy in metadata, but file access is continued fclose()

National Institute of Advanced Industrial Science and Technology Advisory file locking /grid ggfjp file1file2 Process 1Process 2 fopen(“/grid/jp/file2”, “rw”)fopen(“/grid/jp/file2”, “r”) Metadata server FSN1 FSN2 file2 FSN1 File access file2 FSN2 File access Read lock request FSN1 File access Cache flush Disable caching

National Institute of Advanced Industrial Science and Technology Consistent update of metadata (1) Application Gfarm library Metadata server File system node FSN1 open FSN1 close Update metadata Metadata is not updated at unexpected application crash Gfarm v1 – Gfarm library updates metadata

National Institute of Advanced Industrial Science and Technology Consistent update of metadata (2) Application Gfarm library Metadata server File system node FSN1 open FSN1 close or broken pipe Update metadata Metadata is updated by file system node even at unexpected application crash Gfarm v2 – file system node updates metadata

National Institute of Advanced Industrial Science and Technology Generalization of file grouping model files N sets Image files taken by Subaru telescope 10 files executed in parallel N files executed in parallel 10 x N files executed in parallel

National Institute of Advanced Industrial Science and Technology File grouping by directory night1 shot1shot2 ccd0 ccd1 ccd9 shotN... night1-ccd1 shot1shot2 shotN... Symlink/hardlink to night1/shot2/ccd1 gfs_pio_open(“night1/shot2”, &gf)Open a Gfarm file that concatenates ccd0,..., ccd9 gfs_pio_set_view_section(gf, “ccd1”) Set file view to ccd1 section gfs_pio_open(“night1”, &gf)Open a Gfarm file that Concatenates shot1/ccd0,..., and shotN/ccd9

National Institute of Advanced Industrial Science and Technology Summary and future work Gfarm TM v2 aims at global virtual file system having Scalability up to more than ten thousands of clients and file system nodes Scalable file I/O performance POSIX complience (read-write mode, file locking,...) Fault tolerance, robustness and dependability. Design and implementation is discussed Future work Implementation and performance evaluation Evaluation for scalability up to more than ten thousands of nodes Data preservation, automatic replica creation