Grid File System WG – GGF 17 Arun Jagatheesan San Diego Supercomputer Center GGF 17 May 11, 2006 Tokyo, Japan

2 Global Grid Forum-17, TokyoGrid File System Group IP & © The GGF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the GGF Secretariat. The GGF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this recommendation. Please address the information to the GGF Executive Director. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the GGF or other organizations, except as needed for the purpose of developing Grid Recommendations in which case the procedures for copyrights defined in the GGF Document process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the GGF or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE GLOBAL GRID FORUM DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE." Intellectual Property Statement Copyright (C) Global Grid Forum (2004). All Rights Reserved.

3 Global Grid Forum-17, TokyoGrid File System Group Talk Outline Our original charter Status What we reported in GGF-16 What we have now at GGF-17 What we will have for GGF-18 GFS Architecture What, why, how… Summary

4 Global Grid Forum-17, TokyoGrid File System Group Grid File System WG Standard mechanism to describe and organize file-based data in grids Provide a higher level, human readable logical namespace of data and resources in a data grid An architecture that would allow vendors and users to distinguish compatible products that are “GGF GFS complaint”

5 Global Grid Forum-17, TokyoGrid File System Group Example: GFS Namespace /grid ggfjp aistgtrc file1file3 file2 pub file1file2 ftp://meti.go.jp/gsiftp://aist.go.jp/ /a/b/file2 /d/file1 /a/b/file1 /a/b/file3 /b/file1 /c/file3 /pub/ GFS Service File Transport Service Hierarchical namespace with filesystem metadata Unique path name Access control Write lock Other metadata

6 Global Grid Forum-17, TokyoGrid File System Group GFS-WG Status reported in GGF-16 Informational Document : GFS-Architecture Approved now as a GFD-61 That was only version – 1 of the Architecture workbook RNS Specification Under discussion, public review (?) More discussion on this session too New Secretary Christopher Jordon External Alliances SNIA, IETF NFSv4 (??)

7 Global Grid Forum-17, TokyoGrid File System Group GFS-WG Status as of GGF-17 (NOW) Informational Document : GFS-Architecture Approved now as a GFD-61 ( Working on next version with end-to-end architecture Focus will be on “GFS complaint software” for vendors and users RNS Specification Re-factored version of RNS document - public discussion GFSG and community to decide whether incremental progress can be made on this initial draft or wait for community consensus. External Alliances SNIA – Currently a group here in GGF. Look forward to working with our SNIA friends. IETF NFSv4 : We had initial discussions. But, no follow up was made by our GFS as we still need to know more (and less resources to spread out).

8 Global Grid Forum-17, TokyoGrid File System Group What will we have by GGF –18 ? Proposed Deliverables RNS spec finalized OGSA - GFS use case GGF Architecture Workbook version 2 GGF Architecture Standard document – Skeleton Topic not limited to… Data Grid Composition of services System and User-defined Meta-data in GFS GFS Profile Why and what “GGF GFS complaint” product or service Work with our Storage friends in GGF and SNIA

9 Global Grid Forum-17, TokyoGrid File System Group Talk Outline Our original charter Status What we reported in GGF-16 What we have now at GGF-17 What we will have for GGF-18 GFS Architecture What, why, how… Summary

10 Global Grid Forum-17, TokyoGrid File System Group GFS Grid Resources Context (Information) Information about digital entities (location, size, owners,..) Relationship between digital entities (replicas, collection,.) Behavior of the digital entities (services) Content (Data) Structured and unstructured Virtual or derived Commodity (Producers and consumers) Physical storage resources and file systems Logical resources

11 Global Grid Forum-17, TokyoGrid File System Group GFS Resource Providers GFS Resource Providers (GRP) providing content and/or storage GRP /txt3.txt GRP

12 Global Grid Forum-17, TokyoGrid File System Group GFS Administrative Domain GRP GFS Administrative domain with one or more GFS Resource Providers Could include their data centers SNIA note: This is at very higher level than block, file/record level. Yet deals with objects /txt3.txt GRP Research Lab

13 Global Grid Forum-17, TokyoGrid File System Group GFS Administrative domains /…/text1.txt /…//text2.txt GRP /txt3.txt GRP Storage-R-Us Resource Providers data + storage (50) Finance Department data + storage (40) Research Lab data + storage (10)

14 Global Grid Forum-17, TokyoGrid File System Group Grid File System /…/text1.txt /…//text2.txt GRP /txt3.txt GRP Storage-R-Us Resource Providers data + storage (50) Finance Department data + storage (40) Research Lab data + storage (10) /home/arun.sdsc/exp1 /home/arun.sdsc/exp1/text1.txt /home/arun.sdsc/exp1/text2.txt /home/arun.sdsc/exp1/text3.txt data + storage (100) Logical Namespace (Need not be same as physical namespace )

15 Global Grid Forum-17, TokyoGrid File System Group GFS Client Server Software Stack GFS Server (code) GFS WS-Client GFS Client Protocol Traditional FS clients Users see the same namespace. But, the functionalities and data types supported may differ /home/arun.sdsc/exp1 /home/arun.sdsc/exp1/text1.txt /home/arun.sdsc/exp1/text2.txt /home/arun.sdsc/exp1/text3.txt data + storage (100) /…/text1.txt /txt3.txt

16 Global Grid Forum-17, TokyoGrid File System Group GFS Architecture Standard Interfaces (?) GFS Server (code) GFS WS-Client GFS Client Protocol Traditional FS clients /…/text1.txt /txt3.txt Is the user interface to access GFS a standard one? Is the server interface by each vendor a standard? (This will allow server-to- server peer communication

17 Global Grid Forum-17, TokyoGrid File System Group Questions for GFS Architecture Logical resource namespace Logical resources have been popular Work with existing standards Without loosing advanced functionalities Vendor interoperability Single interface Composed from others (services) A single interface of all functionalities (out of the box) Block level protocols Bulk operations, data manipulation, …

18 Global Grid Forum-17, TokyoGrid File System Group World Wide Datagrid

19 Global Grid Forum-17, TokyoGrid File System Group What could be done? Standards with functionalities or operations used by users in production Collaborative logical namespace management Inter/Intra/Multi Organizational namespace Logical storage resources (no mount points) Metadata, annotation support for discovery Bulk operations User groups and User access control Automated data integrity checks

20 Global Grid Forum-17, TokyoGrid File System Group Suggestions, comments Self Assessment Need to restart bi-weekly telecon (at least monthly) F2F discussions (or polycom) More people to work or just provide input than be passive observers (especially from vendors, we got users) Suggestions Focus on what works and what is used in production Products and standards emerge based on what works and user needs Flexibility of the WG’s to refocus the charter along with other production groups or users [what is out there and required]

21 Global Grid Forum-17, TokyoGrid File System Group Summary GFS-WG results delivered, has to evolve more Data Grids : Multiple projects and production users world-wide Multiple common features and patterns observed in academic usage Vendor or Industry Interest for similar commercial use cases (work with them also) GFS-WG more deliverables Conference calls, Architecture, re-factored documents

22 Global Grid Forum-17, TokyoGrid File System Group Acknowledgement RNS, Active Contributors: Arun Jagatheesan, SDSC Christopher Jordon, SDSC Manuel Pereira, IBM Almaden Research Center Osamu Tatebe, University of Tsukuba Architecture Contributors: Ted Anderson (IBM Almaden) Cameron Bahar (Storage Machines) Leo Luan (IBM Almaden) Reagan Moore (SDSC) Ken Wood (Hitachi Data Systems) Jane Xu (IBM Storage Software) Alan Yoder (Network Appliance)