Grid File System WG & Architecture Arun Jagatheesan San Diego Supercomputer Center (SDSC) Global Grid Forum Washington DC

IP & © Intellectual Property Statement The GGF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the GGF Secretariat. The GGF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this recommendation. Please address the information to the GGF Executive Director. Copyright (C) Global Grid Forum (2004). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the GGF or other organizations, except as needed for the purpose of developing Grid Recommendations in which case the procedures for copyrights defined in the GGF Document process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the GGF or its successors or assigns.   This document and the information contained herein is provided on an "AS IS" basis and THE GLOBAL GRID FORUM DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE." This has to be displayed on all GGF presenatations

Agenda for GGF-18 session Introduction and Status Report (10 minutes) GFS Profile discussion (30 minutes) GFS Architecture discussion (20 minutes) OGSA-GFS use case and external interactions GFS User/Provider Community discussion Action Items

Talk Outline GFS Introduction GFS GGF-18 Status ------------------------------------------ GFS Architecture Continued Our Target Issues Introduced Work to be done Summary

Example: GFS Namespace /grid GFS Service Hierarchical namespace with filesystem metadata Unique path name Access control Write lock Other metadata ggf jp aist gtrc file1 file2 file1 file2 file3 pub gsiftp://aist.go.jp/ http://u-tokyo.ac.jp/ ftp://meti.go.jp/ File Transport Service /a/b/file3 /a/b/file1 /a/b/file2 /b/file1 /c/file3 /d/file1 /pub/

GFS Resource Providers I start with by introducing the reality. We have some data and storage on some disk/ hierarchical resource. IT can have its own directories or physical names for the data. These are Grid Resource providers from the GFS perspective. Later these resource providers could be other sources like service registries or any thing dealing with a naming system or catalog. It is highly tempting to say that GFS Resource providers are nothing but SRM – at least looks for me to be so. But please refrain from doing that. We need to have the GSM also agree to this picture. GFS Resource Providers (GRP) providing content and/or storage /txt3.txt GRP GRP

GFS Administrative Domain GFS Administrative domain with one or more GFS Resource Providers Could include their data centers SNIA note: This is at very higher level than block, file/record level. Yet deals with objects Research Lab Multiple such resource provides could be present in an organization or autonomous administrative domain as we called before. /txt3.txt GRP GRP

GFS Administrative domains Research Lab data + storage (10) Storage-R-Us Resource Providers data + storage (50) Finance Department data + storage (40) There could be multiple administrative domains like this that makes up an virtual enterprise or what we call as the Grid GRP /txt3.txt GRP GRP GRP /…/text1.txt /…//text2.txt

Grid File System /home/arun.sdsc/exp1 /home/arun.sdsc/exp1/text1.txt data + storage (100) Logical Namespace (Need not be same as physical namespace ) Research Lab data + storage (10) Storage-R-Us Resource Providers data + storage (50) Finance Department data + storage (40) The resources (data + storage) from all these autonomous administrative domains could be presented as an logical ensemble. The logical namespace created need not reflect the physical layout. Also the logical names do not have any location dependency GRP /txt3.txt GRP GRP GRP /…/text1.txt /…//text2.txt

BUT… We had this for GGF 17 status too GGF-18 Status (NOW) BUT… We had this for GGF 17 status too Informational Document : GFS-Architecture Approved now as a GFD-61 (http://www.ggf.org/documents/GFD.61.pdf) Working on next version with end-to-end architecture Focus is on “GFS complaint software/interfaces” for vendors and users RNS Specification Re-factored version of RNS document External Alliances SNIA – Currently a group here in GGF. Look forward to working with our SNIA friends.

Short-term Deliverables We were supposed to be working on these now. We are delayed. But, we will make progress this quarter Proposed Deliverables RNS spec finalized OGSA - GFS use case GGF Architecture Workbook version 2 GGF Architecture Standard document – Skeleton Topic not limited to… Data Grid Composition of services System and User-defined Meta-data in GFS GFS Profile Why and what “GGF GFS complaint” product or service Work with our Storage friends in GGF and SNIA

Continuing on GFS Architecture Arun Jagatheesan San Diego Supercomputer Center (SDSC)

World Wide Datagrid

Questions on GFS Architecture

Single, standard GFS interface A consolidated interface of all GFS operations “GFS-compliant” – users and vendors/providers Promotes ease of development, usage and deployment Vendor or provider can compose (on demand from other providers) or implement every operation from scratch Challenges: Do we depend or go into a dead-lock for other standard definitions (e.g) ACL, security, … Implementation

Logical resources We need to identify or name each physical resource (e.g. a file system) to replicate data between resources Identify GFS-compliant storage servers using logical names Used for “data virtualization” – proven useful in data grid management systems Challenge New concept as the base for standardization Community consensus

Multi-billion $ data grid recipe Yes that sounds marketing, but its out of the passion for the technology Logical Shared Collections Logical namespace of unstructured data Physically data could be anywhere in any format Multiple groups in a collaborative environment “All the data of my enterprise and my partners” Logical Shared Resources Logical resource from a “data virtualization perspective” Each storage resource is identified as a logical name in the end-user GFS namespace “Data from San Diego” or “Replicate data in customer’s file system”

What additional operations Bulk operations Domain-specific metadata operations Parallel transfers Replication Challenges A complete interface is needed Should be “implementable” and usable

Data ownership and access controls GFS does not own the data It just owns a type of access to data What levels of data integrity and security to provide? Challenges Can a logical namespace control the physical access to data

Multiple administrative domains Collaborative data management Multiple teams of same or different enterprises “Virtual Organization” or “Virtual Enterprise” Challenges Limited Interest Group: “BIG” academic projects and few Global 2000 AAA - Security

Existing Standards Compatibility Regular NFS or CIFS reflect the GFS Namespace Can NFS/CIFS can use GFS namespace? Challenges Is it pragmatic and useful? Don’t we need to make a paradigm shift? What operations can be supported? Does this WG work on this?

Conclusion GFS successfully delivered its earlier deliverables Some activities delayed this quarter. Hope a better one next time GFS Architecture Lot of questions More room for users, vendors and providers to participate and promote these concepts Have a great GGF-18!