© 2008 Open Grid Forum Data Grid Federation by RNS GFS-WG, OGF23 Balcelona Hideo Matsuda Osaka University / NAREGI.

Slides:



Advertisements
Similar presentations
Gfarm v2 and CSF4 Osamu Tatebe University of Tsukuba Xiaohui Wei Jilin University SC08 PRAGMA Presentation at NCHC booth Nov 19,
Advertisements

© 2006 Open Grid Forum Discussion of File Catalog Standardization GFS-WG, OGF24 Singapore Osamu Tatebe, co-chair of GFS-WG Univ. of Tsukuba Sep 16, 2008.
GFS OGF-22 Global Resource Naming Developers: Reagan Moore Arcot Mike.
OGF-23 iRODS Metadata Grid File System Reagan Moore San Diego Supercomputer Center.
© 2006 Open Grid Forum OGF19 Federated Identity Rule-based data management Wed 11:00 AM Mountain Laurel Thurs 11:00 AM Bellflower.
© 2007 Open Grid Forum SAGA: Simple API for Grid Applications Steven Newhouse Application Standards Area Director.
Genesis II Open Source, OGSA Implementation Genesis II: Mapping Grids into the Local File System: Access, RNS, and ByteIO Andrew Grimshaw Genesis II Team.
© 2007Open Grid Forum OGF22, 25th February 2008 OGSA Data Architecture Mario Antonioletti.
© 2006 Open Grid Forum OGSA Next Steps Discussion Providing Value Beyond the Specifications.
© 2007 Open Grid Forum OGSA-RUS Specification Update, Adoption and WS-RF Profile Discussions (Molly Pitcher) Morris Riedel (Forschungszentrum Jülich –
© 2007 Open Grid Forum Data Management Challenge - The View from OGF OGF22 – February 28, 2008 Cambridge, MA, USA Erwin Laure David E. Martin Data Area.
Web Services Technology Topics The boring stuff. WSRF Web Services Resource Framework –managing stateful resources using web services standards Driven.
Jens G Jensen Atlas Petabyte store Supporting Multiple Interfaces to Mass Storage Providing Tape and Mass Storage to Diverse Scientific Communities.
Plateforme de Calcul pour les Sciences du Vivant SRB & gLite V. Breton.
Data Grids: Globus vs SRB. Maturity SRB  Older code base  Widely accepted across multiple communities  Core components are tightly integrated Globus.
© 2008 Open Grid Forum Grid Standards Realizing Basic Grid Use Cases Using Existing Standards and Profiles.
© JBoss Inc The need for context in Web Services Mark Little, presented by Kurt T Stam Red Hat.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Architecture of Grid File System (GFS) - Based on the outline draft - Arun swaran Jagatheesan San Diego Supercomputer Center Global Grid Forum 11 Honolulu,
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks GINGIN Grid Interoperation on Data Movement.
The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.
NAREGI WP4 (Data Grid Environment) Hideo Matsuda Osaka University.
Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.
The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Dataset Caitlin Minteer & Kelly Clynes.
© 2008 Open Grid Forum Independent Software Vendor (ISV) Remote Computing Primer Steven Newhouse.
ILDG Middleware Status Chip Watson ILDG-6 Workshop May 12, 2005.
CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.
Lattice QCD Data Grid Middleware: status report M. Sato, CCS, University of Tsukuba ILDG6, May, 12, 2005.
Web: Minimal Metadata for Data Services Through DIALOGUE Neil Chue Hong AHM2007.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE middleware: gLite Data Management EGEE Tutorial 23rd APAN Meeting, Manila Jan.
Wide Area Data Replication for Scientific Collaborations Ann Chervenak, Robert Schuler, Carl Kesselman USC Information Sciences Institute Scott Koranda.
© 2006 Open Grid Forum Global resource naming for data grid federation GFS-WG, OGF22 Cambridge Osamu Tatebe, co-chair of GFS-WG Univ. of Tsukuba Feb 27,
H IGH E NERGY A CCELERATOR R ESEARCH O RGANIZATION KEKKEK High Availability iRODS System (HAIRS) Yutaka Kawai, KEK Adil Hasan, ULiv December 2nd, 20091Interoperability.
Lattice QCD Data Grid Middleware: Meta Data Catalog (MDC) -- CCS ( tsukuba) proposal -- M. Sato, for ILDG Middleware WG ILDG Workshop, May 2004.
Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,
© 2008 Open Grid Forum File Catalog Development in Japan e-Science Project GFS-WG, OGF24 Singapore Hideo Matsuda Osaka University.
GridNEWS: A distributed Grid platform for efficient storage, annotating, indexing and searching of large audiovisual news content Ioannis Konstantinou.
Managing Data DIRAC Project. Outline  Data management components  Storage Elements  File Catalogs  DIRAC conventions for user data  Data operation.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America gLite Information System Claudio Cherubino.
Interactive Data Analysis on the “Grid” Tech-X/SLAC/PPDG:CS-11 Balamurali Ananthan David Alexander
Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.
USGS GRID Exploratory Status Review Stuart Doescher Mike Neiers USGS/EDC May
Steve Graham WS-ResourceFramework Modeling Stateful Resources With Web services OASIS WSRF TC F2F Wednesday, April 28th, 2004.
EGI-Engage Data Services and Solutions Part 1: Data in the Grid Vincenzo Spinoso EGI.eu/INFN Data Services.
Distributed Data Access Control Mechanisms and the SRM Peter Kunszt Manager Swiss Grid Initiative Swiss National Supercomputing Centre CSCS GGF Grid Data.
The Institute of High Energy of Physics, Chinese Academy of Sciences Sharing LCG files across different platforms Cheng Yaodong, Wang Lu, Liu Aigui, Chen.
FESR Trinacria Grid Virtual Laboratory gLite Information System Muoio Annamaria INFN - Catania gLite 3.0 Tutorial Trigrid Catania,
RENKEI:UGI Takashi Sasaki. Project history The RENKEI project led by Prof. Ken Miura of NII is funded by MEXT during JFY The goal of the project.
INFSO-RI Enabling Grids for E-sciencE University of Coimbra gLite 1.4 Data Management System Salvatore Scifo, Riccardo Bruno Test.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Architecture of LHC File Catalog Valeria Ardizzone INFN Catania – EGEE-II NA3/NA4.
Enabling Grids for E-sciencE EGEE-II INFSO-RI Status of SRB/SRM interface development Fu-Ming Tsai Academia Sinica Grid Computing.
2 nd EGEE/OSG Workshop Data Management in Production Grids 2 nd of series of EGEE/OSG workshops – 1 st on security at HPDC 2006 (Paris) Goal: open discussion.
OGF24 15 September 2008 Data Area Overview Erwin Laure David E. Martin Data Area Directors.
EGEE Data Management Services
Distributed OS.
Grid File System Working Group
The Data Grid: Towards an architecture for Distributed Management
Vincenzo Spinoso EGI.eu/INFN
Data services on the NGS
Cross-health enterprises Medical Data Management on the EGEE grid
Evaluation of “data” grid tools
Introduction to Data Management in EGI
Information System Virginia Martín-Rubio Pascual
GFS-WG: Informal Status Report
NAREGI at KEK and GRID plans
Data services in gLite “s” gLite and LCG.
Mats Rynge USC Information Sciences Institute
Grid related activities at KEK
RNS Interoperability and File Catalog Standardization
Information Services Claudio Cherubino INFN Catania Bologna
Presentation transcript:

© 2008 Open Grid Forum Data Grid Federation by RNS GFS-WG, OGF23 Balcelona Hideo Matsuda Osaka University / NAREGI

2 Use of Metadata in e-Science For scientists, not only data itself but also its metadata is very important. Example of metadata: data ID, locations of data, experiment materials, instruments, and methods Relationships between data are described as metadata.

3 Example of Metadata Metadata is often described with hierarchical representation in many sciences. CMSATLAS run1run2 track1track2 Protein Nucleotide Primate Plant gb|AY BacteriaSequenceStructure sp|P37231pdb|1FM6 Vertebrate High Energy PhysicsMolecular Biology

4 Metadata Management using File Catalog Currently metadata are mainly stored in File Catalogs using their hierarchical namespace functionality. gLite: LFC, Fireman iRODS (SRB): MCAT Globus: RLS NAREGI: Gfarm File Catalog information of different Grid middlewares do not have compatibility to each other. It is not easy to exchange metadata over different Grid middlewares.

5 Resource Namespace Service (1) RNS lets you map any resource into single, hierarchical namespace Resources are referred to in a form of EndpointReference (WS-Addressing) RNS Specification is published as GFD- R-P.101 RNS implementation is available from U.Virginia and U.Tsukuba.

6 Resource Namespace Service (2) Hierarchical namespace management that provides name- to-resource mapping Basic Namespace Component Virtual Directory Non-leaf node in hierarchical namespace tree Junction Name-to-resource mapping that interconnects a reference to any existing resource into hierarchical namespace /grid ogfjp datagfs file1file3 file2 file4 file1file2 EPR1 EPR2 EPR: Endpoint Reference

7 Comparison RNS with File Catalog GFS Naming Profile on top of RNS and File Catalog Service are basically the same File Catalog implies loosely coupled federation, whereas File System Directory implies rather tightly coupled federation File Catalog Standardization is required by many parties

8 Data Grid Federation with RNS (Plan) RNS can interconnect a reference to any existing resource into hierarchical namespace Most of Grid middlewares have GridFTP for data transfer Use RNS as a (standardized?) File Catalog Use GridFTP URL gsiftp://.../ as the address of Endpoint Reference. gLite File Server (SRM) RNS iRODS File Server NAREGI File Server (Gfarm) Globus GridFTP Server Client (1) query (2) EPR list (including address) (3) Access with GridFTP protocol RNS

9 Summary RNS has a potential functionality to federate Data Grids over different middlewares. The federation encourages scientists to proceed international collaboration. RNS implementation is available. It need to be re-designed towards production level.