1. User Domain Storage Elements SURL  TURL LFC Domain (LCG File Catalogue) SA1 – Data Grid Interoperation Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 http://www.ngs.ac.uk/ http://www.isis.rl.ac.uk/ http://www.gridpp.ac.uk/ Grid Data Interoperation (Part II): Data, Metadata, Catalogues SRM DPM, dCache, StoRM, CASTOR,… Data Interoperation Data Mode 1: Pretend SRB is a “Classic SE” Classic SE (still) supported by gLite FTS FTS SRB Disk storage SRM GridFTP Disk storage GridFTP Disk storage GridFTP SRM selects pool node… SRB Info GridFTP Three interoperation modes for data transfers: 1.FTS 2.SRM drives transfer via srmCopy() (not shown) 3.lcg-utils lcg-* SRB Disk storage SRM GridFTP Disk storage GridFTP Disk storage GridFTP BDII Add a static information provider using BDII Data Mode 3: using lcg-utils SURL TURL GUID LFN SRB SURLs are GSIFTP URLs; SRB TURLs are the same as the SURLs. lcg-* tools do not accept GSIFTP SURLs ISIS Neutron source at RAL Proposal Instrument/experiment Dataset File Parameters SRB metadata (key/value pairs) Metadata held in separate catalogue, the iCAT (not to be confused with the iRODS catalogue). Format is XML. iCAT uses Oracle. Data is held in SRB. SRB’s metadata facility is hardly used. Metadata migration iCAT schema: Only basic attributes so far (datasets, instrument, owner) Authors: Jens Jensen, STFC (corresponding) Sam Skipsey, University of Glasgow Chris Moreton-Smith, ISIS, STFC Special thanks to Michael Gleaves and Brian Matthews, STFC, for iCAT discussions, and to Birger Koblitz, CERN, for AMGA support/suggestions Experiences Needs glue to make it work together. Can improve on original use, maybe Had to custom build metadata schema on gLite side Custom build metadata copier /grid/isis/guid/c4756f6e-7963-47ad-ac8a-59726afa4992 vs /grid/isis/NDXINTER/Instrument/data/cycle_08_5/INTER00000544.raw TODO: Other SRB users: eMinerals, eMaterials, RMCS Work on integration with job submission, maybe portal Track work on datasets: provenance Improve metadata support Dataset attributes: Date, owner, run title, status, keywords, location) Currently managing datasets with separate dataset sequence attribute: Once one file is found, the rest of the dataset is located Mirrors original use where metadata is kept with a single file Can also use to move data to/from disks with GridFTP Catalogue Strategy: always clone file to SRM, then register clone in LFC. (Fallback: register GridFTP SURL in other catalogue, or hack LFC, or use AMGA to keep track of replicas.) Metadata Two approaches to file metadata management (primary key): 1.Use the GUID as filename – shallow hierarchy 2.Use the original filename (or algorithmically derived name) The former makes sense on the grid: register meaningful LFNs to point to GUID in LFC. The latter does not depend on LFC/replications. Metadata is associated to primary key. Avoid metadata in filenames! Data mover FTS still supports “Classic SE” ASGC SRM interface to SRB will become preferred Doesn’t move metadata though. iCAT in Google code: http://code.google.com/p/icatproject/ References: S Burke et al: gLite User Guide, CERN EDMS 722398 J Jensen, R Downing, M Hodges, D Ross: SRM and SRB interoperation F Bonifazi et al: LHCb experience with LFC replication, Proc CHEP 2007 M Gleaves: ICAT software suite iCAT metadata is hierarchical: associated with individual file or dataset. Current support is simplistic: dump metadata with any file in dataset (works in current limited scenarios) Data mover todo: Improve Modularise metadata porting Generalise?