1. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 1 Storage Resource Management at Fermilab Timur Perelmutov Don Petravick Fermi National Accelerator Lab

2. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 2 Fermilab Storage ● We have covered: – Enstore – permanent tape storage ● Scales to several petabytes ● Flexible media selection ● Encp access protocol – Dcache – distributed disk storage ● Scales to hundreds of terabytes ● Commodity hardware ● Pluggable transfer protocols – PNFS – namespace and file metadata server for Enstore and Dcache ● Next: SRM – Storage Resource Manager

3. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 3 SRM Collaboration Jefferson Lab Bryan Hess Andy Kowalski Chip Watson Fermilab Don Petravick Timur Perelmutov LBNL Arie Shoshani Alex Sim Junmin Gu EU DataGrid WP2 Peter Kunszt Heinz Stockinger Kurt Stockinger Erwin Laure EU DataGrid WP5 Jean-Philippe Baud Stefano Occhetti Jens Jensen Emil Knezo Owen Synge

4. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 4 SRM Motivation (1) Grid Architecture promises Reservation and Scheduling of the Following Shared Resources – Computing Resources – Network Resources – Storage Resources (often neglected) SRM provides Reservation and Scheduling of the Storage Resources

5. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 5 SRM Motivation (2) High Energy Physics Collaborations span multiple institutions where – A large variety of types of Storage Systems exist ● Single Disk / Raid ● Robotic Tape Storage System (Enstore, HPSS) ● Distributed Disk Cache (dCache) ● Hierarchical Storage System (dCache - Enstore) – Heterogeneous environments and proliferation of custom Mass Storage Systems (MSSs) exist – User applications often need to access data at multiple institutions on multiple MSSs SRM provides Standardized Uniform Access to Heterogeneous Storage

6. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 6 Access to Multiple MSS HPSS Client Enstore Client Jasmine client Enstore JASMine Client USER/APPLICATIONS Dcache client DCache Castor Client CASTOR

7. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 7 Uniform Access via SRM SRM Dcache/ Enstore JASMine Client USER/APPLICATIONS Grid Middleware SRMDCacheSRMCASTOR

8. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 8 Storage Classification Storage systems can be classified by: – Persistence of data ● Permanent ● Temporary – Data access availability ● Data immediately available ● Data needs advanced reservation before utilization (tapes need to be mounted, files need to migrate to disks, etc.) – Supported transfer protocols ● File transfer protocols ● POSIX like access protocols Need management interface that supports all of the above

9. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 9 Storage Resource Managers SRMs are middleware components that manage shared storage resources on the Grid and provide: ● Uniform access to heterogeneous storage ● File Transfer Protocol negotiation ● Dynamic Transfer URL allocation ● Access to permanent and temporary types of storage ● Advanced space and file reservation ● Reliable transfer services

10. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 10 Data life cycle on the grid overview ● Collection or generation –Data is collected from the CMS detector –Data generated by the analysis ● Storage –Data is stored on a shared storage system ● Registration –Data is registered in a Replica Catalog by via a Replica Registration Service (RRS) ● Replication –Data is replicated to a storage geographically close to the consumer ● Replica registration –RRS ● Delivery and consumption Globally Unique Identifier Logical File Name 1 Logical File Name 2 Logical File Name N Physical File Name 1 Physical File Name 2 Physical File Name M Replica Catalogs Mappings

11. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 11 Illustration US-CMS DATAGRID Compact Muon solenoid (CMS) is an experiment at the lepton Hadron Collider (LHC) at CERN in Geneva, Switzerland. ● US-CMS is building a data grid to facilitate physics data analysis at academic institutions across the United States. ● Data grid 3 tier architecture. – Tier 0 CERN, Geneva, Switzerland. – Tier 1 consists of 5 regional centers, FERMILAB in Batavia, IL is a North American Center. – Tier 2 consists of 25 centres, 5 of these are in North America. ● SRM copy is used as a management protocol and reliable replication service for movement of data from tier 0 to tier 1 centers and from tier 1 to tier 2.

12. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 12 Data life cycle on the grid and a role of SRM on the GRID SRM-Client SRM cache SRM dCache 6.GridFTP ERET (pull mode) Enstore CASTOR Replica Catalog Network transfer of DATA 1.DATA Creation 2. SRM- PUT Network transfer 3. Register (via RRS) CERN Tier 0 Replica Manager FNAL Tier 1 archive files stage files 4.SRM- COPY Tier0 to Tier1 5.SRM-GET archive files SRM Tier2 Storage Tier 2 Center Network transfer 9.GridFTP ESTO (push mode) 8.SRM-PUT 7.SRM- COPY Tier1 to Tier2 SRM-Client Retrieve data for analysis 10.SRM-GET Users SRM-Client Network transfer of DATA

13. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 13 ● Two SRM Interface specifications –SRM v1.1provides Data access/transfer Implicit space reservation –SRM v2.1 adds Explicit space reservation Namespace discovery and manipulation Access permissions manipulation ● Fermilab SRM implements SRM v1.1 specification ● SRM v2.1 by the end of 2004 Storage Resource Manager versions

14. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 14 SRM Protocols and Groups of Functions SRM interface consists of the following groups of functions: ● Space Management Functions – v2.1 ● Data Transfer Functions – v1.1 and v2.1 ● Directory Functions – v2.1 ● Permission Functions – v2.1 ● Status Functions – v1.1 and v2.1 SRM v2.1 in the rest of this talk

15. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 15 SRM File and Space Types Volatile Durable Permanent Space Types File Types Blah Volatile File Type Permanent Durable Blah No Archived Yes No Blah Yes LifetimeNo Yes Blah Yes AutoDeleted No Property

16. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 16 SRM Interface Details Space Management Functions ● SrmReserveSpace ● SrmReleaseSpace ● srmUpdateSpace ● srmCompactSpace ● srmGetSpaceMetaData ● srmChangeFileStorageT ype ● srmGetSpaceToken Directory ● SrmMkdir ● srmRmdir ● srmRm ● srmLs ● srmMv Data transfer functions ● srmPrepareToGet ● SrmPrepareToPut ● srmCopy ● SrmRemoveFiles ● srmReleaseFiles ● srmPutDone ● srmAbortRequest ● srmAbortFiles ● srmSuspendRequest ● srmResumeRequest Status Functions ● srmStatusOfGetRequest ● srmStatusOfPutRequest ● srmStatusOfCopyRequest ● srmGetRequestSummary ● srmExtendFileLifeTime ● SrmGetRequestID Permission ● srmSetPermission ● srmReassignToUser ● srmCheckPermission

17. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 17 SRM Interface Details Space Management Functions ● SrmReserveSpace ● SrmReleaseSpace ● srmUpdateSpace ● srmCompactSpace ● srmGetSpaceMetaData ● srmChangeFileStorageT ype ● srmGetSpaceToken Directory ● SrmMkdir ● srmRmdir ● srmRm ● srmLs ● srmMv Data transfer functions ● srmPrepareToGet ● SrmPrepareToPut ● srmCopy ● SrmRemoveFiles ● srmReleaseFiles ● srmPutDone ● srmAbortRequest ● srmAbortFiles ● srmSuspendRequest ● srmResumeRequest Status Functions ● srmStatusOfGetRequest ● srmStatusOfPutRequest ● srmStatusOfCopyRequest ● srmGetRequestSummary ● srmExtendFileLifeTime ● SrmGetRequestID Permission ● srmSetPermission ● srmReassignToUser ● srmCheckPermission

18. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 18 Srm Example 1 - srmPrepareToPut User SRM Node dCache HeadNode dCache PoolNode findPoolNode reserveSpace spaceToken srmPrepareToPut (SpaceToken,size,gsiftp) srmReserveSpace findDoor(gsiftp) Transferurl= gsiftp://doorNode:port/path doorNode,port dCache DoorNode Gridftp ESTOR Where is user space poolNode DATA PutDone

19. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 19 Srm Example - Robust Replication Anywhere cache dCache HRM-COPY (thousands of files) SRM-GET (one file at a time) SRM-Client SRM (performs writes) SRM (performs reads) FNAL GridFTP GET (pull mode) archive files Network transfer Enstore CASTOR stage files CERN

20. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 20 SRM Interface Details Space Management Functions ● SrmReserveSpace ● SrmReleaseSpace ● srmUpdateSpace ● srmCompactSpace ● srmGetSpaceMetaData ● srmChangeFileStorageT ype ● srmGetSpaceToken Directory ● SrmMkdir ● srmRmdir ● srmRm ● srmLs ● srmMv Data transfer functions ● srmPrepareToGet ● SrmPrepareToPut ● srmCopy ● SrmRemoveFiles ● srmReleaseFiles ● srmPutDone ● srmAbortRequest ● srmAbortFiles ● srmSuspendRequest ● srmResumeRequest Status Functions ● srmStatusOfGetRequest ● srmStatusOfPutRequest ● srmStatusOfCopyRequest ● srmGetRequestSummary ● srmExtendFileLifeTime ● SrmGetRequestID Permission ● srmSetPermission ● srmReassignToUser ● srmCheckPermission

21. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 21

22. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 22 Grid File Access Library and SRM Physics Application Replica Manager Client SRM Client Local File I/O rfio I/O open() read() etc. dCap I/O open() read() etc. Grid File Access Library (GFAL) SRM Service dCap Service rfio Service RM Services MSS Service Local Disk POSIX I/O GridFTP Service Wide Area Access Michael Ernst’s slide

23. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 23 Client Access to CMS-DATA Michael Ernst’s slide Client Logical Query Request Interpreter Request Executer Logical Files site-specific file requests site-specific files SRM request planning pinning & file transfer requests property index Replica Catalog MDS Site hosting the Application Network Tier1 Tier2 ? (dCache, dFarm, DRM, NeST)

24. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 24 Status of Fermilab SRM Implementation ● SRM Interface to dCache Storage System – Data Transfer Functions (get, put and copy) – Load balancing, throttling, fairness – Scalable replication mechanism via gridftp – Automatic directory creation – Fault tolerance and reliability achieved by providing persistent storage for transfer requests and retries on failures ● SRM interface as a standalone product, adaptable to work on top of another storage system through a SRM-Storage interface ● A reference implementation of the SRM-Storage interface to a Unix File System

25. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 25 Fermilab SRM Implementation Plans (1) ● Space Management – First implicit as a part of SRM Version 1.1 – Than explicit, through the SRM V2.1 ● Full implementation of SRM Version 2.1 interface ● International Lattice QCD (ILDG) ● Research utilization of Lambda Station Interface by a Storage System. Lambda Station gives optical path allocation and per flow routing ● Monitoring, Administration and Accounting interfaces

26. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 26 Fermilab SRM Implementation Plans (2) ● Integration of SRM to SAMGrid Data Handling System as the universal interface to a storage

27. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 27 Fermilab SRM Implementation Plans (3) ● Integration with Virtual Organization based Authorization Services, development of Storage Specific fine grain Authorization Services

28. 4 Oct 04Storage Resource Manager, Timur Perelmutov, Don Petravick, Fermilab 28 Resources ● The Storage Resource Manager Collaboration, http://sdm.lbl.gov/srm-wg/http://sdm.lbl.gov/srm-wg/ ● Fermilab SRM Project, http://www-isd.fnal.gov/srmhttp://www-isd.fnal.gov/srm ● Patrick Fuhrmann, dCache, Grid Storage Element and enhanced use cases,http://indico.cern.ch/contributionDisplay.py?contribId=233&sessionId=10&confId=0http://indico.cern.ch/contributionDisplay.py?contribId=233&sessionId=10&confId=0 ● DCache, Disk Cache Mass Storage System, http://www.dcache.org/http://www.dcache.org/ ● US-CMS, http://www.uscms.org/http://www.uscms.org/ ● Don Petravick, Lambda Station Proposal, http://hppc.fnal.gov/wawg/omnibus-text.pdfhttp://hppc.fnal.gov/wawg/omnibus-text.pdf ● Arie Shoshany, Replica Registration Service,http://www.ppdg.net/mtgs/28jun04- wb/slides/PPDG-AH-0406-RMS-RRS.ppthttp://www.ppdg.net/mtgs/28jun04- wb/slides/PPDG-AH-0406-RMS-RRS.ppt ● Robert Kennedy, SAMGrid Integration of SRMs, http://indico.cern.ch/contributionDisplay.py?contribId=460&sessionId=7&confId=0 http://indico.cern.ch/contributionDisplay.py?contribId=460&sessionId=7&confId=0 ● Michael Ernst, US-CMS Grid File Access Proposal,http://www.uscms.org/sandc/reviews/doe-nsf/2003-07/docs/GFA-Proposal-Short- v1.0.pdfhttp://www.uscms.org/sandc/reviews/doe-nsf/2003-07/docs/GFA-Proposal-Short- v1.0.pdf ● Michael Ernst, Managed Data Storage and Data Access Services for Data Grids,http://indico.cern.ch/contributionDisplay.py?contribId=190&sessionId=7&confId=0http://indico.cern.ch/contributionDisplay.py?contribId=190&sessionId=7&confId=0 ● Philip DeMar, LambdaStation: A forwarding and admission control service to interface production network facilities with advanced research network paths http://indico.cern.ch/contributionDisplay.py?contribId=359&sessionId=11&confId=0 http://indico.cern.ch/contributionDisplay.py?contribId=359&sessionId=11&confId=0