gLite Data and Metadata Management Hailong Yang (hailong.yang@jsi.buaa.edu.cn) Sino-German Joint Software Institute(JSI) Joint CHAIN/EPIKH School for Application Porting Beijing/China, 19.5.2011

Outline Challenges of data management in a Grid infrastructure Initial definitions Types of Storage Elements File naming conventions File catalogue Be prepared for a bunch of acronyms! Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 2

Challenges Heterogeneity Data are stored on different storage systems using different access technologies Distribution Data are stored in different locations (in most cases there is no shared file system or common namespace) Data need to be moved between different locations Data description Data are stored as files (need to describe and locate them according to their content) Storage Resource Manager interface File Catalogue File Transfer Service Metadata Service Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 3 3

Getting started The Storage Element (SE) is the service which allows users and applications (programs) to store/retrieve data (files) The DMS provide services for location, access and transfer of files User do not need to know the file location, just its logical name. Files can be replicated or transferred to several locations (SEs) as needed. Files are shared within a VO Files are write-once, read-many Files cannot be changed unless remove or replaced Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 4 4

Getting started Files located in the Storage Elements (SEs)… Are mostly write-once, read-many. Accessible by users and applications from “anywhere” in the Grid. Several replicas of one file can be replicated at different sites. Cannot be changed unless remove or replaced. Storage Elements (SEs)… Provide storage space for files. Provide transfer protocol (GSIFTP) ~ GSI based FTP server Provide an interface for the management of disk and tape storage resources: Storage Resource Manager (SRM) Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 5 5

Types of Storage Elements dCache Consists of a server and one or more pool nodes. Centralized admin.: single point of access to the SE. Files are presented in the disk pools under a single virtual filesystem tree. Uses the GSI dCache Access Protocol (gsidcap). Storm Solution best suited to cope with large storage (> or >> 100 TB) Makes full advantage of parallel filesystem (GPFS, Lustre) SRM v2.2 interface CERN Advanced STORage manager (CASTOR) Files are migrated from a disk buffer frontend to a tape mass storage Uses the insecure Remote File I/O protocol (RFIO) Disk Pool Manager (DPM) Used for fairly small SEs (max 10 TB of total space) with disk-based storage only. Uses secure RFIO protocol Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 6 6

Storage Resource Manager (SRM) The SRM is a single interface that takes care of local storage interaction and provides a Grid interface to the outside world. SE CASTOR I talk to them on your behalf I will even allocate space for your files And I will use transfer protocols to send your files there You as a user need to know all the systems!!! SE Storm SRM SE DPM SE dCache Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 7 7

A practical example (1) Storm at SiteA dCache at SiteB DPM at SiteD She is working on a job which needs: - read MonteCarlo simulations from siteA - read experiment data from siteB - read environmental data from siteC - write output to home siteD Storm at SiteA dCache at SiteB DPM at SiteD DPM at SiteC Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011s

File Naming conventions (1) Grid Unique IDentifier (GUID) Every file has a GUID A non-human-readable unique identifier, e.g.: guid:38ed3f60-c402-11d7-a6b0-f53ee5a37e1d Note: all replicas of a file will share the same GUID Logical File Name (LFN) An alias that can be used to refer to a file, e.g.: lfn://grid/gilda/users/mario/myfile.dat Logical File Name 1 Logical File Name N GUID ... Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 9 9

File Naming conventions (2) Storage URL (SURL) or Physical File Name (PFN) The location of an actual file on a storage system, e.g.: srm://aliserv6.ct.infn.it/dpm/home/gilda/project1/test.dat Note: Used by the system to find where the replica is physically stored Transport URL (TURL) Complete URI with the necessary information to access a file in a SE (including the access protocol) e.g.: rfio://lxshare0209.cern.ch//data/alice/ntuples.dat Logical File Name 1 Physical File SURL 1 TURL 1 ... GUID ... ... Logical File Name N Physical File SURL N TURL 1 Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 10

SRM interactions SE SRM 1 Client 4 2 3 5 The client asks the SRM for the file providing an SURL The SRM asks the Storage Element to provide the file The Storage Element notifies the availability of the file and its location The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed The client interacts with the storage using the protocol specified in the TURL Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 11

Needles in a haystack File Catalogue LFC = LCG File Catalogue How do I keep track of all files I have on the Grid? Even if I remember all the LFN’s of my files, what about someone else's files? How does the Grid keep track of the mapping between LFN(s), GUID and SURL(s)? LFC = LCG File Catalogue LCG = LHC Compute Grid LHC = Large Hadron Collider File Catalogue Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 12

File Catalogue Is the service which maintains mappings between LFN(s), GUID and SURL(s) It keeps track of the location of copies (replicas) of files It consists of a unique catalogue, where the LFN is the main key Looks like a “top-level” directory in the Grid For each of the supported VO a separate subdirectory exists under the "/grid" directory. All members of a given VO have read-write permissions in such a directory Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 13

The LFC Service lfn:/grid/gilda/tcaland/mpi.txt File Catalogue SE A SE B User Interface SE C Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 14

The LFC Service User System Metadata Metadata LFN GUID LCF key /grid/dteam/dir1/dir2/file1.root 38ed3f60-c402-11d7 -a6b0… Symlink Replicas srm://host.example.com/foo/bar host.example.com /grid/dteam/mydir/mylink SURLs Further LFNs can be added as symlinks to the main LFN. Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 15

Job submission – example 1 Small files: InputSandbox / OutputSandbox CE WMS User Interface Worker Nodes Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 16

Data Management – example 2 CE WMS User Interface Worker Nodes LFC SE SE Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011 17

LFC Commands Summary of the LFC Catalog commands lfc-chmod Change access mode of the LFC file/directory lfc-chown Change owner and group of the LFC file-directory lfc-delcomment Delete the comment associated with the file/directory lfc-getacl Get file/directory access control lists lfc-ln Make a symbolic link to a file/directory lfc-ls List file/directory entries in a directory lfc-mkdir Create a directory lfc-rename Rename a file/directory lfc-rm Remove a file/directory lfc-setacl Set file/directory access control lists lfc-setcomment Add/replace a comment Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011

lcg utils commands Replica Management File Catalog Interaction lcg-aa lcg-cp Copies a grid file to a local destination lcg-cr Copies a file to a SE and registers the file in the catalog lcg-del Delete one file lcg-rep Replication between SEs and registration of the replica lcg-gt Gets the TURL for a given SURL and transfer protocol lcg-sd Sets file status to “Done” for a given SURL in a SRM request File Catalog Interaction lcg-aa Add an alias in LFC for a given GUID lcg-ra Remove an alias in LFC for a given GUID lcg-rf Registers in LFC a file placed in a SE lcg-uf Unregisters in LFC a file placed in a SE lcg-la Lists the alias for a given SURL, GUID or LFN lcg-lg Get the GUID for a given LFN or SURL lcg-lr Lists the replicas for a given GUID, SURL or LFN Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011

References gLite documentation homepage DM subsystem documentation http://glite.web.cern.ch/glite/documentation/default.asp DM subsystem documentation http://egee-jra1-dm.web.cern.ch/egee-jra1-dm/doc.htm LFC and DPM documentation https://twiki.cern.ch/twiki/bin/view/LCG/DataManagementDocumentation gLite Data Management Tutorial: https://grid.ct.infn.it/twiki/bin/view/GILDA/DataManagement#Create_directory Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19.5.2011

Reference: https://grid.ct.infn.it/twiki/bin/view/GILDA/DataManagement Let’s practice! Reference: https://grid.ct.infn.it/twiki/bin/view/GILDA/DataManagement 21 21

Environment Variables Pointing to the right BDII Pointing to the right LFC echo $LCG_GFAL_INFOSYS export LCG_GFAL_INFOSYS = gilda-bdii.ct.infn.it:2170 echo $LFC_HOST export LFC_HOST = lfc-gilda.ct.infn.it 22 22

Before starting… Make sure to have a proxy created voms-proxy-info -all voms-proxy-init --voms gilda 23 23

LFC: creating a directory List directories Create your own personal directory inside lfc-ls /grid/gilda/tutorials/ lfc-mkdir /grid/gilda/tutorials/epikh You can check the creation typing: lfc-ls /grid/gilda/tutorials/ 24 24

Copying and registering a file 1/2 lcg-cr Copies a file to a SE and registers the file in the catalogue lcg-cr --vo <vo name> -l <LFN destination> -d <SE> <local file> Make sure to have a directory in the LFC (/grid/gilda/tutorials/yourname/) Use the lcg-info or lcg-infosites commands to figure out the available SEs This command will return the GUID for your file 25 25

Copying and registering a file 2/2 lcg-infosites --vo gilda se Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 16241327 3464020 n.a se.hpc.iit.bme.hu 2390269393 97444791 n.a se1-egee.srce.hr 24377757 13973700 n.a gilda-02.pd.infn.it 80605789 4411630 n.a sirius-se.ct.infn.it 4089114112 704958582 n.a grisuse.scope.unina.it lcg-cr --vo gilda -l lfn:/grid/gilda/tutorials/epikh/test.txt -d gilda-02.pd.infn.it file://$HOME/test.txt guid:0d8ef3b9-7f73-4c57-80c9-e827bace8597 26 26

Downloading a file First of all, let’s download a file from a SE to start “playing” with it. Basic Usage: Try it: lcg-cp --vo <vo name> <LFN origin> <local destination> lcg-cp --vo gilda lfn:/grid/gilda/tutorials/epikh/test.txt file://$HOME/test1.txt 27 27

Replicate a file between SEs Basic Usage: Try it: lcg-rep --vo <vo name> -d <destination SE> <LFN of your file> lcg-rep --vo gilda -d sirius-se.ct.infn.it lfn:/grid/gilda/tutorials/epikh/test.txt 28 28

From where it was downloaded? List the Replicas of the file: This command will return the SURL of all replicas A file can be stored on multiple SE's so that a job can download it from the closest SE while is running. lcg-lr --vo gilda lfn:/grid/gilda/tutorials/epikh/test.txt 29 29

Deleting a file Basic Usage: When used with '-a' switch will delete all replicas and delete entry from catalog Try it: lcg-del -a --vo <vo name> <LFN> lcg-del -a --vo gilda lfn:/grid/gilda/tutorials/epikh/test.txt 30 30

Removing a LFC directory Basic Usage: Try it: lfc-rm -r <LFC file path> lfc-rm -r /grid/gilda/tutorials/epikh 31 31

Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19