1. gLite Data and Metadata Management
Hailong Yang
Sino-German Joint Software Institute(JSI)
Joint CHAIN/EPIKH School for Application Porting
Beijing/China,

2. Outline Challenges of data management in a Grid infrastructure
Initial definitions
Types of Storage Elements
File naming conventions
File catalogue
Be prepared for a bunch of acronyms!
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 2

3. Challenges Heterogeneity
Data are stored on different storage systems using different access technologies
Distribution
Data are stored in different locations (in most cases there is no shared file system or common namespace)
Data need to be moved between different locations
Data description
Data are stored as files (need to describe and locate them according to their content)
Storage Resource
Manager interface
File Catalogue
File Transfer Service
Metadata Service
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 3 3

4. Getting started
The Storage Element (SE) is the service which allows users and applications (programs) to store/retrieve data (files)
The DMS provide services for location, access and transfer of files
User do not need to know the file location, just its logical name.
Files can be replicated or transferred to several locations (SEs) as needed.
Files are shared within a VO
Files are write-once, read-many
Files cannot be changed unless remove or replaced
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 4 4

5. Getting started Files located in the Storage Elements (SEs)…
Are mostly write-once, read-many.
Accessible by users and applications from “anywhere” in the Grid.
Several replicas of one file can be replicated at different sites.
Cannot be changed unless remove or replaced.
Storage Elements (SEs)…
Provide storage space for files.
Provide transfer protocol (GSIFTP) ~ GSI based FTP server
Provide an interface for the management of disk and tape storage resources: Storage Resource Manager (SRM)
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 5 5

6. Types of Storage Elements
dCache
Consists of a server and one or more pool nodes.
Centralized admin.: single point of access to the SE.
Files are presented in the disk pools under a single virtual filesystem tree.
Uses the GSI dCache Access Protocol (gsidcap).
Storm
Solution best suited to cope with large storage (> or >> 100 TB)
Makes full advantage of parallel filesystem (GPFS, Lustre)
SRM v2.2 interface
CERN Advanced STORage manager (CASTOR)
Files are migrated from a disk buffer frontend to a tape mass storage
Uses the insecure Remote File I/O protocol (RFIO)
Disk Pool Manager (DPM)
Used for fairly small SEs (max 10 TB of total space) with disk-based storage only.
Uses secure RFIO protocol
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 6 6

7. Storage Resource Manager (SRM)
The SRM is a single interface that takes care of local storage interaction and provides a Grid interface to the outside world.
SE CASTOR
I talk to them on your behalf
I will even allocate space for your files
And I will use transfer protocols to send your files there
You as a user need to know all the systems!!!
SE Storm
SRM
SE DPM
SE dCache
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 7 7

8. A practical example (1) Storm at SiteA dCache at SiteB DPM at SiteD
She is working on a job which needs:
- read MonteCarlo simulations from siteA
- read experiment data from siteB
- read environmental data from siteC
- write output to home siteD
Storm at SiteA
dCache at SiteB
DPM at SiteD
DPM at SiteC
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, s

9. File Naming conventions (1)
Grid Unique IDentifier (GUID)
Every file has a GUID
A non-human-readable unique identifier, e.g.: guid:38ed3f60-c402-11d7-a6b0-f53ee5a37e1d
Note: all replicas of a file will share the same GUID
Logical File Name (LFN)
An alias that can be used to refer to a file, e.g.: lfn://grid/gilda/users/mario/myfile.dat
Logical File Name 1
Logical File Name N
GUID
...
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 9 9

10. File Naming conventions (2)
Storage URL (SURL) or Physical File Name (PFN)
The location of an actual file on a storage system, e.g.: srm://aliserv6.ct.infn.it/dpm/home/gilda/project1/test.dat
Note: Used by the system to find where the replica is physically stored
Transport URL (TURL)
Complete URI with the necessary information to access a file in a SE (including the access protocol) e.g.: rfio://lxshare0209.cern.ch//data/alice/ntuples.dat
Logical File Name 1
Physical File SURL 1
TURL 1
...
GUID
...
...
Logical File Name N
Physical File SURL N
TURL 1
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 10

11. SRM interactions SE SRM 1 Client 4 2 3 5
The client asks the SRM for the file providing an SURL
The SRM asks the Storage Element to provide the file
The Storage Element notifies the availability of the file and its location
The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed
The client interacts with the storage using the protocol specified in the TURL
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 11

12. Needles in a haystack File Catalogue LFC = LCG File Catalogue
How do I keep track of all files I have on the Grid?
Even if I remember all the LFN’s of my files, what about someone else's files?
How does the Grid keep track of the mapping between LFN(s), GUID and SURL(s)?
LFC = LCG File Catalogue
LCG = LHC Compute Grid
LHC = Large Hadron Collider
File Catalogue
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 12

13. File Catalogue
Is the service which maintains mappings between LFN(s), GUID and SURL(s)
It keeps track of the location of copies (replicas) of files
It consists of a unique catalogue, where the LFN is the main key
Looks like a “top-level” directory in the Grid
For each of the supported VO a separate subdirectory exists under the "/grid" directory.
All members of a given VO have read-write permissions in such a directory
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 13

14. The LFC Service lfn:/grid/gilda/tcaland/mpi.txt File Catalogue SE A
SE B
User Interface
SE C
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 14

15. The LFC Service User System Metadata Metadata LFN GUID LCF key
/grid/dteam/dir1/dir2/file1.root
38ed3f60-c402-11d7 -a6b0…
Symlink
Replicas
srm://host.example.com/foo/bar
host.example.com
/grid/dteam/mydir/mylink
SURLs
Further LFNs can be added
as symlinks to the main LFN.
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 15

16. Job submission – example 1
Small files: InputSandbox / OutputSandbox CE
WMS
User Interface
Worker Nodes
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 16

17. Data Management – example 2CE
WMS
User Interface
Worker Nodes
LFC SE SE
Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 17

18. LFC Commands Summary of the LFC Catalog commands lfc-chmod
Change access mode of the LFC file/directory
lfc-chown
Change owner and group of the LFC file-directory
lfc-delcomment
Delete the comment associated with the file/directory
lfc-getacl
Get file/directory access control lists
lfc-ln
Make a symbolic link to a file/directory
lfc-ls
List file/directory entries in a directory
lfc-mkdir
Create a directory
lfc-rename
Rename a file/directory
lfc-rm
Remove a file/directory
lfc-setacl
Set file/directory access control lists
lfc-setcomment
Add/replace a comment
Joint CHAIN/EPiKH School for Application Porting in Beijing/China,

19. lcg utils commands Replica Management File Catalog Interaction lcg-aa
lcg-cp
Copies a grid file to a local destination
lcg-cr
Copies a file to a SE and registers the file in the catalog
lcg-del
Delete one file
lcg-rep
Replication between SEs and registration of the replica
lcg-gt
Gets the TURL for a given SURL and transfer protocol
lcg-sd
Sets file status to “Done” for a given SURL in a SRM request
File Catalog Interaction
lcg-aa
Add an alias in LFC for a given GUID
lcg-ra
Remove an alias in LFC for a given GUID
lcg-rf
Registers in LFC a file placed in a SE
lcg-uf
Unregisters in LFC a file placed in a SE
lcg-la
Lists the alias for a given SURL, GUID or LFN
lcg-lg
Get the GUID for a given LFN or SURL
lcg-lr
Lists the replicas for a given GUID, SURL or LFN
Joint CHAIN/EPiKH School for Application Porting in Beijing/China,

20. References gLite documentation homepage DM subsystem documentation
DM subsystem documentation
LFC and DPM documentation
gLite Data Management Tutorial:
Joint CHAIN/EPiKH School for Application Porting in Beijing/China,

21. Reference: https://grid.ct.infn.it/twiki/bin/view/GILDA/DataManagement
Let’s practice!
Reference: 21 21

22. Environment Variables
Pointing to the right BDII
Pointing to the right LFC
echo $LCG_GFAL_INFOSYS
export LCG_GFAL_INFOSYS = gilda-bdii.ct.infn.it:2170
echo $LFC_HOST
export LFC_HOST = lfc-gilda.ct.infn.it 22 22

23. Before starting… Make sure to have a proxy created
voms-proxy-info -all
voms-proxy-init --voms gilda 23 23

24. LFC: creating a directory
List directories
Create your own personal directory inside
lfc-ls /grid/gilda/tutorials/
lfc-mkdir /grid/gilda/tutorials/epikh
You can check the creation typing:
lfc-ls /grid/gilda/tutorials/ 24 24

25. Copying and registering a file 1/2
lcg-cr
Copies a file to a SE and registers the file in the catalogue
lcg-cr --vo <vo name> -l <LFN destination> -d <SE> <local file>
Make sure to have a directory in the LFC
(/grid/gilda/tutorials/yourname/)
Use the lcg-info or lcg-infosites commands to figure out the available SEs
This command will return the GUID for your file 25 25

26. Copying and registering a file 2/2
lcg-infosites --vo gilda se
Avail Space(Kb) Used Space(Kb) Type SEs
n.a se.hpc.iit.bme.hu
n.a se1-egee.srce.hr
n.a gilda-02.pd.infn.it
n.a sirius-se.ct.infn.it
n.a grisuse.scope.unina.it
lcg-cr --vo gilda -l lfn:/grid/gilda/tutorials/epikh/test.txt
-d gilda-02.pd.infn.it file://$HOME/test.txt
guid:0d8ef3b9-7f73-4c57-80c9-e827bace8597 26 26

27. Downloading a file
First of all, let’s download a file from a SE to start “playing” with it.
Basic Usage:
Try it:
lcg-cp --vo <vo name> <LFN origin> <local destination>
lcg-cp --vo gilda lfn:/grid/gilda/tutorials/epikh/test.txt file://$HOME/test1.txt 27 27

28. Replicate a file between SEs
Basic Usage:
Try it:
lcg-rep --vo <vo name> -d <destination SE> <LFN of your file>
lcg-rep --vo gilda -d sirius-se.ct.infn.it lfn:/grid/gilda/tutorials/epikh/test.txt 28 28

29. From where it was downloaded?
List the Replicas of the file:
This command will return the SURL of all replicas
A file can be stored on multiple SE's so that a job can download it from the closest SE while is running.
lcg-lr --vo gilda lfn:/grid/gilda/tutorials/epikh/test.txt 29 29

30. Deleting a file Basic Usage:
When used with '-a' switch will delete all replicas and delete entry from catalog
Try it:
lcg-del -a --vo <vo name> <LFN>
lcg-del -a --vo gilda lfn:/grid/gilda/tutorials/epikh/test.txt 30 30

31. Removing a LFC directory
Basic Usage:
Try it:
lfc-rm -r <LFC file path>
lfc-rm -r /grid/gilda/tutorials/epikh 31 31

32. Joint CHAIN/EPiKH School for Application Porting in Beijing/China, 19