1. Data Management
Ouafa Bentaleb
CERIST, Algeria
Africa 6, Rabat ­ Joint CHAIN/EUMEDGRID-Support/EPIKH School Application Porting

2. Outline gLite Data Management Data Management Practical Introduction
Scope of data services in gLite
Examples
Storage Elements
LCG File Catalog
Data Management Practical
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 2

3. The overall goal of this presentation
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting

4. Data Management System (DMS)
Provides file manipulation services for users and other Grid services.
DMS enables the location, access and transfer of data
User do not need to know data location, just the logical name
Data is accessed through standard interfaces
Data can be replicated or transferred to several locations as needed
Data is shared within a VO
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 4 4

5. Scope of data services in gLite
Simply, DMS provides all operation that all of us are used to performing
Uploading /downloading files
Creating file /directories
Renaming file /directories
Deleting file /directories
Moving file /directories
Listing directories
Creating symbolic links
Note: Files are write-once, read-many
Files cannot be changed unless remove or replaced
No intention of providing a global file management system
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 5 5

6. Scope of data services in gLite
Resource centers need meet growing demand for storage
Storage Element capable to manage multiple disk pools
Disk Pool Manager (DPM), dCache, CASTOR
Allows users and applications (programs) to store/retrieve data (files)
Data is stored on different storage systems technologies
Common interface required to hide underlying complexity
Storage Resource Manager (SRM) – storage management protocol
GridFTP – secure file transfer
Data is stored at different locations with separate namespace
File catalogue to provide uniform view of Grid data
LCG File Catalog (LFC)
Applications need to access Grid data management services
Data management API
GFAL
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 6 6

7. Data Management Exemple
Resource
Broker
Storage
Element
Computing
DataSets info
Input “sandbox”
Input “sandbox” + Broker Info
Output “sandbox”
“User interface”
LCG FileCatalogue (LFC)
File replicated onto 2 SEs
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting

8. Data Management Exemple
Storage
Element1
“User interface”
LCG FileCatalogue (LFC)
Element 2
File replicated onto 2 SEs
“Myfile.dat”
Myfile.dat
File_on_se1
File_on_se2
guid
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 8 8

9. Data Management Exemple
Storage
Element1
“User interface”
LCG FileCatalogue (LFC)
Element2
“Myfile.dat”
Myfile.dat
“Logical filename”
File_on_se1 (“SURL”: site URL)
File_on_se2 (“SURL”: site URL)
“GUID” Global Unique Identifier
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 9 9

10. Storage Element (SE)
The Storage Element (SE) is the service which allows users and applications (programs) to store/retrieve data (files)
Provide storage space for files.
Provide transfer protocol (GSIFTP) ~ GSI based FTP server
Provide an interface for the management of disk and tape storage resources: Storage Resource Manager (SRM)
Files located in the Storage Elements (SEs)…
Are mostly write-once, read-many.
Accessible by users and applications from “anywhere” in the Grid.
Several replicas of one file can be replicated at different sites.
Cannot be changed unless remove or replaced.
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 10 10

11. A practical example (1) Storm at SiteA dCache at SiteB DPM at SiteD
She is working on a job which needs:
- read MonteCarlo simulations from siteA
- read experiment data from siteB
- read environmental data from siteC
- write output to home siteD
Storm at SiteA
dCache at SiteB
DPM at SiteD
DPM at SiteC
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting

12. File Naming conventions (1)
Grid Unique IDentifier (GUID)
Every file has a GUID
A non-human-readable unique identifier, e.g.: guid:38ed3f60-c402-11d7-a6b0-f53ee5a37e1d
Note: all replicas of a file will share the same GUID
Logical File Name (LFN)
An alias that can be used to refer to a file, e.g.: lfn://grid/gilda/users/maroc/myfile.dat
Logical File Name 1
Logical File Name N
GUID
...
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 12 12

13. File Naming conventions (2)
Storage URL (SURL) or Physical File Name (PFN)
The location of an actual file on a storage system, e.g.: srm://se01.grid.arn.dz/dpm/home/eumed/project1/test.dat
Note: Used by the system to find where the replica is physically stored
Transport URL (TURL)
Complete URI with the necessary information to access a file in a SE (including the access protocol) e.g.: rfio://lxshare0209.cern.ch//data/alice/ntuples.dat
Logical File Name 1
Physical File SURL 1
TURL 1
...
GUID
...
...
Logical File Name N
Physical File SURL N
TURL 1
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 13

14. Needles in a haystack File Catalogue LFC = LCG File Catalogue
How do I keep track of all files I have on the Grid?
Even if I remember all the LFN’s of my files, what about someone else's files?
How does the Grid keep track of the mapping between LFN(s), GUID and SURL(s)?
LFC = LCG File Catalogue
LCG = LHC Compute Grid
LHC = Large Hadron Collider
File Catalogue
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 14

15. File Catalogue
Is the service which maintains mappings between LFN(s), GUID and SURL(s)
It keeps track of the location of copies (replicas) of files
It consists of a unique catalogue, where the LFN is the main key
Looks like a “top-level” directory in the Grid
For each of the supported VO a separate subdirectory exists under the "/grid" directory.
All members of a given VO have read-write permissions in such a directory
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 15

16. The LFC Service lfn:/grid/gilda/tcaland/mpi.txt File Catalogue SE A
SE B
User Interface
SE C
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 16

17. Job submission – example 1
Small files: InputSandbox / OutputSandbox CE
WMS
User Interface
Worker Nodes
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 17

18. Data Management – example 2CE
WMS
User Interface
Worker Nodes
LFC SE SE
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 18

19. LFC Commands Summary of the LFC Catalog commands lfc-chmod
Change access mode of the LFC file/directory
lfc-chown
Change owner and group of the LFC file-directory
lfc-delcomment
Delete the comment associated with the file/directory
lfc-getacl
Get file/directory access control lists
lfc-ln
Make a symbolic link to a file/directory
lfc-ls
List file/directory entries in a directory
lfc-mkdir
Create a directory
lfc-rename
Rename a file/directory
lfc-rm
Remove a file/directory
lfc-setacl
Set file/directory access control lists
lfc-setcomment
Add/replace a comment
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting

20. lcg utils commands Replica Management File Catalog Interaction lcg-aa
lcg-cp
Copies a grid file to a local destination
lcg-cr
Copies a file to a SE and registers the file in the catalog
lcg-del
Delete one file
lcg-rep
Replication between SEs and registration of the replica
lcg-gt
Gets the TURL for a given SURL and transfer protocol
lcg-sd
Sets file status to “Done” for a given SURL in a SRM request
File Catalog Interaction
lcg-aa
Add an alias in LFC for a given GUID
lcg-ra
Remove an alias in LFC for a given GUID
lcg-rf
Registers in LFC a file placed in a SE
lcg-uf
Unregisters in LFC a file placed in a SE
lcg-la
Lists the alias for a given SURL, GUID or LFN
lcg-lg
Get the GUID for a given LFN or SURL
lcg-lr
Lists the replicas for a given GUID, SURL or LFN
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting

21. References gLite documentation homepage DM subsystem documentation
DM subsystem documentation
LFC and DPM documentation
gLite Data Management Tutorial:
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting

22. Reference: https://grid.ct.infn.it/twiki/bin/view/GILDA/DataManagement
Let’s practice!
Reference:
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 22 22

23. Environment Variables
Pointing to the right BDII
Pointing to the right LFC
echo $LCG_GFAL_INFOSYS
export LCG_GFAL_INFOSYS = bdii.eumedgrid.eu:2170
echo $LFC_HOST
export LFC_HOST = lfc.grid.arn.dz
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 23 23

24. Before starting… Make sure to have a proxy created
voms-proxy-info -all
voms-proxy-init --voms eumed
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 24 24

25. LFC: creating a directory
List directories
Create your own personal directory inside
lfc-ls /grid/eumed/tutorials/
lfc-mkdir /grid/eumed/tutorials/epikh
You can check the creation typing:
lfc-ls /grid/eumed/tutorials/
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 25 25

26. Copying and registering a file 1/2
lcg-cr
Copies a file to a SE and registers the file in the catalogue
lcg-cr --vo <vo name> -l <LFN destination> -d <SE> <local file>
Make sure to have a directory in the LFC
(/grid/eumed/tutorials/yourname/)
Use the lcg-info or lcg-infosites commands to figure out the available SEs
This command will return the GUID for your file
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 26 26

27. Copying and registering a file 2/2
lcg-infosites --vo eumed se
Avail Space(Kb) Used Space(Kb) Type SEs
n.a se01.grid.arn.dz
n.a se02.grid.arn.dz
n.a se03.grid.arn.dz
n.a gilda-02.pd.infn.it
n.a sirius-se.ct.infn.it
n.a grisuse.scope.unina.it
lcg-cr --vo eumed -l lfn:/grid/eumed/tutorials/epikh/test.txt
-d se01.grid.arn.dz file://$HOME/test.txt
guid:0d8ef3b9-7f73-4c57-80c9-e827bace8597
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 27 27

28. Downloading a file
First of all, let’s download a file from a SE to start “playing” with it.
Basic Usage:
Try it:
lcg-cp --vo <vo name> <LFN origin> <local destination>
lcg-cp --vo eumed lfn:/grid/eumed/tutorials/epikh/test.txt file://$HOME/test1.txt
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 28 28

29. Replicate a file between SEs
Basic Usage:
Try it:
lcg-rep --vo <vo name> -d <destination SE> <LFN of your file>
lcg-rep --vo eumed -d se01.grid.arn.dz lfn:/grid/eumed/tutorials/epikh/test.txt
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 29 29

30. From where it was downloaded?
List the Replicas of the file:
This command will return the SURL of all replicas
A file can be stored on multiple SE's so that a job can download it from the closest SE while is running.
lcg-lr --vo eumed lfn:/grid/eumed/tutorials/epikh/test.txt
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 30 30

31. Deleting a file Basic Usage:
When used with '-a' switch will delete all replicas and delete entry from catalog
Try it:
lcg-del -a --vo <vo name> <LFN>
lcg-del -a --vo eumed lfn:/grid/eumed/tutorials/epikh/test.txt
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 31 31

32. Removing a LFC directory
Basic Usage:
Try it:
lfc-rm -r <LFC file path>
lfc-rm -r /grid/eumed/tutorials/epikh
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting 32 32

33. Any questions …?
Ouafa Bentaleb, CERIST Rabat, Joint CHAIN/EUMEDGRID-Support/EPIKH ,Africa School Application Porting