1. www.epikh.eu The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) LFC Installation and Configuration Dong Xu (xud@ihep.ac.cn) IHEP, CAS Asia 1 2010 - EPIKH School for Grid Site Administrators Beijing, 01.09.2010

2. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 2/32 Algiers, Joint EPiKH/EUMEDGRID-Support in Algeria, 27.06.2010 Outline Introduction to LFC Installation and configuration tutorial

3. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 3/32 Introduction LFC: The LCG File Catalog Users and programs produce and require data Data may be stored in Grid datasets (files) –Located in Storage Elements (SEs) –Several replicas of one file in different sites –Accessible by Grid users and applications from “anywhere” –Locatable by the WMS (data requirements in JDL) –Data may be copied from/to local filesystems (WNs, UIs) to the Grid Why File Catalogs are used? –They keep track of the locations of copies (replicas) of Grid files –They provide DM tools and APIs through which users and grid services can deal with files

4. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 4/32 SE Types gLite DPMDisk pool managers (dCache and gLite DPM) –manage distributed storage servers in a centralized way –Physical disks or arrays are combined into a common (virtual) file system –Disks can be dynamically added to the pool –GridFTP server –Secure remote access protocols (gsidcap for dCache, gsirfio for DPM) –SRM interface Classic SE: –GridFTP server –Insecure RFIO daemon (rfiod) – only LAN limited file access –Single disk or disk array –No quota management –Does not support the SRM interface Mass Storage Systems (Castor- CERN Advanced STORage manager) –Files migrated between front-end disk and back-end tape storage hierarchies –GridFTP server –Insecure RFIO (Castor) –Provide a SRM interface with all the benefits

5. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 5/32 You as a user need to know all the systems!!! SRM (Storage Resource Manager ) dCache Own system, own protocols and parameters Castor No connection with dCache or DPM gLite DPM Independent system from dCache or Castor SRM I talk to them on your behalf I will even allocate space for your files And I will use transfer protocols to send your files there

6. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 6/32 SRM Interactions Client SRM Storage 5 1 23 4 1.The client asks the SRM for the file providing an SURL (Site URL) 2.The SRM asks the storage system to provide the file 3.The storage system notifies the availability of the file and its location 4.The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed 5.The client interacts with the storage using the protocol specified in the TURL

7. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 7/32 Files Naming Conventions Logical File Name (LFN) An alias created by a user to refer to some item of data, e.g. “lfn:/grid/gilda/20030203/run2/track1” Globally Unique Identifier (GUID) A non-human-readable unique identifier for an item of data, e.g. “guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6” Site URL (SURL) (or Physical File Name (PFN) or Site FN) The location of an actual piece of data on a storage system, e.g. “srm://grid009.ct.infn.it/dpm/ct.infn.it/gilda/output10_1” (SRM) Transport URL (TURL) Temporary locator of a replica + access protocol: understood by a SE, e.g. “rfio://lxshare0209.cern.ch//data/alice/ntuples.dat”

8. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 8/32 What is a file catalog? SE gLite UI File Catalog SE

9. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 9/32 What is a File Catalog? Each file has a unique identifier Files/directories are organized on a Catalogue –Similar to a filesystem (Logical File Name) There is one Catalogue per VO The data can be stored on several Storage Elements (SE) The Catalogue hides the actual location Catalogue SE Logical File Name LFN : /grid/gilda/dornbirn/file.txt Storage Resource Manager srm://trigrid- ce01.unime.it/dpm/unime.it/home/gilda/generate d/2006-09-20/filef026441a-5834-431f-b28d- 06cb7e4c784f Physical Filename /home/gilda/generated/2006-09-20/filef026441a- 5834-431f-b28d-06cb7e4c784f

10. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 10/32 Different Types of Catalog File Catalog Metadata Catalog Metadata Catalog Metadata Catalog LFN Metad ata File Catalog –Filesystem-like view on logical file names –Keeps track of sites where data is stored –Conflict resolution Replica Catalog –Keeps information at a site (Meta Data Catalog) –Attributes of files on the logical level –Boundary between generic middleware and application layer Replica Catalog Site A GUIDSURL LFN Replica Catalog Site A GUIDSURL LFN Replica Catalog Site B GUIDSURL LFN Replica Catalog Site B GUIDSURL LFN GUID Site ID

11. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 11/32 LFC Commands lfc-chmodChange access mode of the LFC file/directory lfc-chownChange owner and group of the LFC file-directory lfc-delcommentDelete the comment associated with the file/directory lfc-getaclGet file/directory access control lists lfc-lnMake a symbolic link to a file/directory lfc-lsList file/directory entries in a directory lfc-mkdirCreate a directory lfc-renameRename a file/directory lfc-rmRemove a file/directory lfc-setaclSet file/directory access control lists lfc-setcommentAdd/replace a comment Summary of the LFC Catalog commands

12. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 12/32 lcg utils commands Replica Management lcg-cp Copies a grid file to a local destination lcg-cr Copies a file to a SE and registers the file in the catalog lcg-del Delete one file lcg-rep Replication between SEs and registration of the replica lcg-gt Gets the TURL for a given SURL and transfer protocol lcg-sd Sets file status to “Done” for a given SURL in a SRM request File Catalog Interactionlcg-aa Add an alias in LFC for a given GUID lcg-ra Remove an alias in LFC for a given GUID lcg-rf Registers in LFC a file placed in a SE lcg-uf Unregisters in LFC a file placed in a SE lcg-la Lists the alias for a given SURL, GUID or LFN lcg-lg Get the GUID for a given LFN or SURL lcg-lr Lists the replicas for a given GUID, SURL or LFN

13. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 13/32 Installation and configuration

14. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 14/32 Check your host Vi /etc/hosts (for example) # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 202.122.37.107 gilda01.ihep.ac.cn gilda01 Start ntpd ntpdate ntp-1.infn.it service ntpd start

15. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 15/32 Repository settings Specify the mrepo host: export MREPO=http://grid-it.cnaf.infn.it/mrepo/repos/ Configure the repository as follows: REPOS="dag ig glite-generic lcg-ca gilda“ Get repositories with: for name in $REPOS; do wget $MREPO/$name.repo -O /etc/yum.repos.d/$name.repo; done Copy LCG-CA wget http://grid-it.cnaf.infn.it/mrepo/repos/sl5/x86_64/lcg-ca.repo -O /etc/yum.repos.d/lcg-ca.repohttp://grid-it.cnaf.infn.it/mrepo/repos/sl5/x86_64/lcg-ca.repo Copy the gilda utils: wget http://grid018.ct.infn.it/mrepo/repos/gilda.repo -O /etc/yum.repos.d/gilda.repohttp://grid018.ct.infn.it/mrepo/repos/gilda.repo -O /etc/yum.repos.d/gilda.repo Copy the lfc utils: wget http://grid-it.cnaf.infn.it/mrepo/repos/sl5/x86_64/glite-lfc_mysql.repo -O /etc/yum.repos.d/glite-lfc_mysql.repohttp://grid-it.cnaf.infn.it/mrepo/repos/sl5/x86_64

16. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 16/32 install by yum generic installation: update the host: yum clean all update CA: yum install lcg-CA install Gilda utils: yum install gilda_utils UI specific Install: install SE: yum install ig_LFC_mysql Update java yum install java

17. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 17/32 get host certificates ready $# cp hostcert.pem /etc/grid-security/ $# cp hostkey.pem /etc/grid-security/ $# chmod 644 /etc/grid-security/hostcert.pm $# chmod 400 /etc/grid-security/hostkey.pem

18. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 18/32 edit site-info.def Edit site-info.def: vi /opt/glite/yaim/examples/siteinfo/ig-site-info.def SITE_NAME=gilda-BJ-IHEP SITE_EMAIL="your email address" MYSQL_PASSWORD=123456 GROUPS_CONF=/opt/glite/yaim/examples/ig-groups.conf USERS_CONF=/opt/glite/yaim/examples/ig-users.conf JAVA_LOCATION=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre VOS="gilda" VO_GILDA_SW_DIR=$VO_SW_DIR/gilda VO_GILDA_DEFAULT_SE=$DPM_HOST VO_GILDA_STORAGE_DIR=$CLASSIC_STORAGE_DIR//gilda VO_GILDA_QUEUES="short long infinite" VO_GILDA_VOMS_SERVERS="vomss://voms.ct.infn.it:8443/voms/gilda?/gilda" VO_GILDA_VOMSES="'gilda voms.ct.infn.it 15001 /C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it gilda'" VO_GILDA_VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA' '/C=IT/O=INFN/CN=INFN CA'"

19. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 19/32 Edit ig-groups.conf Edit /opt/glite/yaim/examples/siteinfo/ig-groups.conf, "/gilda/ROLE=SoftwareManager":::sgm: "/gilda/grelc/das/*":gilda::: "/gilda"::::

20. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 20/32 Edit other files edit ig-lfc_mysql vi /opt/glite/yaim/examples/siteinfo/services/ig-lfc_mysql LFC_HOST=gilda03.ihep.ac.cn #replace your hostname edit ig-lfc_mysql.pre $#vi /opt/glite/yaim/defaults/ig-lfc_mysql.pre LFC_DB=cns_db_lfc LFC_DB_HOST=$LFC_HOST LFC_DB_PASSWORD=123456 LFC_HOST_ALIAS="" LFC_CENTRAL="gilda" LFC_LOCAL="" LFC_READONLY=no

21. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 21/32 configure LFC with ig-yaim $# /opt/glite/yaim/bin/ig_yaim -c -s /opt/glite/yaim/examples/siteinfo/ig-site-info.def -n ig_LFC_mysql

22. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 22/32 test the function of LFC test the function of LFC(from UI with valid user credential) $# export LFC_HOST=your_lfc_host $# lfc-ls /grid/gilda $# lfc-mkdir /grid/gilda/home $# lfc-mkdir /grid/gilda/home/xud $# lcg-cr --vo gilda -l lfn:/grid/gilda/home/xud/testfile3 -d gilda-02.pd.infn.it file:////etc/hosts $# lfc-ls -l /grid/gilda/home/wuwj/testfile3

23. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 23/32 [xud@gilda08.globus]$ voms-proxy-init -voms gilda Enter GRID pass phrase: Your identity: /C=CN/O=HEP/O=IHEP/OU=CC/CN=dong xu Creating temporary proxy............................................. Done Contacting voms.ct.infn.it:15001 [/C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it] "gilda" Done Creating proxy....................................................................................... Done Your proxy is valid until Tue Aug 31 09:36:18 2010 [xud@gilda08.globus]$ echo $LFC_HOST gilda07.ihep.ac.cn

24. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 24/32 [xud@gilda08.globus]$ lfc-ls /grid/gilda generated malanxin mlx tutorials [xud@gilda08.globus]$ lfc-mkdir /grid/gilda/home [xud@gilda08.globus]$ lfc-ls /grid/gilda generated home malanxin mlx tutorials [xud@gilda08.globus]$ lfc-mkdir /grid/gilda/home/xud [xud@gilda08.globus]$ lcg-cr --vo gilda -l lfn:/grid/gilda/home/xud/testfile3 -d gilda-02.pd.infn.it file:////etc/hosts guid:48943c1e-76b7-4890-a2c8-5437d3d7bdbc [xud@gilda08.globus]$ lfc-ls -l /grid/gilda/home/xud/testfile3 -rw-rw-r-- 1 105 101 246 Aug 30 21:38 /grid/gilda/home/xud/testfile3

25. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 25/32 Bibliography Information on the file catalogs LFC, gfal, lcgutils: “Evolution of LCG2 Data Management (JP Baud, J. Casey)” http://indico.cern.ch/contributionDisplay.py?contribId=278&sess ionId=7&confId=0 LFC installation, administration, migration from RLS: Wiki entries indicated through the presentation: http://goc.grid.sinica.edu.tw/gocwiki/How_to_set_up_an_LFC_se rvice

26. EPIKH School for Grid Site Administrators, Beijing, 30.08.2010 26/32 Thank you very much!