LFC Installation and Configuration Dong Xu (xud@ihep.ac.cn) IHEP, CAS Asia 1 2010 - EPIKH School for Grid Site Administrators Beijing, 01.09.2010

Outline Introduction to LFC Installation and configuration tutorial Algiers, Joint EPiKH/EUMEDGRID-Support in Algeria, 27.06.2010

Introduction LFC: The LCG File Catalog Users and programs produce and require data Data may be stored in Grid datasets (files) Located in Storage Elements (SEs) Several replicas of one file in different sites Accessible by Grid users and applications from “anywhere” Locatable by the WMS (data requirements in JDL) Data may be copied from/to local filesystems (WNs, UIs) to the Grid Why File Catalogs are used? They keep track of the locations of copies (replicas) of Grid files They provide DM tools and APIs through which users and grid services can deal with files

SE Types Disk pool managers (dCache and gLite DPM) Classic SE: manage distributed storage servers in a centralized way Physical disks or arrays are combined into a common (virtual) file system Disks can be dynamically added to the pool GridFTP server Secure remote access protocols (gsidcap for dCache, gsirfio for DPM) SRM interface Classic SE: Insecure RFIO daemon (rfiod) – only LAN limited file access Single disk or disk array No quota management Does not support the SRM interface Mass Storage Systems (Castor- CERN Advanced STORage manager) Files migrated between front-end disk and back-end tape storage hierarchies Insecure RFIO (Castor) Provide a SRM interface with all the benefits

SRM (Storage Resource Manager ) dCache Own system, own protocols and parameters You as a user need to know all the systems!!! I talk to them on your behalf I will even allocate space for your files And I will use transfer protocols to send your files there gLite DPM Independent system from dCache or Castor SRM Castor No connection with dCache or DPM

SRM Interactions SRM Client Storage 4 1 2 3 5 The client asks the SRM for the file providing an SURL (Site URL) The SRM asks the storage system to provide the file The storage system notifies the availability of the file and its location The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed The client interacts with the storage using the protocol specified in the TURL

Files Naming Conventions Logical File Name (LFN) An alias created by a user to refer to some item of data, e.g. “lfn:/grid/gilda/20030203/run2/track1” Globally Unique Identifier (GUID) A non-human-readable unique identifier for an item of data, e.g. “guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6” Site URL (SURL) (or Physical File Name (PFN) or Site FN) The location of an actual piece of data on a storage system, e.g. “srm://grid009.ct.infn.it/dpm/ct.infn.it/gilda/output10_1” (SRM) Transport URL (TURL) Temporary locator of a replica + access protocol: understood by a SE, e.g. “rfio://lxshare0209.cern.ch//data/alice/ntuples.dat”

What is a file catalog? File Catalog SE SE SE gLite UI

What is a File Catalog? Each file has a unique identifier Files/directories are organized on a Catalogue Similar to a filesystem (Logical File Name) There is one Catalogue per VO The data can be stored on several Storage Elements (SE) The Catalogue hides the actual location Logical File Name LFN : /grid/gilda/dornbirn/file.txt Storage Resource Manager srm://trigrid-ce01.unime.it/dpm/unime.it/home/gilda/generated/2006-09-20/filef026441a-5834-431f-b28d-06cb7e4c784f Physical Filename /home/gilda/generated/2006-09-20/filef026441a-5834-431f-b28d-06cb7e4c784f Catalogue SE SE SE SE SE

Different Types of Catalog File Catalog Filesystem-like view on logical file names Keeps track of sites where data is stored Conflict resolution Replica Catalog Keeps information at a site (Meta Data Catalog) Attributes of files on the logical level Boundary between generic middleware and application layer Metadata Catalog Metadata Catalog Metadata Catalog Metadata Metadata Metadata File Catalog LFN GUID Site ID File Catalog File Catalog LFN GUID Site ID Site ID Site ID Replica Catalog Site A GUID SURL LFN Replica Catalog Site A Replica Catalog Site B GUID SURL LFN Replica Catalog Site B Replica Catalog Site B LFN GUID SURL LFN GUID SURL SURL SURL

LFC Commands Summary of the LFC Catalog commands lfc-chmod Change access mode of the LFC file/directory lfc-chown Change owner and group of the LFC file-directory lfc-delcomment Delete the comment associated with the file/directory lfc-getacl Get file/directory access control lists lfc-ln Make a symbolic link to a file/directory lfc-ls List file/directory entries in a directory lfc-mkdir Create a directory lfc-rename Rename a file/directory lfc-rm Remove a file/directory lfc-setacl Set file/directory access control lists lfc-setcomment Add/replace a comment

lcg utils commands Replica Management File Catalog Interaction lcg-aa lcg-cp Copies a grid file to a local destination lcg-cr Copies a file to a SE and registers the file in the catalog lcg-del Delete one file lcg-rep Replication between SEs and registration of the replica lcg-gt Gets the TURL for a given SURL and transfer protocol lcg-sd Sets file status to “Done” for a given SURL in a SRM request File Catalog Interaction lcg-aa Add an alias in LFC for a given GUID lcg-ra Remove an alias in LFC for a given GUID lcg-rf Registers in LFC a file placed in a SE lcg-uf Unregisters in LFC a file placed in a SE lcg-la Lists the alias for a given SURL, GUID or LFN lcg-lg Get the GUID for a given LFN or SURL lcg-lr Lists the replicas for a given GUID, SURL or LFN

Installation and configuration

Check your host Vi /etc/hosts (for example) Vi /etc/hosts.allow # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 202.122.37.107 gilda01.ihep.ac.cn gilda01 Vi /etc/hosts.allow [root@gilda07 ~]# more /etc/hosts.allow # # hosts.allow This file describes the names of the hosts which are # allowed to use the local INET services, as decided # by the '/usr/sbin/tcpd' server. slapd: all Start ntpd ntpdate ntp-1.infn.it service ntpd start

Repository settings Specify the mrepo host: export MREPO=http://grid-it.cnaf.infn.it/mrepo/repos/sl5/x86_64/ Configure the repository as follows: REPOS="dag ig glite-generic lcg-ca gilda glite-lfc_mysql” Get repositories with: for name in $REPOS; do wget $MREPO/$name.repo -O /etc/yum.repos.d/$name.repo; done Copy the gilda utils: wget http://grid018.ct.infn.it/mrepo/repos/gilda.repo -O /etc/yum.repos.d/gilda.repo

install by yum update the host: Update java yum install java yum clean all Update java yum install java generic installation: update CA: yum install lcg-CA install Gilda utils: yum install gilda_utils UI specific Install: install LFC: yum install ig_LFC_mysql

get host certificates ready $# cp hostcert.pem /etc/grid-security/ $# cp hostkey.pem /etc/grid-security/ $# chmod 644 /etc/grid-security/hostcert.pm $# chmod 400 /etc/grid-security/hostkey.pem

edit /opt/glite/yaim/examples/siteinfo/ig-site-info.def SITE_NAME=gilda-BJ-IHEP SITE_EMAIL="your email address" MYSQL_PASSWORD=123456 GROUPS_CONF=/opt/glite/yaim/examples/ig-groups.conf USERS_CONF=/opt/glite/yaim/examples/ig-users.conf JAVA_LOCATION=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre VOS="gilda" VO_GILDA_SW_DIR=$VO_SW_DIR/gilda VO_GILDA_DEFAULT_SE=$DPM_HOST VO_GILDA_STORAGE_DIR=$CLASSIC_STORAGE_DIR//gilda VO_GILDA_QUEUES="short long infinite" VO_GILDA_VOMS_SERVERS="vomss://voms.ct.infn.it:8443/voms/gilda?/gilda" VO_GILDA_VOMSES="'gilda voms.ct.infn.it 15001 /C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it gilda'" VO_GILDA_VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA' '/C=IT/O=INFN/CN=INFN CA'“ LFC_HOST=gilda17.ihep.ac.cn # add a new line LFC_DB=cns_db_lfc # add a new line LFC_DB_HOST=$LFC_HOST # add a new line LFC_DB_PASSWORD=123456 # add a new line LFC_HOST_ALIAS="" # add a new line LFC_CENTRAL="gilda" # add a new line LFC_LOCAL="" # add a new line LFC_READONLY=no # add a new line

Edit ig-groups.conf Edit /opt/glite/yaim/examples/siteinfo/ig-groups.conf, "/gilda/ROLE=SoftwareManager":::sgm: "/gilda/grelc/das/*":gilda::: "/gilda":::: Edit /etc/hosts.allow sldap: all # add a new line

Edit other files edit ig-lfc_mysql vi /opt/glite/yaim/examples/siteinfo/services/ig-lfc_mysql LFC_HOST=gilda03.ihep.ac.cn #replace your hostname LFC_DB_PASSWORD=123456

configure LFC with ig-yaim $# /opt/glite/yaim/bin/ig_yaim -c -s /opt/glite/yaim/examples/siteinfo/ig-site-info.def -n ig_LFC_mysql

test the function of LFC test the function of LFC(from UI with valid user credential) $# export LFC_HOST=your_lfc_host $# lfc-ls /grid/gilda $# lfc-mkdir /grid/gilda/home $# lfc-mkdir /grid/gilda/home/xud $# lcg-cr --vo gilda -l lfn:/grid/gilda/home/xud/testfile3 -d gilda-02.pd.infn.it file:////etc/hosts $# lfc-ls -l /grid/gilda/home/wuwj/testfile3

[xud@gilda08 .globus]$ voms-proxy-init -voms gilda Enter GRID pass phrase: Your identity: /C=CN/O=HEP/O=IHEP/OU=CC/CN=dong xu Creating temporary proxy ............................................. Done Contacting voms.ct.infn.it:15001 [/C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it] "gilda" Done Creating proxy ....................................................................................... Done Your proxy is valid until Tue Aug 31 09:36:18 2010 [xud@gilda08 .globus]$ echo $LFC_HOST gilda07.ihep.ac.cn

[xud@gilda08 .globus]$ lfc-ls /grid/gilda generated malanxin mlx tutorials [xud@gilda08 .globus]$ lfc-mkdir /grid/gilda/home home [xud@gilda08 .globus]$ lfc-mkdir /grid/gilda/home/xud [xud@gilda08 .globus]$ lcg-cr --vo gilda -l lfn:/grid/gilda/home/xud/testfile3 -d gilda-02.pd.infn.it file:////etc/hosts guid:48943c1e-76b7-4890-a2c8-5437d3d7bdbc [xud@gilda08 .globus]$ lfc-ls -l /grid/gilda/home/xud/testfile3 -rw-rw-r-- 1 105 101 246 Aug 30 21:38 /grid/gilda/home/xud/testfile3

Bibliography Information on the file catalogs LFC, gfal, lcgutils: “Evolution of LCG2 Data Management (JP Baud, J. Casey)” http://indico.cern.ch/contributionDisplay.py?contribId=278&sessionId=7&confId=0 LFC installation, administration, migration from RLS: Wiki entries indicated through the presentation: http://goc.grid.sinica.edu.tw/gocwiki/How_to_set_up_an_LFC_service

Thank you very much!