The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) LFC Installation and Configuration Dong Xu IHEP, CAS Asia EPIKH School for Grid Site Administrators Beijing,

EPIKH School for Grid Site Administrators, Beijing, /32 Algiers, Joint EPiKH/EUMEDGRID-Support in Algeria, Outline Introduction to LFC Installation and configuration tutorial

EPIKH School for Grid Site Administrators, Beijing, /32 Introduction LFC: The LCG File Catalog Users and programs produce and require data Data may be stored in Grid datasets (files) –Located in Storage Elements (SEs) –Several replicas of one file in different sites –Accessible by Grid users and applications from “anywhere” –Locatable by the WMS (data requirements in JDL) –Data may be copied from/to local filesystems (WNs, UIs) to the Grid Why File Catalogs are used? –They keep track of the locations of copies (replicas) of Grid files –They provide DM tools and APIs through which users and grid services can deal with files

EPIKH School for Grid Site Administrators, Beijing, /32 SE Types gLite DPMDisk pool managers (dCache and gLite DPM) –manage distributed storage servers in a centralized way –Physical disks or arrays are combined into a common (virtual) file system –Disks can be dynamically added to the pool –GridFTP server –Secure remote access protocols (gsidcap for dCache, gsirfio for DPM) –SRM interface Classic SE: –GridFTP server –Insecure RFIO daemon (rfiod) – only LAN limited file access –Single disk or disk array –No quota management –Does not support the SRM interface Mass Storage Systems (Castor- CERN Advanced STORage manager) –Files migrated between front-end disk and back-end tape storage hierarchies –GridFTP server –Insecure RFIO (Castor) –Provide a SRM interface with all the benefits

EPIKH School for Grid Site Administrators, Beijing, /32 You as a user need to know all the systems!!! SRM (Storage Resource Manager ) dCache Own system, own protocols and parameters Castor No connection with dCache or DPM gLite DPM Independent system from dCache or Castor SRM I talk to them on your behalf I will even allocate space for your files And I will use transfer protocols to send your files there

EPIKH School for Grid Site Administrators, Beijing, /32 SRM Interactions Client SRM Storage The client asks the SRM for the file providing an SURL (Site URL) 2.The SRM asks the storage system to provide the file 3.The storage system notifies the availability of the file and its location 4.The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed 5.The client interacts with the storage using the protocol specified in the TURL

EPIKH School for Grid Site Administrators, Beijing, /32 Files Naming Conventions Logical File Name (LFN) An alias created by a user to refer to some item of data, e.g. “lfn:/grid/gilda/ /run2/track1” Globally Unique Identifier (GUID) A non-human-readable unique identifier for an item of data, e.g. “guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6” Site URL (SURL) (or Physical File Name (PFN) or Site FN) The location of an actual piece of data on a storage system, e.g. “srm://grid009.ct.infn.it/dpm/ct.infn.it/gilda/output10_1” (SRM) Transport URL (TURL) Temporary locator of a replica + access protocol: understood by a SE, e.g. “rfio://lxshare0209.cern.ch//data/alice/ntuples.dat”

EPIKH School for Grid Site Administrators, Beijing, /32 What is a file catalog? SE gLite UI File Catalog SE

EPIKH School for Grid Site Administrators, Beijing, /32 What is a File Catalog? Each file has a unique identifier Files/directories are organized on a Catalogue –Similar to a filesystem (Logical File Name) There is one Catalogue per VO The data can be stored on several Storage Elements (SE) The Catalogue hides the actual location Catalogue SE Logical File Name LFN : /grid/gilda/dornbirn/file.txt Storage Resource Manager srm://trigrid- ce01.unime.it/dpm/unime.it/home/gilda/generate d/ /filef026441a f-b28d- 06cb7e4c784f Physical Filename /home/gilda/generated/ /filef026441a f-b28d-06cb7e4c784f

EPIKH School for Grid Site Administrators, Beijing, /32 Different Types of Catalog File Catalog Metadata Catalog Metadata Catalog Metadata Catalog LFN Metad ata File Catalog –Filesystem-like view on logical file names –Keeps track of sites where data is stored –Conflict resolution Replica Catalog –Keeps information at a site (Meta Data Catalog) –Attributes of files on the logical level –Boundary between generic middleware and application layer Replica Catalog Site A GUIDSURL LFN Replica Catalog Site A GUIDSURL LFN Replica Catalog Site B GUIDSURL LFN Replica Catalog Site B GUIDSURL LFN GUID Site ID

EPIKH School for Grid Site Administrators, Beijing, /32 LFC Commands lfc-chmodChange access mode of the LFC file/directory lfc-chownChange owner and group of the LFC file-directory lfc-delcommentDelete the comment associated with the file/directory lfc-getaclGet file/directory access control lists lfc-lnMake a symbolic link to a file/directory lfc-lsList file/directory entries in a directory lfc-mkdirCreate a directory lfc-renameRename a file/directory lfc-rmRemove a file/directory lfc-setaclSet file/directory access control lists lfc-setcommentAdd/replace a comment Summary of the LFC Catalog commands

EPIKH School for Grid Site Administrators, Beijing, /32 lcg utils commands Replica Management lcg-cp Copies a grid file to a local destination lcg-cr Copies a file to a SE and registers the file in the catalog lcg-del Delete one file lcg-rep Replication between SEs and registration of the replica lcg-gt Gets the TURL for a given SURL and transfer protocol lcg-sd Sets file status to “Done” for a given SURL in a SRM request File Catalog Interactionlcg-aa Add an alias in LFC for a given GUID lcg-ra Remove an alias in LFC for a given GUID lcg-rf Registers in LFC a file placed in a SE lcg-uf Unregisters in LFC a file placed in a SE lcg-la Lists the alias for a given SURL, GUID or LFN lcg-lg Get the GUID for a given LFN or SURL lcg-lr Lists the replicas for a given GUID, SURL or LFN

EPIKH School for Grid Site Administrators, Beijing, /32 Installation and configuration

EPIKH School for Grid Site Administrators, Beijing, /32 Check your host Vi /etc/hosts (for example) # Do not remove the following line, or various programs # that require network functionality will fail localhost.localdomain localhost gilda01.ihep.ac.cn gilda01 Start ntpd ntpdate ntp-1.infn.it service ntpd start

EPIKH School for Grid Site Administrators, Beijing, /32 Repository settings Specify the mrepo host: export MREPO= Configure the repository as follows: REPOS="dag ig glite-generic lcg-ca gilda“ Get repositories with: for name in $REPOS; do wget $MREPO/$name.repo -O /etc/yum.repos.d/$name.repo; done Copy LCG-CA wget -O /etc/yum.repos.d/lcg-ca.repohttp://grid-it.cnaf.infn.it/mrepo/repos/sl5/x86_64/lcg-ca.repo Copy the gilda utils: wget -O /etc/yum.repos.d/gilda.repohttp://grid018.ct.infn.it/mrepo/repos/gilda.repo -O /etc/yum.repos.d/gilda.repo Copy the lfc utils: wget -O /etc/yum.repos.d/glite-lfc_mysql.repohttp://grid-it.cnaf.infn.it/mrepo/repos/sl5/x86_64

EPIKH School for Grid Site Administrators, Beijing, /32 install by yum generic installation: update the host: yum clean all update CA: yum install lcg-CA install Gilda utils: yum install gilda_utils UI specific Install: install SE: yum install ig_LFC_mysql Update java yum install java

EPIKH School for Grid Site Administrators, Beijing, /32 get host certificates ready $# cp hostcert.pem /etc/grid-security/ $# cp hostkey.pem /etc/grid-security/ $# chmod 644 /etc/grid-security/hostcert.pm $# chmod 400 /etc/grid-security/hostkey.pem

EPIKH School for Grid Site Administrators, Beijing, /32 edit site-info.def Edit site-info.def: vi /opt/glite/yaim/examples/siteinfo/ig-site-info.def SITE_NAME=gilda-BJ-IHEP SITE_ ="your address" MYSQL_PASSWORD= GROUPS_CONF=/opt/glite/yaim/examples/ig-groups.conf USERS_CONF=/opt/glite/yaim/examples/ig-users.conf JAVA_LOCATION=/usr/lib/jvm/java openjdk x86_64/jre VOS="gilda" VO_GILDA_SW_DIR=$VO_SW_DIR/gilda VO_GILDA_DEFAULT_SE=$DPM_HOST VO_GILDA_STORAGE_DIR=$CLASSIC_STORAGE_DIR//gilda VO_GILDA_QUEUES="short long infinite" VO_GILDA_VOMS_SERVERS="vomss://voms.ct.infn.it:8443/voms/gilda?/gilda" VO_GILDA_VOMSES="'gilda voms.ct.infn.it /C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it gilda'" VO_GILDA_VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA' '/C=IT/O=INFN/CN=INFN CA'"

EPIKH School for Grid Site Administrators, Beijing, /32 Edit ig-groups.conf Edit /opt/glite/yaim/examples/siteinfo/ig-groups.conf, "/gilda/ROLE=SoftwareManager":::sgm: "/gilda/grelc/das/*":gilda::: "/gilda"::::

EPIKH School for Grid Site Administrators, Beijing, /32 Edit other files edit ig-lfc_mysql vi /opt/glite/yaim/examples/siteinfo/services/ig-lfc_mysql LFC_HOST=gilda03.ihep.ac.cn #replace your hostname edit ig-lfc_mysql.pre $#vi /opt/glite/yaim/defaults/ig-lfc_mysql.pre LFC_DB=cns_db_lfc LFC_DB_HOST=$LFC_HOST LFC_DB_PASSWORD= LFC_HOST_ALIAS="" LFC_CENTRAL="gilda" LFC_LOCAL="" LFC_READONLY=no

EPIKH School for Grid Site Administrators, Beijing, /32 configure LFC with ig-yaim $# /opt/glite/yaim/bin/ig_yaim -c -s /opt/glite/yaim/examples/siteinfo/ig-site-info.def -n ig_LFC_mysql

EPIKH School for Grid Site Administrators, Beijing, /32 test the function of LFC test the function of LFC(from UI with valid user credential) $# export LFC_HOST=your_lfc_host $# lfc-ls /grid/gilda $# lfc-mkdir /grid/gilda/home $# lfc-mkdir /grid/gilda/home/xud $# lcg-cr --vo gilda -l lfn:/grid/gilda/home/xud/testfile3 -d gilda-02.pd.infn.it file:////etc/hosts $# lfc-ls -l /grid/gilda/home/wuwj/testfile3

EPIKH School for Grid Site Administrators, Beijing, /32 voms-proxy-init -voms gilda Enter GRID pass phrase: Your identity: /C=CN/O=HEP/O=IHEP/OU=CC/CN=dong xu Creating temporary proxy Done Contacting voms.ct.infn.it:15001 [/C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it] "gilda" Done Creating proxy Done Your proxy is valid until Tue Aug 31 09:36: echo $LFC_HOST gilda07.ihep.ac.cn

EPIKH School for Grid Site Administrators, Beijing, /32 lfc-ls /grid/gilda generated malanxin mlx tutorials lfc-mkdir /grid/gilda/home lfc-ls /grid/gilda generated home malanxin mlx tutorials lfc-mkdir /grid/gilda/home/xud lcg-cr --vo gilda -l lfn:/grid/gilda/home/xud/testfile3 -d gilda-02.pd.infn.it file:////etc/hosts guid:48943c1e-76b a2c8-5437d3d7bdbc lfc-ls -l /grid/gilda/home/xud/testfile3 -rw-rw-r Aug 30 21:38 /grid/gilda/home/xud/testfile3

EPIKH School for Grid Site Administrators, Beijing, /32 Bibliography Information on the file catalogs LFC, gfal, lcgutils: “Evolution of LCG2 Data Management (JP Baud, J. Casey)” ionId=7&confId=0 LFC installation, administration, migration from RLS: Wiki entries indicated through the presentation: rvice

EPIKH School for Grid Site Administrators, Beijing, /32 Thank you very much!