1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Experiences with LFC and comparison with RNS Erwin Laure Erwin.Laure@cern.ch Jean-Philippe Baud Jean-Philippe.Baud@cern.ch Akos Frohner Akos.Frohner@cern.ch

2. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 2 LFC overview Hierarchical namespace POSIX like command line tools POSIX style permissions Virtual users and groups API comparison RNS add RNS list RNS move RNS query permissions in LFC bulk methods in LFC LFC Overview

3. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 3 The LFC stores mappings between –Users’ file names and file locations on the Grid –Stores Permissions and –Ownership –Simple metadata Provides a hierarchical name space Supports GSI security model –Including VOMS based ACLs  Very fine grained control  Implementation based on virtual IDs –Soon: encrypted channels Simple DLI interface –Data Location Interface –GUID Location –Integration with WMS&RBs LCG “File” Catalog … File replica 2 GUID File replica 1 File replica m LFC file name 1 LFC file name n … “Replicas” are “Copies” All files are “Write Once” /grid /vo /data file

4. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 4 MySQL and ORACLE back-ends –Ensures scalability and allows small scale deployment –Read only replication of catalogue is in production (using Oracle streams)‏ Multi-threaded C server –Supports multiple instances for load balancing Thread-safe C clients –Python & Perl bindings –No WS interfaces (apart from DLI) –Command line interface Supports sessions to avoid authentication costs –GSI is very expensive! Bulk methods to reduce the number of round trips – Under test by ATLAS --> 20 times faster Widely used in EGEE: –largest LFC instance contains 8 millions entries LFC

5. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 5 How does it look ? Hierarchical namespace > lfc-ls -l /grid/dteam drwxrwxr-x 2 18947 2688 0 Jun 09 11:11 tests > lfc-ls -l /grid/dteam/tests drwxrwxr-x 2 18947 2688 0 Jun 09 11:15 tutorial > lfc-ls -l /grid/dteam/tests/tutorial drwxrwxr-x 2 18947 2688 0 Jun 09 11:19 directory1 -rw-rw-r-- 1 18947 2688 179 Jun 09 11:17 file.6008 -rw-rw-r-- 1 18947 2688 179 Jun 09 11:18 file.6009

6. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 6 Using LFC From a UI (as a user)‏  export LFC_HOST=mylfc.domain.name  voms-proxy-init –voms myVO  lfc-ls /grid/myVO  lfc-mkdir /grid/myVO/test_dir  lfc-rm –R /grid/myVO/test_dir  lcg-cr --vo myVO –d mySE.domain.name file:/path/to/file

7. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 7 Permissions Permissions/ownership  lfc-chmod  lfc-chown (only admin can use it)‏ Virtual uid, gid  lfc-ls –l …  lfc-enterusrmap, lfc-modifyusrmap, lfc-listusrmap  lfc-entergrpmap, lfc-modifygrpmap, lfc-listgrpmap POSIX ACLs  lfc-getacl, lfc-setacl Secondary groups via VOMS groups & roles

8. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 8 DN: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=erwin/CN=589293/CN=Erwin Laure LFC daemon LFC DB voms-proxy-init -voms dteam lfc-ls /grid/dteam/generated no need to create pool accounts no need to change the /etc/passwd file faster check on ACL than with string/pattern matching on DN/FQAN Does this DN exists in Cns_userinfo? No -> create it! Does this FQAN exists in Cns_groupinfo? No -> create it! DPNS DB Cns_userinfo 5428 /DC=ch/DC=cer/n../CN=589293/CN=Erwin Laure Cns_groupinfo 105 dteam Virtual users and groups

9. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 9 LFC overview Hierarchical namespace POSIX like command line tools POSIX style permissions Virtual users and groups API comparison RNS add RNS list RNS move RNS query permissions performance improvements Comparison Overview

10. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 10 RNS: add (string: entry_name, EndpointReferenceType: entry_reference)‏ LFC: lfc_mkdir (string path, mode_t)‏ lfc_mkdirg (string path, string guid, mode_t)‏ lfc_creatg (string path, string guid, mode_t)‏ lfc_addreplica (string path,...)‏ lfc_symlink (string srcpath, string dstpath)‏... /rns:add/entry_name ~ LFN /rns:add/entry_reference ~ replica or symlink RNS is missing the detailed attributes of the replicas. API: add

11. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 11 RNS: list(string: entry_name_regexp)‏ LFC: lfc_DIR lfc_opendir...(string path)‏ dirent lfc_readdir... (lfc_DIR)‏ lfc_readdirc (), lfc_readdirg(), lfc_readdirx ()‏ lfc_readdirxc (), lfc_readdirxp (), lfc_readdirxr ()‏ lfc_rewinddir (lfc_DIR)‏ lfc_closedir (lfc_DIR)‏ lfc_filereplica lfc_listreplica (string, string, int, lfc_list)‏ lfc_filereplica lfc_listreplicax (string, string, string, int, lfc_list)‏ lfc_filereplicax lfc_listrepset (string, int, lfc_list)‏ The problematic part is listing directories recursively. API: list

12. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 12 RNS: move (EndpointReferenceType: parent, string: entry_name)‏ LFC: lfc_rename(string src, string dst)‏ API: move

13. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 13 RNS: query()‏ LFC: lfc_stat (string path, lfc_filestat)‏ lfc_statg (), lfc_statr (), lfc_statx (), lfc_lstat()‏ lfc_getlinks (string, string, int *, struct lfc_linkinfo[])‏ lfc_getreplica (string, string, string, int *, struct lfc_filereplica[])‏ lfc_getreplicas (int, string[], string, int *, lfc_filereplicas[])‏ lfc_getreplicasl (int, string[], string, int *, lfc_filereplicas[])‏ lfc_getreplicax (string, string, string, int *, lfc_filereplicax[])‏ The attributes of a namespace entry should be defined, allowing specialized queries and lookups. API: query

14. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 14 RNS: remove (string: entry_name_regexp)‏ LFC: lfc_unlink (string)‏ lfc_undelete (string)‏ lfc_delcomment (string)‏ lfc_delete (string)‏ lfc_delfilesbyguid (int, string[], int, int *, int **)‏ lfc_delfilesbyname (int, string[], int, int *, int **)‏ lfc_delfilesbypattern (string, string, int, int *, struct lfc_filestatus **)‏ lfc_delreplica (string, struct lfc_fileid *, string)‏ lfc_delreplicas (int, string[], string, int *, int **)‏ lfc_rmdir (string)‏ API: remove

15. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 15 LFC: bulk methods: cuts the round-trip time lfc_delreplicas(), lfc_delfilesbyguid(), lfc_delfilesbyname(), lfc_delfilesbypattern(), lfc_getreplicas(), lfc_getreplicasl() session: cuts the authentication time lfc_startsess(), lfc_endsess() transactions: cuts the authentication time, and provides roll-back for complex changes lfc_starttrans(), lfc_aborttrans(), lfc_endtrans()‏ API: performance improvements

16. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 16 Posix style permission with grid credentials: lfc_access (string path, int)‏ lfc_chmod (string path, mode_t)‏ lfc_chown (string path, uid_t, gid_t)‏ lfc_getacl (string path, int nentries, lfc_acl[])‏ lfc_setacl (string path, int nentries, lfc_acl[])‏ Virtual users and groups (mapping to Posix numbers): lfc_entergrpmap (gid_t, string)lfc_getgrpbynam (string, gid_t *) lfc_getgrpbygid (gid_t, string)lfc_getgrpbygids (int, gid_t *, string[]) lfc_getgrpmap (int *, struct lfc_groupinfo **)‏ lfc_modifygrpmap (gid_t, string)lfc_rmgrpmap (gid_t, string)‏ lfc_enterusrmap (uid_t, string) lfc_getusrbynam (string, uid_t *) lfc_getusrbyuid (uid_t, string)‏ lfc_getusrmap (int *, struct lfc_userinfo **)‏ lfc_modifyusrmap (uid_t, string)lfc_rmusrmap (uid_t, string)‏ API: permissions

17. Enabling Grids for E-sciencE INFSO-RI-508833 OGF GFS-WG: LFC 17 Documentation Main LFC/DPM documentation page –https://twiki.cern.ch/twiki/bin/view/LCG/DataManagementTophttps://twiki.cern.ch/twiki/bin/view/LCG/DataManagementTop LFC Admin Guide –https://twiki.cern.ch/twiki/bin/view/LCG/LfcAdminGuidehttps://twiki.cern.ch/twiki/bin/view/LCG/LfcAdminGuide Troubleshooting page –https://twiki.cern.ch/twiki/bin/view/LCG/LfcTroubleshootinghttps://twiki.cern.ch/twiki/bin/view/LCG/LfcTroubleshooting