1. EMI is partially funded by the European Commission under Grant Agreement RI-261611 DPM in EMI-II HTTP and NFS interfaces Oliver Keeble On behalf of DPM team CERN EGI CF 29/03/2012

2. EMI INFSO-RI-261611 Over 250 DPM endpoints HTTP and NFS interfaces for DPM/LFC

3. EMI INFSO-RI-261611 Reliable grid disk storage solution Simplicity Stability VOMS security Access protocol support SRM Key Features HTTP and NFS interfaces for DPM/LFC

4. EMI INFSO-RI-261611 Architecture HTTP and NFS interfaces for DPM/LFC HEAD NODE DISK NODES

5. EMI INFSO-RI-261611 DPM daemons HTTP and NFS interfaces for DPM/LFC CLIENT DPNSDPMSRMHTTP/DAVNFS GRIDFTP RFIOHTTP/DAVNFSXROOT HEAD NODE DISK NODE(s) FILE METADATA OPS FILE ACCESS OPS RFIO HTTP NFS XROOT

6. EMI INFSO-RI-261611 EPEL compliant packaging (with source rpms) NFS HTTP / DAV Nagios probes GridPP Contrib Admin Tools Initial SL6 support Synchronous GET requests (dpm_get) Thread safe client authz choice (CSEC_MECH) Improved DNS load balancing What comes with EMI-II HTTP and NFS interfaces for DPM/LFC

7. EMI INFSO-RI-261611 Installation is from the EMI-II repositories. We recommend following EMI instructions for repos – Gives EMI repos priority We make simultaneous releases to EPEL. – packages will arrive in EPEL earlier. Installation for EMI-II HTTP and NFS interfaces for DPM/LFC

8. EMI INFSO-RI-261611 Yaim is supported: /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n emi_dpm_disk /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n emi_dpm_mysql YAIM configuration HTTP and NFS interfaces for DPM/LFC

9. EMI INFSO-RI-261611 In EMI-II, packaging is EPEL compliant, so you should find config files where you expect. – Configs in /etc/sysconfig – Logs in /var/log – Etc… Underlying config HTTP and NFS interfaces for DPM/LFC

10. EMI INFSO-RI-261611 ● Standard HTTP and DAV access ● Use with standard clients ● Plus, with some extra on the clients ● Support for multi-stream transfers – Big throughput gain for WAN ● 3rd party copies using HTTP – FTS will support this – Destination can be any HTTP that support PUT ● Source can be DPM and, in the future, dCache DPM-HTTP/DAV: Motivation HTTP and NFS interfaces for DPM/LFC

11. EMI INFSO-RI-261611 ● For each node (LFC, DPM Head or Disk) ● Will arrive automatically with metapackage ● Otherwise, yum install lcgdm-dav-server ● Configuration files are under the expected location ● /etc/httpd/conf.d ● Specific file: zlcgdm-dav.conf ● All the rest is standard Apache stuff DPM-HTTP/DAV: Installation HTTP and NFS interfaces for DPM/LFC

12. EMI INFSO-RI-261611 ● Any web browser can be used to explore the namespace and get a file ● Some metadata given (mode, size, owner...) ● Any DAV client can be used to retrieve more metadata (replicas, comments, …) ● Almost none support redirections (Some do on GET) ● i.e. Cadaver, MS Windows Explorer ● A specific client aware of some extensions is provided ● htcopy – Download and upload with multiple streams – 3rd party copies (including delegation) DPM-HTTP/DAV: Usage HTTP and NFS interfaces for DPM/LFC

13. EMI INFSO-RI-261611 And if you install it on LFC… HTTP and NFS interfaces for DPM/LFC CLIENT LFC DPM HEAD DPM DISK GET / PUT REDIRECT DATA This has been demonstrated using the HTTP interface to LFC & DPM Can even incorporate dCache

14. EMI INFSO-RI-261611 One daemon, deployed on all nodes Current implementation relies on DPNS and DPM daemon Future implementations will talk directly to the database, using our new DMLITE library DPM-NFS: Server Overview https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/NFS41/Admin/Overview HTTP and NFS interfaces for DPM/LFC

15. EMI INFSO-RI-261611 EMI2 to have the first production version – Read-only at the start – Writable with a subsequent update Userspace daemon (based on Ganesha) Fully functional Already deployed at a few sites as a beta – ASGC, UK GridPP, … – And we keep releasing a new beta every month Unclear how to provide X509 authorization – Kerberos or machine trust possible though DPM-NFS: Server Availability HTTP and NFS interfaces for DPM/LFC

16. EMI INFSO-RI-261611 In EMI2, the metapackages (emi-dpm*) will include the DPM NFS daemon – Otherwise: yum install dpm-nfs-server dpm-devel glite-yaim-dpm vim – DPM_NFS="yes“ yaim -r -s site-info/site-info.def -n emi- dpm_mysql -f config_DPM_nfs – Starting dpm-nfs: [ OK ] But make sure you check the latest instructions DPM-NFS: Server Installation https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/NFS41/Admin/Setup HTTP and NFS interfaces for DPM/LFC

17. EMI INFSO-RI-261611 DPM-NFS: Clients HTTP and NFS interfaces for DPM/LFC pNFS support in kernel >= 2.6.38 – Your Fedora / Debian has it… seriously nfs-utils >= 1.2.3 But the ones we currently support do not – RHEL5: 2.6.18 – Debian 6/Squeeze: 2.6.32 RHEL 6.2 has a pNFS aware kernel We provide packages for RHEL5/RHEL6 – Enabled pNFS in the elrepo mainline kernel – nfs-utils we package ourselves This slide should be blank…

18. EMI INFSO-RI-261611 SLC5 (requires elrepo) – rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org – rpm -Uvh http://elrepo.org/elrepo-release-5-3.el5.elrepo.noarch.rpmhttp://elrepo.org/elrepo-release-5-3.el5.elrepo.noarch.rpm – Add nfs41 repo (see details in wiki link below) – yum install --enablerepo=elrepo-kernel --enablerepo=nfs41 kernel-ml nfs-utils – echo nfs_layout_nfsv41_files >> /etc/rc.modules – chmod +x /etc/rc.modules – (without these two last steps, pNFS file layout will not be enabled) SLC6 – kernel with pNFS is already there – but make sure the nfs_layout_nfsv41_files module is loaded https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/NFS41/Client/Install DPM-NFS: Client Installation HTTP and NFS interfaces for DPM/LFC

19. EMI INFSO-RI-261611 In EMI-II we will release – HTTP support for DPM and LFC – NFS support for DPM Part of a strategic orientation toward support of standards Please try them and let us know – dpm-users-forum@cern.ch – GGUS https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm Summary HTTP and NFS interfaces for DPM/LFC

20. EMI INFSO-RI-261611 Extras HTTP and NFS interfaces for DPM/LFC

21. EMI INFSO-RI-261611 ● On DPM ● Head – Defaults are good enough – change DPMHost and NSHost if not localhost ● Disk – Must change DPMHost, NSHost and NodeType ● On LFC ● Need to change the NodeType and Location ● Must remove DPMHost ● Read-only by default DPM-DAV: Configuration HTTP and NFS interfaces for DPM/LFC

22. EMI INFSO-RI-261611 ● mod_ssl gives support for user certificates ● If proxies wanted, mod_gridsite must be installed and enabled ● Gridsite is used for delegation too (3rd party copies) ● Remember to configure properly the certificate and revocation list location ● More detailed information: ● https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/WebDA V/Setup https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/WebDA V/Setup DPM-DAV: Configuration HTTP and NFS interfaces for DPM/LFC

23. EMI INFSO-RI-261611 ● Global Access Service ● What is it – Browse an LFC through HTTP – Click on the file you want – You get it ● Will be enabled by default – Both on DPM and LFC (and dCache will join) – The number of replicas you get in one go can be configured (3 by default) DPM-DAV: Coming soon HTTP and NFS interfaces for DPM/LFC

24. EMI INFSO-RI-261611 DPM is widely deployed and used – 32PB advertised in the InfoSys – Over 200 sites 56 countries Usage outside HEP – 7 sites over 1PB – Over 300 VOs with access to DPMs DPM numbers HTTP and NFS interfaces for DPM/LFC