1. Andrea Manzi CERN On behalf of the DPM team HEPiX Fall 2014 Workshop DPM performance tuning hints for HTTP/WebDAV and Xrootd 1 16/10/2014

2. Outline  DPM Overview  Performance Assessment  Internal Testbed  CMS AAA  Performance Tuning  Results  Summary 2 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd

3. Disk Pool Manager (DPM)  The Disk Pool Manager (DPM) is a lightweight storage solution for grid sites. It offers a simple way to create a disk-based grid storage element and supports relevant protocols for file management and access.  It focuses on manageability (ease of installation, configuration, low effort of maintenance), while providing all required functionalities for a grid storage solution:  support for multiple disk server nodes  multiple file replicas in disk pools  multi VOs support  multi protocol: xrootd, HTTP/WebDAV, GridFTP, SRM 3 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd

4. DPM Collaboration  Started in 2013 in order to maintain, develop and support the DPM Software stack for the benefit of the user communities members of the collaboration:  CERN, CESNET, GridPP, Japan, France, Italy and ASGC. 4 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd

5. Deployment statistics  DPM is the most widely deployed grid storage system in EGI and WLCG  Over 200 sites in 50 regions  Over 300 VOs  WLCG statistics:  http://wlcg- mon.cern.ch/dashboard/request.py/siteview#currentV iew=Storage+Deployment http://wlcg- mon.cern.ch/dashboard/request.py/siteview#currentV iew=Storage+Deployment  ~50 PB (10 sites with > 1PB) 5 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd

6. Emphasis  DPM emphasis has moved to better support Xrootd and HTTP/WebDav protocols w.r.t HEP historical data access and transfers ( e.g. SRM, RFIO)  Xrootd Federations are fundamental for some LHC experiments computing models  AAA for CMS and FAX for ATLAS  HTTP/Webdav based federation for LHCb is under evaluation and a prototype available:  http://federation.desy.de/fed/lhcb/ http://federation.desy.de/fed/lhcb/ 6 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd

7. RFIOD DPM HEAD NS) DPM HEAD NS) New Dmlite Architecture 7 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd DPM DPNS DMLITE MYSQL MEM CACHE Legacy SRM Grid FTP Grid FTP XRootD Web DAV Web DAV ADAPTER The goal is to reduce interactions towards legacy components ( bottleneck for performance)

8. Internal Performance Assessment 8 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd  For ~1 year we have been improving our sets of tests headed to METADATA performance and reliability under load  Goals are:  Perf tests: Regular maximum stress exercises on our internal testbeds  quick response: evaluate day by day any change in how the testbeds perform

9. DPM performance tuning hints for HTTP/WebDAV and Xrootd 9 16/10/2014  Nightly performance tests run towards the CERN dev testbeds  Very heavy stress generators  Protocols: HTTP, WebDav, Xrootd  Concurrency: sweeping from 1 to 1000  Tasks:  Peak metadata performance: repeated stat of 10k files by 1-1000 clients  Small reads from each file ( few bytes)  HW: 4 core, 8GB mem, 1GB Net Internal Performance Assessment

10. DPM performance tuning hints for HTTP/WebDAV and Xrootd 10 16/10/2014 Performance Results: 1.8.8 release WebDav peak stat performance

11. DPM performance tuning hints for HTTP/WebDAV and Xrootd 11 16/10/2014 Performance Results: 1.8.8 release HTTP small file GET peak performance

12. DPM performance tuning hints for HTTP/WebDAV and Xrootd 12 16/10/2014 Performance Results: 1.8.8 release Xrootd stat rate

13. The CMS AAA tests 13 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd  CMS has started a performance assessment campaign to understand if sites can sustain the expected load for LHC Run2  Thanks to F. Fanzago for the info and the graphs!  CMS target for tests:  File-opening test: access total rate of 100 Hz at a site  Very aggressive test, as the entire CMS infra would access a single storage element  https://indico.cern.ch/event/324705/contribution/17/ material/slides/1.pdf https://indico.cern.ch/event/324705/contribution/17/ material/slides/1.pdf

14. – Tests run up 100 jobs simultaneously, opening files at rate of 2 Hz each. TEST TARGET IS 100 Hz Opening Plots for some DPM sites DPM performance tuning hints for HTTP/WebDAV and Xrootd 14 Federica Fanzago - INFN Padova

15. Performance Tuning DPM performance tuning hints for HTTP/WebDAV and Xrootd 15 16/10/2014  The internal performance tests + interaction with CMS sites with low performance has highlighted many performance bottlenecks and configuration parameters to be tuned  The new 1.8.9 release is the result of this activity  Caching to avoid DPNS/DPM services interaction  WIKI for Performance Tuning valid for 1.8.8 too  https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/A dmin/TuningHints https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/A dmin/TuningHints

16. Tuning Hints  Some tuning hints are related to Mysql connections and pool size  max connection=1000  NsPoolSize= 256  Others to number of threads for the legacy daemons:  DPM-> NB_FTHREADS=60  DPNS-> NB_THREADS=80  Limits also are important  hard nofile 65000  soft nproc 65000  soft nofile 65000  hard nproc 65000 DPM performance tuning hints for HTTP/WebDAV and Xrootd 16 16/10/2014

17. Tuning Hints: Memcache  Usage of the Memcache plugin with a local instance or external cluster of Memcached server(s).  2GB cache with 256 connections pool size DPM performance tuning hints for HTTP/WebDAV and Xrootd 17 16/10/2014 DPM HEAD NS) DPM HEAD NS) DMLITE MYSQL MEM CACHE Grid FTP Grid FTP XRootD Web DAV Web DAV ADAPTER Mem cached Mem cached Mem cached Mem cached Mem cached Mem cached

18. Performance Tuning: 1.8.8 Before tuning After tuning Strong collaboration between site manager and DPM developers Optimization of DPM configuration parameters Performance not optimal due to headnode’s hardware 18 Federica Fanzago - INFN Padova

19. DPM performance tuning hints for HTTP/WebDAV and Xrootd 19 16/10/2014 Performance Tuning: 1.8.9 WebDav peak performance in 1.8.9 … a 5X increase, no more clogging

20. DPM performance tuning hints for HTTP/WebDAV and Xrootd 20 16/10/2014 Performance Tuning: 1.8.9 HTTP small file GET peak performance in 1.8.9, 20X increase

21. DPM performance tuning hints for HTTP/WebDAV and Xrootd 21 16/10/2014 Performance Tuning: 1.8.9 Xrootd stat rate with DPM 1.8.9, 4X increase

22. Performance Tuning: Puppet  Configuration changes are now possible also with Puppet as it has become the official DPM configuration mechanism ( Yaim is deprecated)  We have been developing a series of modules to configure the various plugins and frontends  Available on PuppetForge and GitHub  https://github.com/cern-it-sdc-id/ https://github.com/cern-it-sdc-id/  https://forge.puppetlabs.com/lcgdm https://forge.puppetlabs.com/lcgdm DPM performance tuning hints for HTTP/WebDAV and Xrootd 22 16/10/2014

23. Summary  Good synergy between Experiments, Site Managers and DPM Developers  DPM Xrootd and HTTP/WebDav Federations are ready for Run2  DPM 1.8.8 might satisfy the modern perf requirements, but needs powerful hardware  DPM 1.8.9 should meet them with more normal machines, e.g. 4cores, 8G mem, good disks, average network. 23 16/10/2014 DPM performance tuning hints for HTTP/WebDAV and Xrootd

24. DPM HTTPS://SVNWEB.CERN.CH/TRAC/LCGDM QUESTIONS? HTTPS://SVNWEB.CERN.CH/TRAC/LCGDM 24 16/10/2014DPM performance tuning hints for HTTP/WebDAV and Xrootd