EMI is partially funded by the European Commission under Grant Agreement RI Future Proof Storage with DPM Oliver Keeble (on behalf of the CERN IT-GT-DMS section)

EMI INFSO-RI Disk storage for the grid 36PB – 10 sites with > 1PB Over 200 sites in 50 regions Over 300 VOs have access to DPMs DPM today Future Proof Storage with DPM, EGI TF Prague2

EMI INFSO-RI Current production version is – lcgdm lcgdm-183 – In EPEL, EMI1 and EMI2 – No gLite – see tutorial on Wed on how to upgrade What’s new – EPEL compliance – HTTP/DAV frontend (old dpm httpd is gone) – NFSv4.1 frontend (read only) – Thread safe clients – Synchronous get requests – … DPM today Future Proof Storage with DPM, EGI TF Prague3

EMI INFSO-RI Next core release: – First DMLITE release (see later…) DAV frontend will start using the new libraries Other components not using dmlite for now – Improved replication mechanism – 32bit client support – Lots of other small fixes – Feature complete… finishing certification Change in the release process – Independent component releases – Faster, lighter releases Future Proof Storage with DPM, EGI TF Prague4

EMI INFSO-RI DMLITE is the result of a significant refactoring effort to make DPM modular Better separation between frontends and backend – Cleaner, more open, much improved performance Improved integration with standard building blocks – Hadoop, Memcache, S3, Lustre, … Refactoring & DMLITE Future Proof Storage with DPM, EGI TF Prague5

EMI INFSO-RI Standard protocols, standard clients – HTTP/DAV + extras, WAN transfers, 3 rd party copy… – NFS 4.1/pNFS (r/o) – Xrootd (rewritten) Forthcoming – GridFTP Legacy interfaces remain untouched – No large scale revalidation required Improved Frontends Future Proof Storage with DPM, EGI TF Prague6

EMI INFSO-RI Standard protocols, standard clients – HTTP/DAV + extras, WAN transfers, 3 rd party copy… – NFS 4.1/pNFS (r/o) – Xrootd (rewritten) Forthcoming – GridFTP – Ubiquitous access to grid storage Legacy interfaces remain untouched – No large scale revalidation required Improved Frontends Future Proof Storage with DPM, EGI TF Prague7

EMI INFSO-RI This is where DMLITE shines – Plugin based, open for constant evolution Improved nameserver performance – Connection pooling, improved SQL, memcache layer, … Support for multiple pool types – Legacy DPM, Hadoop/HDFS, S3, … – Sharing a single namespace if desired – Possibility for opportunistic pools Federation – See the following presentation And this is the beginning, much more coming – Python bindings, Lustre, VFS, … Improved Backends Future Proof Storage with DPM, EGI TF Prague8

EMI INFSO-RI This is where DMLITE shines – Plugin based, open for constant evolution Improved nameserver performance – Connection pooling, improved SQL, memcache layer, … Support for multiple pool types – Legacy DPM, Hadoop/HDFS, S3, … – Sharing a single namespace if desired – Possibility for opportunistic pools Federation – See the following presentation And this is the beginning, much more coming – Python bindings, Lustre, VFS, … Improved Backends Future Proof Storage with DPM, EGI TF Prague9

EMI INFSO-RI I/O performance Future Proof Storage with DPM, EGI TF Prague10 LAN / Chunk Size: 10k-20k / File Size: 2G ProtocolN. ReadsRead SizeRead Time HTTP50022,773, HTTP100046,027, XROOT50022,773, XROOT100046,027, RFIO50022,773, RFIO100046,027, PRELIMINARY

EMI INFSO-RI Performance, performance, performance Future Proof Storage with DPM, EGI TF Prague11

EMI INFSO-RI Puppet for configuration – Popular among large data centers – Lots of modules for popular tools (which we now rely on – apache, memcache, nagios, …) Nagios for monitoring – We reuse as much as possible And there’s a lot already available – Added specific plugins for detailed status and performance monitoring Easy administration Future Proof Storage with DPM, EGI TF Prague12

EMI INFSO-RI Puppet for configuration – Popular among large data centers – Lots of modules for popular tools (which we now rely on – apache, memcache, nagios, …) Nagios for monitoring – We reuse as much as possible And there’s a lot already available – Added specific plugins for detailed status and performance monitoring Easy administration Future Proof Storage with DPM, EGI TF Prague13

EMI INFSO-RI Standards – Leverage existing components & clients – Mature ecosystem – Less maintenance work Community – Guidance by the stakeholders – Independence from funding cycles – Now in talks for the creation of a “DPM Collaboration” to drive the project post-EMI Why “future proof”? Future Proof Storage with DPM, EGI TF Prague14

EMI INFSO-RI DPM is the most numerous SE on the WLCG infrastructure The project has never been in better shape We are soliciting statements of support for a collaboration In discussions with – GridPP (UK) – WLCG France – Taipei (WLCG Tier 1 using DPM) The DPM Collaboration Future Proof Storage with DPM, EGI TF Prague15

EMI INFSO-RI DPM around the world Future Proof Storage with DPM, EGI TF Prague16

EMI INFSO-RI DPM has received a lot of investment thanks to EMI – These developments are now being released This investment has gone into – Making it modular – Supporting standards (HTTP, NFSv4.1,…) – Profiting from existing technology – Performance – Manageability Summary Future Proof Storage with DPM, EGI TF Prague17

EMI INFSO-RI Issue with LFC API / Py26 (#84716) – Tracked down to issue with EPEL5 Swig version – Currently considering the best solution Cleaning up invalid LFC replicas (#83335) – Solved… bulk requests using API/CLI tools DAV EMI2 install failure (#85141) – Related to dependencies on gridsite, solved with the EPEL update EMI2 LFC dies regularly (#85161) – Happens to any of lfc/dpm/dpns daemons – Documented as a known issue with bit support (#81508) – Available in EMI2 and EPEL repositories Recent Issues Future Proof Storage with DPM, EGI TF Prague18