1. DPM status and directions
DPM workshop 2016
Speaker: Fabrizio Furano, on behalf of the DPM team
22/11/2016
DPM workshop 2016

2. Introduction A very important year has passed
Lots of development and cleanup
Two releases, (maintenance) and (feature)
The coming year will see a transition that surely includes:
Less and less SRM activity
Space reporting
More storage, maybe less tiny sites
We discuss the current DPM status
We present the roadmap for the important transitions ahead
22/11/2016
DPM workshop 2016

3. DPM macro-features A distributed software for big data storage
Clusterization of pools of disks and storage in general, local or networked
DB-based metadata persistency, pretty good scalability demonstrated by the range of the deployments
Multiple data protocols for the same cluster, coherent content
HTTP, WebDAV, Xrootd, GridFTP1/2
Historical support for the Grid standards
SRM, X509/VOMS
Focus on giving simplified setup tools
YAIM-style standalone puppet template
Continuous support
DPM exists thanks to its considerable and active user base
22/11/2016
DPM workshop 2016

4. DPM status Infosys numbers 73PBs in total, provided to Grid computing
~160 instances. We lost a tail of tiny sites
The overall storage capacity continued to grow
Several sites larger than 2 PB, 20 sites larger than 1PB. The largest so far is 6.5PB
Its admin is among our special guests today
Our focus continues to be on
Consolidation, keeping sysadmin cost at the lowest
Performance, scalability
High quality HTTP, WebDAV, Xrootd, GridFTP support
Support. In touch with sysadmins as much as we can
We like how dpm-users-forum is doing
22/11/2016
DPM workshop 2016

5. Starting from
Spring 2016 has seen the release of DPM , maintenance release
Several small things, including the porting of the ‘dbpool’ mods to dpmd
Those that increased 3X the performance of dpns/lfc some years ago
No more hiccups on DPMd  less hiccups on SRM access
We think that the glorious dpmd/dpnsdaemon couple reached its maximum with
Both for performance and features
They work pretty well indeed for what they can give
22/11/2016
DPM workshop 2016

6. DPM status: 1.9.0 Very big effort, a landmark
About 350 tickets, 700 commits in 10 months
Some tickets trivial, others required months
DPM (DMLite bundle 0.8.x) went EPEL-test in October
Last step of a transition from the very old lcg-dm codebase
No known core performance issues
NB the old LCGDM stack remained untouched, that has some very well known ones
Epel-prod status after the workshop
Metapackages automatically produced and included
22/11/2016
DPM workshop 2016

7. 1.9.0 highlights
New Disk Operations Management Engine (DOME) makes the legacy stack totally optional
Allows us to make a roadmap for obsolescence of SRM/rfio/dpmd/dpns/Csec/libshift
Space reporting on top directories, in the browser and according to RFC 4331 (WebDAV)
Many improvements to dmlite-shell, becomes more complete
DMLite Stat functions now do not use exceptions as return values  MUCH less ugly/scary exception messages in the log
Core now can calculate and properly queue checksum requests
File pulls for volatile pools, including queueing
Minipuppet standalone setup becomes even simpler, system-oriented rather than component-oriented
22/11/2016
DPM workshop 2016

8. Release status Stable situation Support EL6, Centos7, drop SL5
Automatically coherent metapackages, no more EMI-updates
Detailed release status tomorrow given by Andrea
22/11/2016
DPM workshop 2016

9. Where we are

10. The will (From DPM 2015) OPTIONAL IT-SDC DMLite section dmlite core
WebDAV
Xrootd
mysql
profiler
adapter
gsiftp
mem
cache
OPTIONAL
Legacy daemons
rfio
dpns
srm
Legacy clients
CSec
dpm (daemon)
Got 3X faster
in 1.8.7/8
Used less and less
Main
clog
until 1.8.9/1.8.11
VFS
Options/
Others
Oracle S3
HDFS
Dynamic
Feds
(Ugr)
Very difficult to evolve
DEV is frozen
IT-SDC
DPM

11. Now: DPM 1.9.0 Released to EPEL-testing in late summer
Bring the new mgmt engine: DOME
Massive cleanup/development action
Substitutes dpmd, dpns, rfio, libshift, Csec in one go
General harmonization, checksums, space reporting, quotas, …
Big improvements to dmlite-shell, can be used as a DPM shell
Coherent metapackages (finally!)
Puppet-based interactive minisetup (or miniPuppet)
22/11/2016
DPM workshop 2016

12. DPM in one sight (Q4/2016) DMLite section dmlite core WebDAV Xrootd
mysql
profiler
adapter
gsiftp
mem
cache
Legacy daemons
rfio
dpns
srm
Legacy clients
CSec
dpm (daemon)
Communication to DPMd
Now can be dropped
DOME
adapter
VFS
Options/
Others S3
HDFS
Dynamic
Feds
(Ugr)
DOME

13. The new DPM core: DOME
Last December (2015) we were exploring the possibility of a new core for DPM
The codename was Disk Operations Management Engine ( DOME )
We implemented it in 2016, following the direction that was described 1 year ago
(Tech talks about DOME later today)
Main directions:
Daemon based on FastCGI
Open JSON-based command set
Command set is a deep review/cleanup of the historical one (~20 years old?)
Allow SRM-less operation
Coherent directory-based space reporting and directory quotas
Allow making the legacy LCGDM stack optional
22/11/2016
DPM workshop 2016

14. Directory-based space accounting
The ‘size’ field of a dir is the sum of all the items it contains (subdirs included)
Embedded in DOME
We also implemented it in DPMd, as an exception to the code freeze
So the directory counters will always be aligned, even using SRM put/get 
A write/delete now updates both dir counters and spacetoken counters (if a spacetoken was specified)
The big challenge was to evolve from the spacetokens ‘orthogonality’, and create a simpler semantics for writes that is also correct. This is a very hard problem per se.
We also wanted to make the transition as easy as possible for sites
22/11/2016
DPM workshop 2016

15. Spacetokens -> Quotatokens
A good old spacetoken now can be associated to a directory path
Writes into that path will put files into the pool pointed to by the spacetoken
Writes into that path will use the spacetoken t_space field as a quota
Hence, a quotatoken is the evolution of a spacetoken, which can now be associated to a directory path
This kept the backwards compatibility, no need to re-enter the same config information, the old spacetokens remain and get promoted
Old spacetokens just need to be associated to directories, e.g.
ATLASSCRATCH(pool1, 100TB) /dpm/<site>/home/atlas/scratch
And from this point on all the writes to /atlas/scratch will populate pool1, pointed to by ATLASSCRATCH with a quota of 100TB
22/11/2016
DPM workshop 2016

16. Quotatokens
A quotatoken can be seen as a TAG that can be attached to a directory tree
Gives a pool where subsequent writes will write
Gives a limit to how much can be written there (quota)
The quota is enforced versus the size of the directory subtree that is linked to the same quotatoken (unlike SRM and its spacetks counters)
All the management can be done easily with dmlite-shell (see the talk later today)
22/11/2016
DPM workshop 2016

17. DOME speaks JSON through HTTPS
Totally documented, open internal communication protocol
Internal because remote clients do not see it, nor they are allowed to use it (unlike dpns for example)
Remote clients see only the frontends: xrootd, httpd, gridftp
Its clients are a DMLite plugin and DMLite-shell
DOME was designed with configuration simplicity in mind. No shift.conf, no NSCONFIG
22/11/2016
DPM workshop 2016

18. DOME security and IPC DOME resides in:
Head node (substitutes DPMd, dpns, shift, rfio, Csec)
ALL disk nodes (substitutes rfio, Csec)
DOME only accepts authenticated clients (HTTPS, with or without VOMS extensions)
By default, DOME accepts requests only from the hosts composing the cluster
This is automatic and zero-config, no more libshift
The hostname of disk servers must match the one in its host certificate
DOME keeps track of the status of the disk servers. Disk servers that are down are now correctly detected and avoided
When they come back, they will be automatically detected too
22/11/2016
DPM workshop 2016

19. Incremental transition
All these features are already in DPM and tested by us and some helping sites
Special thanks to A Sartirana for helping us sort a nasty bug with users that have multiple DNs
Switching them ON completely requires updating head node and all servers
We decided to allow sysadmins to enable them incrementally following their preferred pace
22/11/2016
DPM workshop 2016

20. Transition to SRM-less
Making the transition possible means giving the features through other means (possibly agreed)
Stopping to use it is not sufficient
Making this properly meant a big dev effort
Design the architectural changes
Define a coherent feature set (quotas, pool semantics, command set, etc)
Minimize the change for sysadmins
Prepare a roadmap for a smooth transition
Translate this chatter into reality
We don’t see other efforts like that in the future
The future DPM roadmap likely spans two years
Dates are there purely as indicative guesses
Pace will be decided by sysadmins and experiments
22/11/2016
DPM workshop 2016

21. Adapting to SRM-less operation
1.9.0 in EPEL
Site starts updates
ADM Configures
quotatokens
AND primes
the space calcs
DmLite::DomeAdapter
substitutes Adapter
(dmlite+gridftp
config change)
Experiments
gradually
stop using SRM
All heads AND disks
are updated
Load on SRM
goes to zero
Sysadmins
Experiments
DPM team
…continuous support…
2018?
Maybe 2019?
Q4/2016
Somewhere 2017?
Good moment
for a minor release
DPM/dpns/rfio still used
(through DMLite::Adapter and SRM)
DOME used only for dmlite-shell
New powerful dmlite-shell is available
Can report used space per path
Free space reporting is like before (Only SRM can report on free spc)
All the DOME features are enabled
WebDAV reports used AND free spc
per path
Quotas are properly enforced
Can create the ATLAS space report
SRM/DPM still report on spacetokens
SRM/DPM/dpns/rfio are now optional
DPM/dpns/rfio/SRM
can be uninstalled

22. 1.9.0 Roadmap step 1 (1/2) Trigger: Site upgrades head node from EPEL
Disk nodes can be done incrementally
Dome is running, active only for the shell commands
Main data/metadata flow is like
Dmlite still uses Adapter for the main load, hence DPMd/dpns
Administrator can assign each spacetoken to a directory (just to be prepared to the subsequent steps)
Administrator gets used to the shell and the new commands
DPM builds now provide metapkgs in EPEL, no more bouncing with EMI-updates (see talk about release status tomorrow)
22/11/2016
DPM workshop 2016

23. 1.9.0 Roadmap step 1 (2/2)
Used space reporting on directories now works like ‘du’, it can be seen with WebDAV or a browser or the dmlite-shell info command
Directory spaces should be aligned ONCE using dmlite-mysql-dirspaces
At step 1, only SRM spacetokens can report on free space, like before
Transfers via SRM or dmlite frontends give consistent results wrt used space.
An srm transfer updates both ST and dir counters
A gridftp-only/xrootd/http transfer updates both ST and dir counters
ST accounting and dir should stay ideally the same if there are no cross-spacetoken writes (e.g. writing to /atlasscratch using a prod spacetoken)
Minipuppet setup supports enabling DomeAdapter but this will be OFF by default
22/11/2016
DPM workshop 2016

24. 1.9.0 Roadmap step 2 (1/2) Pre-requirements: Action:
The sysadmin has already installed in all the disk servers, incrementally, with the pace he decides
DOME is running in all the disk servers and the head node
The sysadmin has reviewed his spacetokens and applied them to proper directory subtrees
Action:
Flip a switch in the config (puppet manifest) of the head node ( of course can be reversed  )
DMLite::DomeAdapter then substitutes DMLite::Adapter
22/11/2016
DPM workshop 2016

25. 1.9.0 Roadmap step 2 (2/2)
After the config switch the legacy stack is used ONLY by SRM calls
DMLite stack has no interaction anymore with old stack
Quotatokens now are fully enforced for non-SRM activity
File pulls can now be used to experiment with volatile filesystems (as GET/open requests go through DOME now)
Checksum queueing is effective
Space reporting numbers can be seen using the shell
Can be exported to whatever format by writing simple shell/python scripts
22/11/2016
DPM workshop 2016

26. Space reporting
So far, the glorious dpm-listspaces has been the tool for getting the numbers
dpm-listspaces contains very complex code just to calculate those numbers and fix on the fly the sometimes strange information coming from spacetokens and pools
The DOME values are correct and coherent by construction, no need for complex code
Space reporting with DPM1.9 “Dome flavour” is implemented in the shell (and it’s easy code)
Depending on the plans for a new WLCG infosys, we may need a new script one day
Depending on the plans for the ATLAS JSON space reporting we may need either a script or a trivial CGI application
Once the details are clear, it should be pretty straightforward
22/11/2016
DPM workshop 2016

27. Space reporting
Space reporting is a very difficult topic, even to describe
More details later today, see talk by O.Keeble
22/11/2016
DPM workshop 2016

28. A word on the ARGUS poller
DPM has an optional component that synchronizes banned users with a remote ARGUS service
Technically dpns-arguspoll will continue to work for all the described timeline
Will stop working when the old stack is uninstalled
This will start happening likely in 1-2 years. Will anyone still need the ARGUS poller ? Do we need a replacement ?
The replacement could be written easily in pure Python, perfect conditions for a contribution
22/11/2016
DPM workshop 2016

29. File pulls DOME provides the hooks to implement file pulls
Very similar to the analogous historical feature of xrootd, with DPM features
When a pool is marked as Volatile
If a non existing file is requested, a script will be invoked to pull it into a disk server
DOME takes care of queuing the requests and provides the feature
Can be used to set up simple automatic ‘data caches’ for full files
Needs a complete DPM “DOME flavour” setup
22/11/2016
DPM workshop 2016

30. Metapackages They used to be in EMI-updates
Created separately ( a parallel build plan triggered manually )
Fetched separately
Updated manually ( Cristina Aiftimiei, on our trigger )
Now they come with the regular dmlite builds
More precise deps, automatically produced right
Can contain utility scripts and other things
Now the DPM metapkgs are officially DPM
dmlite-dpmhead
dmlite-dpmdisk
dmlite-dpmhead-dome (for the DOME flavour)
dmlite-dpmdisk-dome (for the DOME flavour)
Temporary cosmetic limitation: they carry the version number of DMLite (0.8.x)
Cosmetic Fix: We will soon promote DMLite to 1.9.x
22/11/2016
DPM workshop 2016

31. DPM shell With 1.9.0 the shell becomes much richer
Goal: All the DPM administration can be done through it
Autocompletion, name space access, inline help, etc…
More details on Andrea’s talk, covering also the drain operations
22/11/2016
DPM workshop 2016

32. Does my DPM work ?
Georgios contributed a nice script that tests the main features of a DPM cluster
Tests the main core features, plus GridFTP, HTTP/DAV and Xrootd
Has become an important tool to do a first, extensive run of functional tests
More details on Georgios’ talk
22/11/2016
DPM workshop 2016

33. Multiple checksums
Inside DMLite each file can have multiple checksums of different types
DOME fixes some more historical shortcomings:
Each frontend may now treat any kind of checksum
DOME queues the requests, in order not to overload servers. The parameters are customizable
A cross-checksum campaign can take place, or at least it will be smoother
22/11/2016
DPM workshop 2016

34. Puppet setup
The setup of a DPM can be done with puppet in standalone mode
YAIM-ish feel. Run it manually in a machine to configure that machine
No need to puppetize the site
My impression is that it’s pretty good, doing several things that yaim could not do reliably or not at all in its best days
22/11/2016
DPM workshop 2016

35. Other kinds of setup There may be other kinds of setup for a DPM
Saw some comments around ansible, we like the idea of one more possibility fostered by the community
Could be that Docker is an even more powerful solution for distribution. Quite tempting, however questionable for existing servers.
The puppet templates describe quite clearly the setup steps and the subtleties of the config files, they can be ported in principle to anything else
We welcome contributions. If anyone likes the setup manager XYZ and wants to contribute XYZ configurations for DPM, we will include them in the releases or the website with pleasure
We have no more plans on setup beside maintenance. You may.
22/11/2016
DPM workshop 2016

36. DPM setup More information on the setup tomorrow morning
Followed by the DPM ‘clinic’, open Q&A
22/11/2016
DPM workshop 2016

37. Other scenarios
With DOME it becomes much easier to distribute DPM disk servers across sites
All the internal communication happens through the same channel, HTTP(s)-based JSON
Moving a disk server to a neighboring site becomes technically easy
Pro: full support for gridftp, xrootd, http
More details in the talk of Alessandra Doria
In theory DOME flavour is very close to allow redundant DOMEhead setups
So far no idea whether GridFTP supports this
DOME Would likely need some minor fixes to do it all, yet…
Beware: adding hardware does not always mean more reliability or more performance. It may just mean more stuff, which is more cost
So far we had no proof that a redundant setup would enhance reliability in a practical setup. The interactions (DNS, protocol frontends, experiment frameworks, …) and the constraints are too complicated for an easy assessment
However it cannot be excluded, the assessment may be an interesting proposal for a contribution from some brave sysadmin that we would support
22/11/2016
DPM workshop 2016

38. A few words on the docs
In the medium term we will migrate all the documentation pages (now in TRAC) to a new Twiki web
This will be the right occasion to do some cleanup of old stuff and restructure a bit
Please get in touch if you would like to fit something into the new site
22/11/2016
DPM workshop 2016

39. Direction We don’t foresee bold steps like the ones we just made
Priority is following the happening of the discussed timeline and helping sites to give good service through it
Starting from now, we blend into maintenance, where some code consolidation may be possible
Consolidating the plugins ( = less of them to maintain and configure ) would simplify more the setup and make the system more efficient
Consolidation of the version number lcgdm=dmlite !
The lcg-dm legacy stack is still frozen, we won’t touch it
We are not sure we want to port it to openssl and Rawhide
The DMLite stack surely will be ported
Emphasis will be on support, helping sites to consolidate with the new features
Non-SRM metadata performance seems adequate so far (has not changed since last year). Plugin consolidation may enhance it by a factor 2-4, to be evaluated if we think it’s needed.
22/11/2016
DPM workshop 2016

40. Last word on Tuning Hints
Poorly configured servers seem quite common
Likely coming from older YAIM setups or incomplete manual config
A poorly configured server (head OR disk) will not only be slow, it may also behave weirdly (resource constraints may cause internal failures)
The best vaccine we know is to apply the Tuning hints… please do it in its entirety:
22/11/2016
DPM workshop 2016

41. Conclusion Natively Multi-VO Multi-protocol scalable storage system
Goals: flexibility, performance, scalability, low maintenance, ease of setup
Philosophy: coordinate native frontends implementing the protocols
1.9.0 is the step that was foreseen long ago
1.9.0 is a round of rationalization, addresses historical topics:
Space reporting
SRM-less operation
Flexible Checksums
Possibility to work with (maybe as) file caches
Use REST tech, exportable to other systems, easy to understand by others
Performance goals: meet or exceed the requirements
Next iteration 2017 will likely be about refactoring of small things, a maintenance release to followup with the roadmap
Important: please say clearly if you want a DPM 2017 workshop.
22/11/2016
DPM workshop 2016