1. DPM Basics and its status and plans Wahid Bhimji University of Edinburgh GridPP Storage Workshop – Apr 2010 Apr-101Wahid Bhimji – DPM

2. This talk: Not covering all the basics of DPM – many already know those. See gridpp wiki: http://www.gridpp.ac.uk/wiki/Disk_Pool_Manager or Sams talks in July: http://hepwww.rl.ac.uk/sysman/June2009/agenda.html Will cover: -An introduction of terms. -List some commonly used tools for reference -Details of recent developments and plans for discussion Apr-10Wahid Bhimji – DPM2

3. A standard DPM setup Apr-10Wahid Bhimji – DPM3 DPM Head Node DPM Head Node Pool Node Pool Node Head Node (dpm-mysql) - SRMV2 Interface - DPM server: Request tracking (MySQL) - DPM Nameserver (DPNS) (MySQL) User Worker Node Worker Node DISK Pool node (dpm-disk) -Directly mounted disk - RFIO– local transfers -GridFTP– WAN transfers … Pool Node Pool Node DISK Pool Node Pool Node DISK gridftp rfio …

4. Available tools Some built-in and some in the gridpp toolkit: http://www.gridpp.ac.uk/wiki/DPM-admin-tools Latest version 2.6.3 Let Sam or I know if you need another tool For help ask the GRIDPP-STORAGE jiscmail list or dpm-users-forum@cern for generic dpm questions Apr-10Wahid Bhimji – DPM4

5. Common activities Install / upgrade: See GridPP Wiki Look at space usage, list files Move data around Allocate or modify space: –pools given to VOs (or collection of VOs) –Spacetokens to groups within VOs Apr-10Wahid Bhimji – DPM5

6. Finding out whats on your storage dpns-ls : List files in namespace dpm-disk-to-dpns – translate filepath to dpns dpm-list-disk - list all files on a filesystem Overall picture dpm-listspace: Includes usage in spacetokens dpm-qryconf: Includes usage in filesystem dpns-du: For space usage within a namespace path. Apr-10Wahid Bhimji – DPM6

7. Moving files dpm-delreplica or rfrm : remove from disk and namespace (but not LFC / remote catalogues) dpm-replicate or rfcp : copy single file dpm-drain: move a whole filesystem -Need to wrap in an until loop to persist if it crashes. dpm-sql-spacetoken-replicate-hotfiles – (new(ish) in toolkit) – replicate all files within a spacetoken Apr-10Wahid Bhimji – DPM7

8. Allocating Space ALL ATLAS VO space should be in SPACETOKENS Create: dpm-reservespace Update dpm-updatespace - do not use dpm- reserve again Delete: dpm-releasespace Moving a filesystem between pools: dpm-rmfs --server pool2 --fs /gridstorage018 dpm-addfs --poolname cms --server pool2--fs /gridstorage018 Apr-10Wahid Bhimji – DPM8

9. DPM Status Apr-10Wahid Bhimji – DPM9

10. Support Last year central DPM support from CERN was at the level of 0.5 FTE. Now improved to ~2 FTE: – Jean-Philippe Baud has returned – Additional developer : Ricardo Rocha Apr-10Wahid Bhimji – DPM10

11. Versions Latest version – still 1.7.2(-5) 1.7.3 (in certification since Oct.) Provides: Checksum treatment Fix for unstable SRM server (currently patched binary) Future Program of work https://twiki.cern.ch/twiki/bin/view/LCG/DMProgra mOfWork Apr-10Wahid Bhimji – DPM11

12. Future Plans - Highlights DPM 1.7.4 (in certif.) RFIO tuning of read ahead buffers via env variable File replication in parallel for improved dpm-drain. DPM 1.7.5 RFIO – improved spacetoken/ checksum handling Improved drain using rfio DPM 1.7.6 Internal monitoring of disk servers - NAGIOS DPM 1.8.0 (Schema changes) Usage information as a pre-requisite for quotas Banning of users and groups DPM 1.8.1 Priority by role in request queue Limit IO slots by server by type of access Apr-10Wahid Bhimji – DPM12

13. Long(er) term plans - Confirmed Quotas. Accounting: Space uses, files: read, written, replicated - bytes Replace HTTPG by HTTPS for SRM servers NFS 4.1 interface VO admin role Glue2 support Not Scheduled Add GUIDs to file entries to be able to use DPM as local LFC Automatic replication of hot files. attributes, not I/O. Integration with Argus (banning of users and groups). Better file system selection algorithm.. Support for distributed file systems like Lustre/Hadoop Apr-10Wahid Bhimji – DPM13

14. Unconfirmed Support for striping Support for BitTorrent Multi-vo support for xrootd WEB admin interface Gateway hosts to allow sites having very few IP addresses visible externally to use DPM Provision of native python interfaces Support for python 2.6 Apr-10Wahid Bhimji – DPM14

15. Conclusions / Discussion DPM has many tools – use not always obvious Significant support in UK – use us Now good support at CERN too And a plan of action – We can influence this – We can also help develop features of interest E.g asked to help with Nagious plugins – What are the priorities? Apr-10Wahid Bhimji – DPM15