DPM Basics and its status and plans Wahid Bhimji University of Edinburgh GridPP Storage Workshop – Apr 2010 Apr-101Wahid Bhimji – DPM

This talk: Not covering all the basics of DPM – many already know those. See gridpp wiki: or Sams talks in July: Will cover: -An introduction of terms. -List some commonly used tools for reference -Details of recent developments and plans for discussion Apr-10Wahid Bhimji – DPM2

A standard DPM setup Apr-10Wahid Bhimji – DPM3 DPM Head Node DPM Head Node Pool Node Pool Node Head Node (dpm-mysql) - SRMV2 Interface - DPM server: Request tracking (MySQL) - DPM Nameserver (DPNS) (MySQL) User Worker Node Worker Node DISK Pool node (dpm-disk) -Directly mounted disk - RFIO– local transfers -GridFTP– WAN transfers … Pool Node Pool Node DISK Pool Node Pool Node DISK gridftp rfio …

Available tools Some built-in and some in the gridpp toolkit: Latest version Let Sam or I know if you need another tool For help ask the GRIDPP-STORAGE jiscmail list or for generic dpm questions Apr-10Wahid Bhimji – DPM4

Common activities Install / upgrade: See GridPP Wiki Look at space usage, list files Move data around Allocate or modify space: –pools given to VOs (or collection of VOs) –Spacetokens to groups within VOs Apr-10Wahid Bhimji – DPM5

Finding out whats on your storage dpns-ls : List files in namespace dpm-disk-to-dpns – translate filepath to dpns dpm-list-disk - list all files on a filesystem Overall picture dpm-listspace: Includes usage in spacetokens dpm-qryconf: Includes usage in filesystem dpns-du: For space usage within a namespace path. Apr-10Wahid Bhimji – DPM6

Moving files dpm-delreplica or rfrm : remove from disk and namespace (but not LFC / remote catalogues) dpm-replicate or rfcp : copy single file dpm-drain: move a whole filesystem -Need to wrap in an until loop to persist if it crashes. dpm-sql-spacetoken-replicate-hotfiles – (new(ish) in toolkit) – replicate all files within a spacetoken Apr-10Wahid Bhimji – DPM7

Allocating Space ALL ATLAS VO space should be in SPACETOKENS Create: dpm-reservespace Update dpm-updatespace - do not use dpm- reserve again Delete: dpm-releasespace Moving a filesystem between pools: dpm-rmfs --server pool2 --fs /gridstorage018 dpm-addfs --poolname cms --server pool2--fs /gridstorage018 Apr-10Wahid Bhimji – DPM8

DPM Status Apr-10Wahid Bhimji – DPM9

Support Last year central DPM support from CERN was at the level of 0.5 FTE. Now improved to ~2 FTE: – Jean-Philippe Baud has returned – Additional developer : Ricardo Rocha Apr-10Wahid Bhimji – DPM10

Versions Latest version – still 1.7.2(-5) (in certification since Oct.) Provides: Checksum treatment Fix for unstable SRM server (currently patched binary) Future Program of work mOfWork Apr-10Wahid Bhimji – DPM11

Future Plans - Highlights DPM (in certif.) RFIO tuning of read ahead buffers via env variable File replication in parallel for improved dpm-drain. DPM RFIO – improved spacetoken/ checksum handling Improved drain using rfio DPM Internal monitoring of disk servers - NAGIOS DPM (Schema changes) Usage information as a pre-requisite for quotas Banning of users and groups DPM Priority by role in request queue Limit IO slots by server by type of access Apr-10Wahid Bhimji – DPM12

Long(er) term plans - Confirmed Quotas. Accounting: Space uses, files: read, written, replicated - bytes Replace HTTPG by HTTPS for SRM servers NFS 4.1 interface VO admin role Glue2 support Not Scheduled Add GUIDs to file entries to be able to use DPM as local LFC Automatic replication of hot files. attributes, not I/O. Integration with Argus (banning of users and groups). Better file system selection algorithm.. Support for distributed file systems like Lustre/Hadoop Apr-10Wahid Bhimji – DPM13

Unconfirmed Support for striping Support for BitTorrent Multi-vo support for xrootd WEB admin interface Gateway hosts to allow sites having very few IP addresses visible externally to use DPM Provision of native python interfaces Support for python 2.6 Apr-10Wahid Bhimji – DPM14

Conclusions / Discussion DPM has many tools – use not always obvious Significant support in UK – use us Now good support at CERN too And a plan of action – We can influence this – We can also help develop features of interest E.g asked to help with Nagious plugins – What are the priorities? Apr-10Wahid Bhimji – DPM15