CCRC'08 experience at PIC storage POV Francisco Martínez Gerard Bernabeu Esther Acción

Outline Preparation Results Issues With proposals Conclusions

Preparation Little preparation was needed Most operations were carried over from Feb run Only the definitions of the tokens and namespace was needed to be modified It was handed to us by experiments the last day, that at least is an improvement from Feb run :) LHCb did not provide us the requeriments :(

Preparation dCache version was 1.8.0-12p6 (Feb run ready) We decided not to upgrade to the latest dCache version, the recommended one for CCRC08 run2, for its late arrival. It turned out to be an excellent choice. Enstore version was 1.0.1 No special versions for CCRC08, enstore seems to be pretty LHC-agnostic.

Results: network pools wan wn

Results: enstore Only IBM robot was used No special load or behavior CCRC08 running was not apparent

Issues Data on pools was poorly balanced Interferences PinManager bug Mover queues dimensioning Enstore movers

Issues Data on pools was poorly balanced Some pools were almost full Some pools were freshly added and had no data on them This lead to multiple crashes on pools due to overload dc046

Issues Proposal for better balancing Deep analysis of the cost assignment policy on dCache formerly no work at all in this area, we have the naive default policy Implementation of Brian Bockleman's scripts for load balancing of files and non-automatic file replication Replica Manager that-works® Dcache-pfm: Physical File Manager

Issues Interferences IFCA was trying to read files that we had marked as without access (STK robot) Those accesses generated unnecessary mover queuing Thanks to Brian Bockleman for sorting this out! Would be nice if Experiments can enforce files marked offline in some way Cannot be done at dCache level

Issues PinManager bug Running dCache version 1.8.0-12p6 has a bug regarding the pinManager Some files become unaccesible due to lcg-gt hanging A cron restarting the pinManager solved this Current dCache release 1.8.0-15p5 has already solved this

Issues Mover queues dimensioning Has to be done depending on the type of access of jobs Low throughput-high duration profile, with locking of files These need a high number of movers. Maybe a deadlock detection mechanism would help at application level! Throughput intensive profile These suggest low number of movers

Issues Proposal for mover queues Jon Backen, head dCache admin at FNAL is coming to visit PIC They have implemented drastical solutions to this problem We will need to see if we can use those configuration queues with 1800 movers

Issues Enstore movers Had an issue with a memory leak that made the movers go offline Was solved before run2 started Enstore accounting We have the info, but it is difficult to parse or analyze. It is advisable to implement some way to visualize that data

Conclusions Storage component at PIC seems to be roughly ready to start data-taking, but Still pending issues with dCache Load balancing is specially critical Kudos to Enstore: no issues We have good feedback with experiments thanks to the Liaisons, but Experiments as a whole are many times not too communicative