LHCb input to DM and SM TEGs

Remarks to DM and SM TEGS Introduction m We have already provided some input during our dedicated session of the TEG m Here are a list of questions we want to address for each session o Not exhaustive, limited to one slide per session m It would have been useful to hear what are the proposals for evolution of SM and DM as well o These should be proposals, not diktat o Extensive discussions are needed with users and sites before embarking on full scale “prototypes” that no longer are prototypes m Whatever the future is, WLCG must ensure the long term support (no external “firm”) o Do not underestimate the amount of work needed for users to adapt o Therefore plan well ahead, gather functionality requirements… F2F Data and Storage Management TEG, Amsterdam2

Remarks to DM and SM TEGS F2F Data and Storage Management TEG, Amsterdam3 Data Placement and Federation m Does it imply replica catalogs are no longer needed? o How is brokering of jobs done? o Random or “where most files are” m When is data transfer performed? o By the WMS: implies a priori brokering, incompatible with pilot jobs o By the job: inefficiency of slots o Is it a cache (life time?) or just a download facility? m What is the advantage compared to placement using popularity? o Limited number of “master replicas” (e.g. 2) o Add replicas when popularity increases o Remove replicas when popularity decreases o … but still with a catalog and job brokering m What is the minimal number of sites for which it becomes worth it?

Remarks to DM and SM TEGS WAN protocols and FTS m We need third party transfer! o http 3 rd party transfer? OK if commercial, why support it ourselves? m We need a transfer “batch” system! o Asynchronous bulk files transfer m Whatever reliable and efficient underlying protocol is used is just fine… m There is a need to define the service class where the file has to be put (or one service class per SE) m What about the dedicated network (OPN)? o Requires a service for using it? o Not all bells and whistles may be necessary m The real point is for a user (experiment): o Transfer this list of LFNs to this SE (SE = storage class at site) P The actual physical source is irrelevant P The TS should discover whether there is an online replica, if not it should bring it online before making the transfer P Ideally it (or the SE) should register the new replica (keep consistency) m All this was already said in… Mumbai (February 2006)! m FTS 3 was looking promising… why is it dead? F2F Data and Storage Management TEG, Amsterdam4

Remarks to DM and SM TEGS Management of Catalogues and Namespaces m See Data placement… m Do we need a replica catalog? o LHCb answer is YES: we want to be able to do brokering of jobs o May only contain information on the SEs (+ file metadata + usability flags) m Do we need a catalog with URLs? o Not necessarily: the URL can be formed from the SE information and the LFN (trivial catalog), as SE information is quite static. m Do we need a single URL (used for transfers and for protocol access)? o No problem as long as the access is transparent and fast m See SRM slide for more comments… o Namespace vs storage class? F2F Data and Storage Management TEG, Amsterdam5

Remarks to DM and SM TEGS Security and Access Control m We MUST protect our data from deletion o LHCb doesn’t care about protecting from access so much m The current situation is INACCEPTABLE o Anyone with little knowledge (available on the web) can delete all files in Castor! o VOMS (or equivalent) identification and authorisation is a MUST! What about ARGUS? P Identity and role P Currently in Castor we have only 2 (uid, gid)! P Protection done by the LFC, but all backdoors are open o Backdoors should be closed (nsrm, stager_xxx active commands…) o Explicit “delete” permission would be desirable m Change of DN should be straightforward (not trivial, but OK in LFC, DPM) o Action from VO administrator F2F Data and Storage Management TEG, Amsterdam6

Remarks to DM and SM TEGS Separation of disk and tape m We need two (and only two) storage classes: o T1D0 and T0D1 o This is because no space (storage class) change is possible in some implementations of SRM m T1D0 has two functions: o Archive of T0D1 o Permanent storage of read-few data (RAW, RECO) P For this the BringOnline functionality is mandatory P We need to access the data directly from the MSS without a need for replication onto another storage P Pinning is also a must (suboptimal usage of tape drives without it) d Help to the garbage collector F2F Data and Storage Management TEG, Amsterdam7

Remarks to DM and SM TEGS Storage Interfaces: SRM and Clouds m Clouds??? m We need an interface for: o BringOnline o Pinning o Defining the storage class (unless different endpoints are used, i.e. different SEs) o Currently (Mumbai) this is done by gfal (what is its future?) m SRM is far from perfect but… o It provides the above o All efforts put into defining a standard were a miserable failure… don’t expect any other interface will be any better m … but… o We could probably wrap the minimal functionality on top of the SE native interface, if available P BringOnline and pinning not available for dCache except in SRM P Can xroot provide this functionality? P Isn’t there the danger it becomes as clumsy as SRM depending on the implementation? F2F Data and Storage Management TEG, Amsterdam8

Remarks to DM and SM TEGS Storage IO, LAN Protocols m What is wrong with POSIX file: protocol? o Very efficient in the StoRM-GPFS implementation used at CNAF o Of course abuse is could be a danger (recursive ls) but this could be taken into account in the implementation (throttling) o Almost anybody can now write a fuse plugin to make it happen, so why not use a powerful commercial protocol? m Should access protocols be more than protocols? o i.e. interact behind the scene with the MSS, discover the file location etc… o Can a tURL be just an access point? P ://diskserver.cern.ch// P … or better file: P Avoid accepting URLs like “/castor/cern.ch/…” d Needs to be fixed in application layer? No “guesses”? o Do we need different URLs for different operations? P Transfer and posix-like access F2F Data and Storage Management TEG, Amsterdam9

Remarks to DM and SM TEGS Conclusion m Whatever the future is: o Consider we need a permanently running system o No disruption of service of more than 24 hours for migration o Millions of replicas are in the current system and… it works (even when not optimal) o Any drastic change requires a lot of work on the user side (experiments framework) o Old and new systems must exist in parallel m Requirements may be different depending on the Computing Model of experiments o 7 to 12 analysis centers (LHCb) is different from 50 to 70 centers o Solutions may not be universal and complication may not be required F2F Data and Storage Management TEG, Amsterdam10