SRM2 Migration Strategy Stephen Burke STFC/RAL GSSD meeting, April 12th 2007

SRM2 migration strategy Introduction How do we migrate from SRM v1 to v2 with the minimum disturbance to sites and users? Different host or same host? Considered CASTOR, DPM and dCache T1 and T2 may be different What are the issues? Conclusions/recommendations GLUE schema v1.3 12th April 2007 SRM2 migration strategy

SRM2 migration strategy Overview Stability SURLs Storage classes DNS Information system Storage accounting File catalogues GFAL/lcg-utils/FTS Conclusions 12th April 2007 SRM2 migration strategy

SRM2 migration strategy Stability Will SRM v2 be stable enough to run on production SEs during testing? Tier 2s probably don’t have the resources to do anything else But testers should be careful not to overload it Tier 1s may want to be more careful How stable are the v2 interfaces? Same SRM interface != same file system Migration will be easier if the file system is common dCache is a single product managing both protocols internally – but could have separate instances DPM has two separate interfaces on different ports – could be separate hosts CASTOR SRM is a thin layer above CASTOR, so multiple interfaces are easy 12th April 2007 SRM2 migration strategy

SRM2 migration strategy SURLs Four formats: srm://se.rl.ac.uk/path/to/file srm://se.rl.ac.uk:8443/path/to/file srm://se.rl.ac.uk/srm/managerv1?SFN=/path/to/file srm://se.rl.ac.uk:8443/srm/managerv1?SFN=/path/to/file All accepted by lcg-utils + SRMs srmcp may need the port? Should only store type 1) in catalogues Port may change Same file may be accessible by v1 or v2 SFN should be the same between v1 and v2 if possible If file systems are separate, should segment the namespaces? 12th April 2007 SRM2 migration strategy

SRM2 migration strategy Storage classes Experiments want T1D0, T0D1, (T1D1?) Only T0D1 at T2 DPM can only do T0D1 Want to simulate these with SRM1, then migrate to space tokens in SRM2 Ideally without copying files In v1 can use different hosts and/or different SFN paths In v2 space tokens should be independent of SFN, but may still want to segment the namespace What are T1s doing already? RAL has separate hostnames, but also segments the SFN name space srm://ralsrma.rl.ac.uk/castor/ads.rl.ac.uk/prod/grid/hep/disk0tape1/dteam/file 12th April 2007 SRM2 migration strategy

SRM2 migration strategy DNS issues May want to point “old” v1 hostnames to a “new” v2 machine Aliases should be OK, SRMs don’t parse the hostname (or port) in a SURL Host certificate validation is separate Old hostnames may persist for a long time in catalogues Must keep the old DNS names registered for some time ~2 years? Unless catalogues can be migrated? 12th April 2007 SRM2 migration strategy

SRM2 migration strategy Information system GLUE schema wasn’t designed for SRM, and is not used entirely correctly at the moment SRM endpoints are published in both GlueService and GlueSEControlProtocol lcg-utils currently uses GlueService Historical type “srm_v1” needs to change to correct “SRM” + separate version gstat now generates warnings for this Hostname in SURL should be matched against hostname in endpoint, not SEUniqueID GlueSAPath is used as a prefix for SFNs Usually one SA per VO at the moment 12th April 2007 SRM2 migration strategy

Information system issues Old endpoints need to be published as long as SURLs exist in catalogues which refer to them Can publish multiple ControlProtocols per SE, or have separate SEs lcg-utils/FTS look up GlueService YAIM currently doesn’t configure the CP ControlProtocol is orthogonal to SA, so e.g. all VOs see the same protocol(s) for a given SE SRMs themselves do not use the information system Storage accounting currently uses size information in the SA Need to avoid double-counting if possible 12th April 2007 SRM2 migration strategy

SRM2 migration strategy File catalogues File catalogues should only store type 1) SURLs No port or “managervx” – look up in information system Except that srmcp doesn’t use the information system SURLs should be canonicalised (converted to type 1) before being stored in a catalogue Applies to VO tools/catalogues too If SFN changes, catalogues will need to be migrated May be difficult? Another sub-group … 12th April 2007 SRM2 migration strategy

SRM2 migration strategy lcg-utils/GFAL/FTS lcg-utils/GFAL can accept all SURL formats New space token parameter Port and SRM type/endpoint are filled in from the information system if necessary If v1 and v2 are both available, v1 is preferred (unless a space token is specified) – must use full SURL to select v2 SURLs are canonicalised (port and SRM endpoint removed) before being stored in the LFC Migration can be transparent if SFN doesn’t change FTS 2 not yet available for testing All SURL formats? What happens with srmcp copies? 12th April 2007 SRM2 migration strategy

SRM2 migration strategy Summary With the right configuration it should be possible to make the transition from v1 to v2 almost seamlessly But once VOs start using storage classes seriously it will be hard to go back to v1 Tier 2s will generally run DPM or dCache, and should configure both v1 and v2 endpoints on the production SE Tier 1s may want to separate v1 and v2 on different hosts Try to keep the same SFNs Probably need to keep old names in DNS and info system for some time 12th April 2007 SRM2 migration strategy

GLUE schema v1.3

SRM2 migration strategy Schema changes New schema version 1.3, backward-compatible with 1.2 Hard to get new versions agreed – 18 months from 1.2 to 1.3 Various additions to enhance SRM description Suggested by LCG SRM group Some iteration with schema group GLUE is now an OGF group Working on a major 2.0 release Not backward-compatible At least a year away 12th April 2007 SRM2 migration strategy

SRM2 migration strategy New SE features in 1.3 Split all sizes into online and nearline New attributes for implementation type and version (DPM, dCache etc) New Status, e.g. Draining SA (≈ Space) now has SRM2 attributes RetentionPolicy, AccessLatency, ExpirationMode More size info in SA: Total, Used, Free, Reserved Can the implementations actually calculate them? Free space is important – need to know if disks are full! SACapability allows arbitrary extra information But no structure, just “string” New SAVOInfo – multiple Path/Tag/ACL triplets per SA Tag ≈ Space Token, ACL ≈ VO New AccessProtocolMaxStreams 12th April 2007 SRM2 migration strategy

SRM2 migration strategy 1.3 schema deployment LDAP schema is now in certification, will be deployed to production BDIIs soon Top-level BDIIs first Need information providers from each SRM group Probably need YAIM updates to configure Clients (lcg-utils, FTS, storage accounting, …) need to be updated to use the new information Can add things incrementally – everything should be backward-compatible 12th April 2007 SRM2 migration strategy