1. SRM2 Migration Strategy
Stephen Burke
STFC/RAL
GSSD meeting, April 12th 2007

2. SRM2 migration strategy
Introduction
How do we migrate from SRM v1 to v2 with the minimum disturbance to sites and users?
Different host or same host?
Considered CASTOR, DPM and dCache
T1 and T2 may be different
What are the issues?
Conclusions/recommendations
GLUE schema v1.3
12th April 2007
SRM2 migration strategy

3. SRM2 migration strategy
Overview
Stability
SURLs
Storage classes
DNS
Information system
Storage accounting
File catalogues
GFAL/lcg-utils/FTS
Conclusions
12th April 2007
SRM2 migration strategy

4. SRM2 migration strategy
Stability
Will SRM v2 be stable enough to run on production SEs during testing?
Tier 2s probably don’t have the resources to do anything else
But testers should be careful not to overload it
Tier 1s may want to be more careful
How stable are the v2 interfaces?
Same SRM interface != same file system
Migration will be easier if the file system is common
dCache is a single product managing both protocols internally – but could have separate instances
DPM has two separate interfaces on different ports – could be separate hosts
CASTOR SRM is a thin layer above CASTOR, so multiple interfaces are easy
12th April 2007
SRM2 migration strategy

5. SRM2 migration strategy
SURLs
Four formats:
srm://se.rl.ac.uk/path/to/file
srm://se.rl.ac.uk:8443/path/to/file
srm://se.rl.ac.uk/srm/managerv1?SFN=/path/to/file
srm://se.rl.ac.uk:8443/srm/managerv1?SFN=/path/to/file
All accepted by lcg-utils + SRMs
srmcp may need the port?
Should only store type 1) in catalogues
Port may change
Same file may be accessible by v1 or v2
SFN should be the same between v1 and v2 if possible
If file systems are separate, should segment the namespaces?
12th April 2007
SRM2 migration strategy

6. SRM2 migration strategy
Storage classes
Experiments want T1D0, T0D1, (T1D1?)
Only T0D1 at T2
DPM can only do T0D1
Want to simulate these with SRM1, then migrate to space tokens in SRM2
Ideally without copying files
In v1 can use different hosts and/or different SFN paths
In v2 space tokens should be independent of SFN, but may still want to segment the namespace
What are T1s doing already?
RAL has separate hostnames, but also segments the SFN name space
srm://ralsrma.rl.ac.uk/castor/ads.rl.ac.uk/prod/grid/hep/disk0tape1/dteam/file
12th April 2007
SRM2 migration strategy

7. SRM2 migration strategy
DNS issues
May want to point “old” v1 hostnames to a “new” v2 machine
Aliases should be OK, SRMs don’t parse the hostname (or port) in a SURL
Host certificate validation is separate
Old hostnames may persist for a long time in catalogues
Must keep the old DNS names registered for some time
~2 years?
Unless catalogues can be migrated?
12th April 2007
SRM2 migration strategy

8. SRM2 migration strategy
Information system
GLUE schema wasn’t designed for SRM, and is not used entirely correctly at the moment
SRM endpoints are published in both GlueService and GlueSEControlProtocol
lcg-utils currently uses GlueService
Historical type “srm_v1” needs to change to correct “SRM” + separate version
gstat now generates warnings for this
Hostname in SURL should be matched against hostname in endpoint, not SEUniqueID
GlueSAPath is used as a prefix for SFNs
Usually one SA per VO at the moment
12th April 2007
SRM2 migration strategy

9. Information system issues
Old endpoints need to be published as long as SURLs exist in catalogues which refer to them
Can publish multiple ControlProtocols per SE, or have separate SEs
lcg-utils/FTS look up GlueService
YAIM currently doesn’t configure the CP
ControlProtocol is orthogonal to SA, so e.g. all VOs see the same protocol(s) for a given SE
SRMs themselves do not use the information system
Storage accounting currently uses size information in the SA
Need to avoid double-counting if possible
12th April 2007
SRM2 migration strategy

10. SRM2 migration strategy
File catalogues
File catalogues should only store type 1) SURLs
No port or “managervx” – look up in information system
Except that srmcp doesn’t use the information system
SURLs should be canonicalised (converted to type 1) before being stored in a catalogue
Applies to VO tools/catalogues too
If SFN changes, catalogues will need to be migrated
May be difficult?
Another sub-group …
12th April 2007
SRM2 migration strategy

11. SRM2 migration strategy
lcg-utils/GFAL/FTS
lcg-utils/GFAL can accept all SURL formats
New space token parameter
Port and SRM type/endpoint are filled in from the information system if necessary
If v1 and v2 are both available, v1 is preferred (unless a space token is specified) – must use full SURL to select v2
SURLs are canonicalised (port and SRM endpoint removed) before being stored in the LFC
Migration can be transparent if SFN doesn’t change
FTS 2 not yet available for testing
All SURL formats?
What happens with srmcp copies?
12th April 2007
SRM2 migration strategy

12. SRM2 migration strategy
Summary
With the right configuration it should be possible to make the transition from v1 to v2 almost seamlessly
But once VOs start using storage classes seriously it will be hard to go back to v1
Tier 2s will generally run DPM or dCache, and should configure both v1 and v2 endpoints on the production SE
Tier 1s may want to separate v1 and v2 on different hosts
Try to keep the same SFNs
Probably need to keep old names in DNS and info system for some time
12th April 2007
SRM2 migration strategy

13. GLUE schema v1.3

14. SRM2 migration strategy
Schema changes
New schema version 1.3, backward-compatible with 1.2
Hard to get new versions agreed – 18 months from 1.2 to 1.3
Various additions to enhance SRM description
Suggested by LCG SRM group
Some iteration with schema group
GLUE is now an OGF group
Working on a major 2.0 release
Not backward-compatible
At least a year away
12th April 2007
SRM2 migration strategy

15. SRM2 migration strategy
New SE features in 1.3
Split all sizes into online and nearline
New attributes for implementation type and version (DPM, dCache etc)
New Status, e.g. Draining
SA (≈ Space) now has SRM2 attributes
RetentionPolicy, AccessLatency, ExpirationMode
More size info in SA: Total, Used, Free, Reserved
Can the implementations actually calculate them?
Free space is important – need to know if disks are full!
SACapability allows arbitrary extra information
But no structure, just “string”
New SAVOInfo – multiple Path/Tag/ACL triplets per SA
Tag ≈ Space Token, ACL ≈ VO
New AccessProtocolMaxStreams
12th April 2007
SRM2 migration strategy

16. SRM2 migration strategy
1.3 schema deployment
LDAP schema is now in certification, will be deployed to production BDIIs soon
Top-level BDIIs first
Need information providers from each SRM group
Probably need YAIM updates to configure
Clients (lcg-utils, FTS, storage accounting, …) need to be updated to use the new information
Can add things incrementally – everything should be backward-compatible
12th April 2007
SRM2 migration strategy