1. GridPP Storage perspectives
Brian Davies
pre-GDB Data Management , CERN
13 September 2016

2. Areas of interest of GridPP Storage
ATLAS analysis for new schema.
Volumes
Network needs/slot capacities
T2 Evolution
What changes happen to those SEs which remain as well as those which remain?
Structure?
Testing particular areas of interest
(Non)-LHC VO support
LFC,FTS(dev/ops),WMS, perfSONAR/TCP tuning
(cloud/VO support)
T1 activity to put Grid middleware on CEPH

3. UK ATLAS T2 evolvement use case
Site
Maximal Slot Count
Normal Slot Count
Pledged Storage (TB)
Actual (TB)
Additional slots
Factor increase
Glasgow
4500
4000
1305
2583
2100
1.53
QMUL
4300
3000
1386
2585
1.70
Lancaster
3100
2500
1229
1700
1.68
Manchester
3900
1080
1750
1400
1.47
Σ Main T2s
15800
12500
5000
9018
RHUL
3400
2000
963
1418
ECDF
600
400
540
891
Oxford
1300
800
729
823
RALPP
1200
360
757
L'pool
1000
486
735
Sheffield
760
387
429
B'Ham
200
189
335
Cambridge
250
207
321
Sussex
100 45 50
Durham
2200 44
Brunel
1150 33
UCL 60 10 81 19
Imperial
Σ Small T2s
15050
7370
3987
5874
RAL T1
11400
8000
5300 1
Site need to accommodate additional capacities
Large increase in WAN traffic
90% LAN traffic is WN
6/15GB/s of WN traffic at T1 is for ATLAS Jobs
Disk servers/ head nodes to cope with additional connections
CMS use ~2.5MB/s per job
Storage pledge meet
What to do with 5874 TB of “free” space

4. T2 Evolution Context VOs want to use fewer storage endpoints
Fewer sites want to support storage
Reduction in funded manpower
Sites need to run with fewer people.
(And need to transition with fewer people, too, as decline has already started)
Sites also need to work well with non-LHC assumptions, techniques.

5. T2 Storage Characteristics
Churn (cache like?)
but low reuse rate
Large Size variation - 3PB down to <500TB
Access methods
SRM, Xroot, GridFTP, HTTP…
X509+VOMS authentication
Total Capacity
Unlikely that total demand for T2 storage will fall.
What do we do with existing storage?
Allow to "slowly degrade" at T2Cs?
"Physically consolidate" at T2Ds?

6. Reduce Expenditure by:
Reliability Requirements
Features
Maintenance
Manpower

7. Expenditure Method Types:
Reduce
Reliability Requirements
Features
Maintenance
Manpower
Egalitarian
Barebones
Network
Caching/Read only

8. Some existing solutions..

9. Data CVMFS Caching / Read only
Read only data provision, optimised for small, slowly changing datasets.
CVMFS + (Xrootd backhaul) + [future cache tiers]
Deployed for LIGO US by OSG - tested in production.
Some exploration at RAL for non-LHC VOs
Not suitable for rapidly changing, large datasets.
Easy transition (for all sites with CVMFS up to date…)
Caching /
Read only

10. T2C Caching WLCG Data Caching Group. Two solutions:
Xrootd - mature, not widely used?, compatible with xrootd federation
"DPM" - In development, protocol agnostic, but DPM specific.
Easy transition (for sites with DPM or Xrootd services)
Caching /
Read only

11. ARC Cache ARC "prefetch" cache for jobs
Working happily at Durham for some months now.
Easy transition (for sites with ARC CEs)
Caching /
Read only

12. Cache questions
None of the mentioned solutions solve job output issues.
Centralised object store for log files?
Is caching efficient?
Small scale studies suggest 90% of data is read maybe twice, at best.
Caching as an proxy for acceptance of low reliability?
How is this different from redirection, in practice?
Caching /
Read only

13. Pure network model UCL CPU paired to remote SE at QMUL.
Original localSE model from 10+ years ago.
Can this apply to all sites/Regions
Tests by Alastair Dewhurst against Oxford T2 for ATLAS for more sophisticated (federated) solutions.
VOs already federating so use their frameworks
Currently resolving HammerCloud's assumptions re data locality.
Obviously, only test ATLAS workloads…
Network

14. Simpler storage Current T2 storage is specialist - big RAID-6 arrays.
Work by M.Ebert on ZFS, software 'RAID':
faster, more flexible, more extensible than HW RAID
small efficiency gains via compression (~4%)
Small step (server local changes).
(Also heads off RAID6 scaling 8TB disks)
Egalitarian

15. SRM retirement? Barebones Egalitarian but spacetokens still an issue.
Slow movement on this, due to existing dependencies.
Agreement "in principle" that not necessary for T2 workflows
but spacetokens still an issue.
Still no good, general-purpose alternative for protecting/guaranteeing VO space.
Barebones
Egalitarian

16. Single protocol? Barebones Egalitarian
HTTP(S)/WebDAV?
HTTP Deployment taskforce signed off on features
XROOT? [or xrootd]
"unusual", but xrootd also supports HTTP(S)
GridFTP?
Advantage(?) of "Globus Connect" compatibility
Barebones
Egalitarian

17. SE "retirement" Without SRM, what is the need for a traditional SE?
Revert to Classic SEs
Xrootd, HTTP, GridFTP endpoints in front of:
ceph/hdfs/other distributed storage [~posix]
Object storage via [xrootd/S3/SWIFT/…]
Transition is hard.
(Unless we can dump all of the data at a site…)
Unless churn is used to our advantage
ATLAS delete 2PB/month from UK T2s (is this normal?)
Barebones
Egalitarian

18. Support for non-LHC VOs
LFC, FTS, WMS, DIRAC usage for non-LHC VOs non negligible
Support for products…
Sharing technology/monitoring/hardware useful
New communities require alternative technologies
Licensing issues

19. Tier 1 storage evolution
CERN switched from Castor to EOS for disk only storage several years ago.
RAL have been working on a replacement storage service.
Why not choose DPM/dCache?
Grid storage not popular with non-LHC users. They want things like S3.
Other groups at RAL are also using Ceph so expertise readily available.
Ceph is an object store:
Fits LHC use case well (external file catalogues, don’t update files)
Want to take advantage of this simplicity
Need to add support for XrootD and GridFTP

20. GridFTP and XrootD for Echo
CERN have developed Ceph plugin for xrootd
Being used for the Castor tape buffers.
Significant work now focusing on performance improvements.
RAL are developing GridFTP plugin for Ceph
With help from CERN and Brian Bockelman.
Most functionality in place, starting looking at performance.
Multiple stream assembly planned to
improve GridFTP performance via FTS.

21. Echo Status No SRM – is that a problem?
The Castor SRM will deal with all the tape stuff.
Currently just support ATLAS and CMS who claim to not need one!!
Other VOs to follow.
Accounting will be provide via .json file
Echo is now registered as a ‘T3D’ for ATLAS.
Starting to add SE and run functional tests
Andrew Lahiff is validating CMS jobs.
Waiting for GridFTP improvements before adding to PhEDEx
Aim to have production quality SE by April 2017.

22. Future areas of interest
AAI/AARC
EUDAT/SAGE/ESiWACE project collaboration/co-ordination
Further non-HEP community support

23. Summary GridPP supporting/following various storage topics.
Balance LHC/nonLHC requirements
T2 evolution on track
T1 (r)evolution progressing
But Tape staying the Same
Collaboration between LHC/nonLHC beneficial to both parties

24. Questions
Thanks to A.Dewhurst and S.Skipsey for additional/majority material for this talk.
Thanks to gridpp-storage/GridPP community( particulary J.Jensen and M.Ebert )

25. Backups

26. DYNaFED

27. Network

28. Network