1. The Atlas Petabyte Datastore A grid enabled, networked data storage system: CrystalGrid Workshop 15 th Sept 2004 David Corney. d.r.corney@rl.ac.uk

2. Data Store Overview General purpose, multi user, data archive. In use over 20 years. Four major upgrades. Current capacity 1PB – largest (non dedicated) multi user system in UK academia? Grid Interfaces: –SE (Storage Element) – will be SRM compliant –SRB interface (Storage Resource Broker)

3. STK 9310

4. Switch_1Switch_2 RS6000 fsc0fsc1 fsc0 9940B 12345678 1114111415 fsc1fsc0fsc1fsc0 1213121315 rmt1 rmt4rmt3rmt2 rmt5-8 AAAAAAAA STK 9310 Powder Horn Gbit network 1.2TB

5. flfstk tapeserv Farm Server flfsys (+libflf) user flfscan data transfer (libvtp) catalogue data STK tape drive cellmgr Catalogue Server (brian) flfdoexp (+libflf) flfdoback (+libflf) datastore (script) Robot Server (buxton) ACSLS API control info (mount/ dismount) data Tape Robot flfsys user commands (sysreq) SE recycling (+libflf) read Atlas Datastore Architecture 28 Feb 03 - 2 B Strong SSI CSI flfsys farm commands (sysreq) LMU flfsys admin commands (sysreq) administrators flfaio IBM tape drive flfqryoff (copy of flfsys code) Backup catalogue stats flfsys tape commands (sysreq) servesys pathtape long name (sysreq) short name (sysreq) frontend backend Pathtape Server (rusty) (sysreq) importexport flfsys import/export commands (sysreq) libvtp User Node I/E Server (dylan) ? Copy B Copy C ACSLS cache disk Copy A vtp user program tape (sysreq)

6. Strategy - De-couple user and application from storage media. Upgrades and media migration occur behind the scenes High resilience - very few Single Point Failures High reliability high, availability (99.9986% in 2003) Constant environmental monitoring linked to alarm/call out Lifetime data integrity checks hardware and software Fire safe and off-site backups; Tested disaster recovery procedures; media migration, recycling Easy to exploit (endless) new technology Technology watch to monitor future technology path

7. Robot History M860 –110GB STK 4400 –1.2Tbytes IBM 3494 –30Tbytes STK 9310 –1Pbyte

8. Hardware upgrade - completed Jun 2003 STK 9310 Powderhorn with 6000 slots (1.2Pbytes) 4 IBM 3590 B drives now phased out –10 Gbyte native –10 Mbyte/s transfer 8 New STK 9940B drives –200 Gbyte native –30Mbytes/sec/drive transfer –240Mbyte/sec theoretical maximum bandwidth 4 RS6000 Data servers (+ 4 others) 1Gbit networking (Expected to become 10Gbit by 2005) Data Migration to new media completed ~ Feb 2004

9. Users Particle Physics Community (LHC: CMS, Atlas, LHcb,….) ISIS, British Atmospheric Data Centre EISCAT (Radar research) National Earth Observation Data Centre World Data Centre, BITD Central Laser Facility Diamond… National Crystallography Service, Southampton University, WASP, VIRGO Consortium Integrative Biology, Others…

10. Interfaces Light weight interfaces: –Client server configuration: tape command for many platforms Virtual Tape Protocol (VTP) Fortran and C callable library Heavy weight interfaces: –SRB interface –SE interface developed for EDG/GRIDPP/GRIDPP2…

11. SRB-ADS architecture SRB MCAT Database SRB MCAT Server SRB ADS Server SRB Client SRB Disk Server (Local Server) Atlas Data Store SRB ADS Server SRB-ISIS server instance SRB-BADC server instance SRB-CCLRC server instance Port 5600 Port 5601 Port 5602

12. Adding Interfaces for ADS ADS Central Catalog Server flfsys ADS Farm Server ADS Farm Server ADS Farm Server SRB01 Server SRB02 Server ADS Farm Server VTP Interface SRB Interface LCG Servers SE Interface SRB Users SE Users VTP Users (tape)

13. Logical Resource for Containers ADS-cache resource ADS-tape resource ADS-logical-resource Sput –c Ssyncont Copies container from cache to tape

14. The Storage Element (SE) mass storage interface

15. The Storage Element (SE) A component of European Data Grid (EDG/EGEE) middleware developed by CCLRCs e-Science & PPD departments Uniform Grid Interface which enables a standard protocol for mass data transfer across the grid, between the many diverse Mass Storage Systems, including: –Atlas Petabyte Data Store –CASTOR –ENSTOR –HPSS –Others…

16. SE Deployment CERN – Castor and disk UAB Barcelona – Castor RAL – Atlas DataStore and Disk ESA/ESRIN – disk CC-IN2P3 – HPSS INFN / CNAF – disk FZK Karlsruhe – disk

17. Questions?

18. Digital Curation Centre Joint collaboration between CCLRC, UKOLN, and Edinburgh and Glasgow Universities. Provide advice, support, research and Development into aspects of Digital Curation for the UK HE community Funded jointly by JISC and EPSRC - £1m/year for three years initially. Feb 2004- 2007 Establish collaboration with industrial partners…

19. Objectives Vibrant research programme –addressing the wider issues of digital curation Collaborative Associates Network of Data Organisations –strong links across existing community of practice –engagement with curators (individuals & organisations) Services –to evaluate tools, methods, standards and policies –a repository of tools and technical information Virtuous circle –expertise, experience & requirement feed into the DCC research programme

20. Digital Curation Centre - Organisation Service Operations Group management & governance Industry research collaborators standards bodies users: communities of practice U. of Edinburgh U. of Glasgow UKOLN (Bath) NDCC/NeSC focus & physical presence curation organisations e.g. DPC JISC & Research Councils Management Board Advisory Group Collaborative Associates Network of Data Organisations CCLRC Steering & Policy Committee Research Co-ordination Committee

21. CCLRCs role within the DCC Standards watch Standards definition and publication Tools watch Tools selection and certification Registry of metadata standards Metadata research

22. DCC role in Certification DCC will help to create –Standards against which to perform audit and certification OAIS Reference Model and follow-on work –Processes for accreditation and certification Work in Digital Repository Certification Task Force –Organisation(s) to perform accreditation and certification

23. ADS SRB Interface for CMS SRB MCAT server SRB ADS Server ADS Server Farm Oracle Database Cluster ADS Tape Robot System ADS Pathtape server File System File System SRB Client SRB File Server SRB Client SRB File Server CSF disk server At RAL CMS node External to RAL SRB Client, External