Stato dello storage al Tier1 Luca dell’Agnello Mercoledi’ 18 Maggio 2011

2 INFN-CNAF CNAF is the central computing facility of INFN – Italian Tier-1 computing centre for the LHC experiments ATLAS, CMS, ALICE and LHCb... – … but also one of the main processing facilities for other experiments: BaBar and CDF Astro and Space physics VIRGO (Italy), ARGO (Tibet), AMS (Satellite), PAMELA (Satellite) and MAGIC (Canary Islands) More... Disk shares CPU shares Year CPU power [HS06] Disk Space [PB] Tape Space [PB] k k K8.410

EsperimentAllocated (TB)Pledged (TB) Alice Atlas CMS2200 LHCb BABAR SuperB6445 CDF AMS ARGO AUGER FERMI/GLAST2054 MAGIC PAMELA VIRGO Storage shares (May 2011)

Storage resources 8.4 PB of disk on-line under GEMSS 7 DDN S2A9950 (2 TB SATA disks for data, 300 GB SAS disks for metadata) 7 EMC EMC NSD 10 Gbps servers on DDN 60 NSD 1Gbps servers on EMC GridFTP (mostly WAN transfers) servers Most with 10 Gbps connection 1 tape library Sl8500 (10 PB on line) with 20 T10Kb drives – 1 TB tape capacity, 1 Gbps of bandwidth for each drive – Drives interconnected to library and tsm-hsm servers via dedicated SAN (TAN) – TSM server common to all GEMSS instances All storage systems and disk-servers interconnected via SAN (FC4/FC8)

S SAN: Star topology CORE FC director BR48000 BR24000 sw4 BR3BR4 ddn1ddn2ddn3ddn4ddn5ddn6ddn7 emc3emc4emc5emc6 emc1 sw4sw3 sw1 sw6 sw5 emc2 sw2 BR5BR6 S S SSS S Tape drive S Server sw5 FC switch Disks array

The requirements Ricerca di soluzione comune  Adatta per esigenze esperimenti LHC Sistema scalabile a dimensioni O(10) PB Disponibilità di sistema HSM per archiviazione e recupero dinamico da tape Migliaia di accessi concorrenti Throughput aggregato O(10) GB/s  Flessibile per necessità esperimenti non LHC Essenziale non aumentare difficoltà di uso rispetto a soluzioni home-made  Semplicità gestione ed alta affidabilità essenziali TB Storage al Tier1 Protipo Tier1 dal (Alcune) scelte cruciali: batch system Mass Storage System

7 2003: CASTOR chosen as MSS (and phased out Jan 2011) Large variety of issues both at set-up/admin level and at VO’s level (complexity, scalability, stability, support) 2007: start of a project to realize GEMSS, a new grid- enabled HSM solution based on industrial components (parallel file-system and standard archival utility) StoRM adopted as SRM layer and extended to include the methods required to manage data on tape GPFS and TSM by IBM chosen as building blocks An interface between GPFS and TSM implemented (not all needed functionalities provided out of the box) Mass Storage System at CNAF: the evolution (1)

8 Q2 2008: First implementation (D1T1, the easy case) in production for LHCb (CCRC’08) Q2 2009: GEMSS (StoRM/GPFS/TSM), the full HSM solution, ready for production Q3 2009: CMS moving from CASTOR to GEMSS Q1 2010: the other LHC experiments moving to GEMSS End of 2010: all experiments moved from CASTOR to GEMSS All data present on CASTOR tapes copied to TSM tapes CASTOR tapes recycled after data check Mass Storage System at CNAF: the evolution (2)

9 Building blocks of GEMSS system ILM DATA FILE GEMSS DATA MIGRATION PROCESS DATA FILE StoRM GridFTP GPFS DATA FILE WAN DATA FILE TSM DATA FILE GEMSS DATA RECALL PROCESS DATA FILE WORKER NODE SAN TAN LAN SAN Disk-centric system with five building blocks 1. GPFS: disk-storage software infrastructure 2. TSM: tape management system 3. StoRM: SRM service 4. TSM-GPFS interface 5. Globus GridFTP: WAN data transfers

Storage setup Disk storage partitioned in several GPFS clusters One cluster for each (major) experiment with: – several NSD’s (es. 8 for Atlas, 12 for CMS) for data (LAN) – 2 NSD’s for metadata – 2-4 gridftp servers (WAN) – 1 storm end-point (1 BE FE’s) – 2-3 tsm-hsm servers (if needed) data SATA drives SAS drives Metadata NDS Data NSD Ethernet Core Switch FARM WAN 1Gbit 10Gbit 4Gbit 8Gbit SAN 20Gbit ~60Gbit gridFTP 10Gbit Largest file- systems in production: Atlas and CMS (2.2 PB)

Migration Recall GEMSS 1/2 HSM-1HSM-2 TSM-Server TSM DB StoRM GridFTP WAN I/O SRM request Disk Tape Library Storage Area Network Tape Area Netwok

GEMSS 2/2 HSM-1 HSM-2 TSM-Server TSM DB StoRM GridFTP WAN I/O SRM request GPFS Server LAN Worker Node Disk Tape Library Sorting Files by Tape SANTAN

GEMSS HA TSM DB is stored on a CX on the SAN TSM DB is backed up every 2 hours on a different CX disk and every 12 hours on tape with a persistency of 6 days TSM-SERVER have a secondary server in stand-by – It’s possible to move the DB on the CX directly to the secondary server – With a floating IP all client are redirect to the new server We have 2 or 3 TSM-HSM clients for VO for failover GPFS servers, StoRM FEs and GridFTP servers are in cluster – StoRM BE is a single point of failure (cold spare ready) – Nagios (proactive) alarms

Customer (and our) satisfaction INFN T1 availability > MoU threshold in the last months – Storage component big player for this To be honest: 1 incident (CMS) last year – Risk of loss of data In general good feed-back from experiments – CMS CMS – Atlas reprocessing very good efficiency (waiting for official report) Atlas reprocessing CMS queue (May 15)Farm- CMS storage traffic

Yearly statistics LAN trafficWAN traffic Tape trafficMounts/hour

What’s new Key-word: consolidation – Complete revision of the system – Implementation of new features (e.g. DMAPI driven recalls) – Implementation of test-bed for certification of GEMSS – Verification of compatibility of GPFS, TSM, StoRM and home-made layer Original plan: NFS 4.1, HTTP etc… – Not really tested yet  – IPv6 support ? New (internal) requirement: tiered storage – Aggregation of different technologies – Implementation with GPFS policies?

Backup slides

Soluzioni all’esame Storage dei RAW al CNAF, RECO e DST a LNF Buffer disco (~ 20 TB) per evitare uso locale della libreria Upgrade di banda accesso LNF 1÷2Gbps Da dettagliare il modo in cui effettuare le copie remote – Soluzione Interim con gridftp – In esame accesso con cluster GPFS geografico Iniziata discussione tecnica con KLOE CdG Aprile 2011

GEMSS layout for a typical LHC Experiments at INFN Tier-1

21 GEMSS in production for CMS Good-performance achieved in transfer throughput – High use of the available bandwidth – (up to 8 Gbps) Verification with Job Robot jobs in different periods shows that CMS workflows efficiency was not impacted by the change of storage system – “Castor + SL4” vs “TSM + SL4” vs “TSM + SL5” As from the current experience, CMS gives a very positive feedback on the new system – Very good stability observed so far CNAF ➝ T1-US_FNAL CNAF ➝ T2_CH_CAF GEMSS went in production for CMS in October 2009 ✦ w/o major changes to the layout - only StoRM upgrade, with checksum and authz supportbeing deployed soon also

One year statistics Native GPFS (only lan traffic) GridFTP (mostly WAN traffic)

Upgrade StoRM StoRM installato su end-point Atlas e archive a Marzo – problemi riscontrati non ancora risolti memory leak sul BE (sotto carico) Segmentation fault sui FE – Work-around (nagios) In attesa nuova versione (1.7.0 da EMI 1) – Meta’ Giugno?