Daniele Cesini – INFN-CNAF CSN3@Arenzano - 19/09/2017 CNAF and CSN3 Daniele Cesini – INFN-CNAF CSN3@Arenzano - 19/09/2017

CNAF Mission: 4 pillars CNAF Technology Transfer & External Funds: to support innovation and development projects and the recruitment of temporary staff. TT towards industry, public administration and society at large. Scientific Computing: support for the 4 WLCG experiments, 30+ Astro-particle and GW experiments, theoretical physics, beams simulations. CNAF Research and Innovation: Distributed Systems (CLOUD and GRID), external projects Software Developments for experiments and ext. projects Tracking on the new hardware technology INFN National IT Services: Administrative services Networking services Document repositories Code repositories Web sites …. 2 CSN3 - 19/09/2017 Daniele Cesini – INFN-CNAF

Current Organization Chart Research and Innovation Tech. Transfer & External Projects Scientific Computing Daniele Cesini – INFN-CNAF CCR - 23/05/2017

Experiments @CNAF 4 LHC 34 non-LHC CNAF-Tier1 is officially supporting about 40 experiments 4 LHC 34 non-LHC 22 GR2 + VIRGO 5 GR3 AGATA/GAMMA, FAMU, NEWCHIM/FARCOS, NUCLEX FAZIA 7 GR1 non LHC Ten Virtual Organizations in opportunistic usage via Grid services (on both Tier1 and IGI-Bologna site) Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

User Support @ Tier1 CNAF 6 group members (post-docs) 3 group members, one per experiment, dedicated to ATLAS, CMS, LHCb 3 group members dedicated to all the other experiments 1 close external collaboration for ALICE 1 group coordinator from the Tier1 staff Technical support for non-trivial problems provided by ALL the CNAF sysadmins CNAF Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

User Support activities The group acts as a first level of support for the users Incident initial analysis and escalation if needed Provides information to access and use the data center Takes care of communications between users and CNAF operations Tracks middleware bugs if needed Reproduces problematic situations Can create proxy for all VOs or belong to local account groups Provides consultancy to users for computing models creation Collects and tracks user requirements towards the datacenter Including tracking of extrapledges requests Represent INFN-Tier1 in WLCG coordination daily meeting (Run Coordinator) Daniele Cesini – INFN-CNAF CCR WS - 23/05/2017

Il Tier1@CNAF – numbers in 2017 25.000 CPU core 27PB disk storage 70PB tape storage (pledged) 45 PB used Two small HPC farms: 30TFlops dp infiniband interconnect (with GPUs and MICs) 20TFlops dp CPU only Omnipath interconnect 1.2MW Electrical Power (Max available for IT) Used 0.7MW PUE = 1.6 Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

CNAF Future Data transfer rate to CNAF tapes will increase in the next years 250+ PB by 2023 100 PB disk by 2023 100k CPU cores by 2023 CNAF power and cooling infrastructure already fits these requirements Adequate up to LHC RUN3 (2023) Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

Access to the farm (CPU) Grid services with digital certificates authentication gLite-WMS gLite-CE DIRAC Local access via “Bsub” Login to bastion.cnaf Login into a User Interface Batch job submission to the farm LSF bsub Interactive via Cloud@CNAF New CLOUD infrastructure Not officially funded (yet) Best fits the needs of small collaboration without a highly distributed computing model Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

“Cloud” infrastructure 1392 VCPUs 4.75TB RAM 50TB Disk Storage Web interface to manage VMs and storage SSH access FAZIA first use case Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

Resource distribution @ T1 GR3 no LHC CPU: 0.1% GR3 no LHC Disk: 0.4% GR3 no LHC Tape: 1.7% Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

CSN3 Resources at CNAF FAMU Disk Usage: 5TB 15% 17% 16% AGATA TAPE Usage – last 90 days FAMU Disk Usage: 5TB FAMU CPU Usage – year 2016 Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

Extrapledge management We try to accommodate extrapledge requests, in particular for CPU Handled manually RobinHood Identification of temporary inactive VOs Old resources in offline racks that can be turned on if needed Much more difficult for storage Working on automatic solution to offload to external resources Cloud ARUBA, Microsoft HNSciCloud project Other INFN sites Extension to Bari Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

Resource Usage – number of jobs GRID JOBS LOCAL JOBS LSF handles the batch queues Pledges are respected changing dynamically the priority of jobs in the queues in pre- defined time windows Short jobs are highly penalized Short jobs that fail immediately are even more penalized Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

LHC Tier1 Availability and Reliability Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

WAN@CNAF (Status) + 20 Gbit/s towards INFN-Bari LHC ONE LHC OPN RAL SARA (NL) PIC TRIUMPH BNL FNAL TW-ASGC NDGF KR-KISTI RRC-KI JINR LHC ONE Main Tier-2s KIT T1 IN2P3 T1 GARR-MI GARR Mi1 General IP NEXUS 7018 GARR Bo1 20 Gb/s For General IP Connectivity GARR BO1 GARR-BO 60 Gb Physical Link (6x10Gb) Shared by LHCOPN and LHCONE. 20Gb/s 60Gb/s LHCOPN 40 Gb/s CERN LHCONE up to 60 Gb/s (GEANT Peering) Cisco6506-E CNAF TIER1 + 20 Gbit/s towards INFN-Bari (*)Courtesy: Stefano.Zani Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

Data Management Services On-line disks storage Distributed filesystem based on IBM GPFS (now SpectrumScale) Tape Storage Disk buffer for recalled files Long term data preservation POSIX local access to the disk storage from the UI and WN Remote access through: Standard Grid services (SRM/GRIDFTP) Auth/AuthZ based on digital certificates, but simplified wrappers can be provided if needed Xrootd service Standard Web Protocols (HTTPD/WEBDAV) In progress: using Federated AAI (INFN single sign-on) Custom user services Daniele Cesini – INFN-CNAF CCR - 23/05/2017

2014 HPC Cluster 27 Worker Nodes CPU: 904HT cores 640 HT cores E5-2640 48 HT cores X5650 48 HT cores E5-2620 168 HT cores E5-2683v3 15 GPUs: 8 Tesla K40 7 Tesla K20 2x(4GRID K1) 2 MICs: 2 x Xeon Phi 5100 Dedicated STORAGE 2 disks server 60 TB shared disk space 4 TB shared home Infiniband interconnect (QDR) Ethernet interconnect 48x1Gb/s + 8x10Gb/s Disk server 60+4 TB 1Gb/s 10Gb/s 2x10Gb/s Worker Nodes IB QDR ETH CPU GPU MIC TOT TFLOPS (DP - PEAK) 6.5 19.2 2.0 27.7 Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

2017 HPC Cluster 12 Worker Nodes Dedicated STORAGE 2 disk servers + 2 JBOD 300 TB shared disk space (150 with replica 2) 18 TB SSD based file system using 1 SSD on each WN – used for home dirs OmniPath interconnect (100Gbit/s) Ethernet interconnect 48x1Gb/s + 4x10Gb/s 12 Worker Nodes CPU: 768 HT cores Dual E5-2683v4 @2.1 GHz (16 core) 1 KNL node: 64 core (256HT core) CPU GPU MIC TOT TFLOPS (DP - PEAK) 6.5 2.6 8.8 Disk server 150 TB OPA 100Gb/s 12 Gb/s SAS 2x10Gb/s Worker Nodes ETH 150 TB 12 Gb/s SAS - WN installed, storage under testing - Will be used by CERN only - Can be expanded 18TB SSD ETH 1Gb/s Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

Contacts and Coordination Meetings GGUS Ticketing system: https://ggus.eu Mailing lists user-support<at>cnaf.infn.it hpc-support<at>cnaf.infn.it Monitoring system: https://mon-tier1.cr.cnaf.infn.it/ FAQs and UserGuide https://www.cnaf.infn.it/utenti-faq/ https://www.cnaf.infn.it/wp-content/uploads/2016/12/tier1-user-guide- v7.pdf Monthly CDG meetings https://agenda.cnaf.infn.it/categoryDisplay.py?categId=5 No meeting, no problems Not enough: we will organized ad-hoc meeting to define the desiderata and complaints Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017

Conclusion We foster the CNAF data center usage for: Massive batch computation Interactive computing on the Cloud@CNAF infrastructure Experiment data preservation Simplified data management services available Not only grid services Small HPC clusters are available for developments, tests and small production If needed (and funded) they can expanded The User Support team is fully committed to provide assistance in porting computing models to the CNAF infrastructures Daniele Cesini – INFN-CNAF CSN3 - 19/09/2017