INFN Computing infrastructure - Workload management at the Tier-1

Introduction to INFN-T1

The Tier-1 at INFN-CNAF INFN-T1 started as a computing center for LHC experiments Nowadays provides services and resources to ~30 other scientific collaborations ~1.000 WNs , ~21.500 computing slots, ~200 kHS06 Small HPC cluster available with IBA (~33 TFlops) 22 PB SAN disk (GPFS), 43 PB on tape (TSM) integrated as an HSM Supporting LTDP for CDF experiment Dedicated network channel (60 Gb/s) for LHC OPN + LHC ONE 20 Gb/s reserved for LHC ONE Upgrade to 100 Gb/s connection in 2017 25/10/2016

Computing resource usage Computing resources always completely used at INFN-T1 with a large amount of waiting jobs (50% of the running jobs) Expected huge resource request (and increase) in the next years mostly coming from  LHC experiments INFN Tier-1 farm usage in 2016 INFN Tier-1 resource increase 25/10/2016

WAN@CNAF (OPN+ONE upgrade 4060Gb/s) LHC OPN RAL SARA PIC TRIUMPH BNL FNAL TW-ASGC NDGF KR-KISTI RRC-KI JINR Main Tier-2s LHC ONE GARR Mi1 IN2P3 General IP NEXUS GARR Bo1 GARR BO1 20 Gb/s For General IP Connectivity 20Gb/s 40Gb/s 40 Gb Physical Link (4x10Gb) Shared by LHCOPN and LHCONE. 60 Gb Physical Link (6x10Gb) Shared by LHCOPN and LHCONE. Cisco7600 CNAF TIER1 25/10/2016

Physical TIER1 network upgrade to 60Gb/s 26-Sep: Upgraded T1 physical WAN bandwidth (LHC-OPN/ONE) from 4x10Gb/s to 6x10Gb/s. 3-Oct: reached 60 gb/s OPN+ONE (Daily) OPN+ONE (weekly statistics) OPN (Daily) ONE (Daily) OPN: 90% of peak traffic was Xrootd towards WNs from CERN machines. ONE: traffic on ONE generated mainly by T2s 6 25/10/2016

Storage Resources 23 PB (net) of disks GPFS 4.1 3-4 PB for each file system 32 PB on tapes TSM 7.2 17 tape drives (T10KD) High density installations: 2015 tender - 10 PB (net) in 3 racks Decreased number of servers 150 → 100 2016 tender: 3 PB Huawei OceanStor 6800 V3 25/10/2016

Computing resources Farm power: 200K HS-06 2016 tender still to be delivered 2016 tender: HUAWEI x6800 255 x Dual Xeon e5-2618L v4, 128GB ram Should provide 400HS each We continuously get requests of extra pledges, it’s difficult to switch off old nodes But we decommissioned many racks this year. New hardware for virtualization infrastructure Managed by Ovirt New VMWare contract with special discount price 25/10/2016

Monitoring Complete refactory of monitoring and alarm tools across all CNAF functional units Past: nagios, lemon, home made probes and sensors, legacy UIs Future: Central Infrastructure, Sensu+Uchiwa, InfluxDB and Grafana, community probes and sensors (home made when necessary) 25/10/2016

Provisioning All nodes in a common data-base (home made solution) Warranty information Purchase data OS and network configuration Can interface nagios to data-base to automatically build the node inventory Recently moved from quattor to puppet+foreman Installation and configuration is fully automated Several manifests and configuration shared among different departments Better support achieved by community 25/10/2016

Toward a (semi-)elastic Data Center? Planning to upgrade the Data Center to host resources at least until the end of LHC Run 3 (2023) Extending (dynamically) the farm to remote hosts as a complementary solution Cloud bursting on commercial provider Tests of “opportunistic computing” on Cloud providers Static allocation of remote resources First production use case: part of 2016 pledged resources for WLCG experiments at INFN- T1 are in Bari-ReCaS Participating in HNSCicloud PCP project 25/10/2016

Workload management

Current situation T1 for LHC Experiments (ATLAS, ALICE, CMS, LHCB) Other “heavy users” (AMS, VIRGO) Other VOs (auger, biomed, borexino, juno,…) 2 approaches to submit jobs to INFN-T1 Grid (preferred) Locally 25/10/2016

Grid submission Users reach our site via cream CEs (6 configured so far) or… through one of the many UIs configured One generic UI available Many experiment specific UIs present to fulfill specific requirements in terms of software requirements or specific configuration 25/10/2016

Local submission Generic UI is available for local submission too To be able to log-in users must sign CNAF AUP. Access is given through bastion hosts and then users can ssh to their experiment UI. 25/10/2016

Batch system LSF (currently v9.x) is the production batch system at INFN-T1 Adopted many years ago, to solve issues had with torque+maui Never a problem related to bug, scalability, performance Always found a configuration to fulfill every request coming from experiments Recently adopted multi-core job configuration Installation with “best-practices” configuration Shared cNFS file system among all clients nodes 3 redundant masters Single LSF instance for the whole farm Managing 1000 WNs, 50 UIs, 6CEs LHCb T2 resources 25/10/2016

Batch system 2 more, separate, LSF batch systems T3 Bologna Kept separated because resources are not part of INFN-T1 pledges HPC cluster Cluster with some special settings, kept separate for now 25/10/2016

LSF problems Mainly one: cost of the license Agreement with IBM, acquired at a special price, but still very expensive! Main concern is for the future requirement of computing resources Support is over by the end of 2018 In 2019 we will still be able to use our purchased licenses but no right to upgrade and no support. 25/10/2016

Planning the switch HTCondor seems to be the best alternative for us We have software relying on LSF that needs to be adapted May be more time consuming compared to batch system switch itself We need to instruct users Are command wrappers for basic submission and query available? We are not in hurry, since the number of licenses is big enough to fulfill our requirements for the near future. 25/10/2016