Evolution of CERN Facilities Ian Bird, Helge Meinhard – CERN CWP Workshop Sam Diego, 23/01/2017 23 Jan 2017 CWP - San Diego

CERN Facilities today 2017-18/19 Upgrade internal networking capacity Refresh tape infrastructure 2017: 225k cores  325k 150 PB raw  250 PB 2018: ??? 23 Jan 2017 CWP - San Diego

Under construction – 2nd network hub Provide redundancy to external connectivity (GEANT, LHCOPN, LHCONE, ESNet, etc.) Physically separate from main CC – in Prévessin Project started some time ago – originally conceived as part of Wigner project to ensure business continuity Delayed by lack of funding Hopefully to be delivered in 2017 23 Jan 2017 CWP - San Diego

CERN Prévessin Data Centre? Drivers: Optimise the overall cost of LHC computing for CERN over the long-term (~20 years) ”online” and “offline” Political sensitivity to many large (multi-MW) data centres in the Pays de Gex Replacement of Wigner DC needed circa 2019 (+ 1 or 2) Significantly increased demands on CERN computing for Run3, Run4 And others: neutrino platforms, HPC uses, design for FCC, etc., etc. Consideration that software trigger systems/HLT could be part of a shared facility to enable synergies, operations costs, resource re-use Commercial resources are not likely to be cost-effective/practical for many of these needs Guaranteed availability (e.g. for HLT, Tier 0, calibration, data quality) Bandwidth/latency for HLT Reasonably scaled local facility for the fixed part of the likely load will still be optimal cost Elastic/on-demand needs should be separate from this 23 Jan 2017 CWP - San Diego

New DC – study A study group, mandated by the CERN Directorate, was set up to investigate whether a new DC on the Prévessin site might be a better solution than multiple smaller DCs, and other alternatives for additional capacity to look at the feasibility: technical, cost and schedule as very high bandwidth would be required from pits to new DC Assumed starting point to be a copy/paste of the GreenITCube design of GSI/FAIR Reported end July Essentially a positive recommendation on feasibility, although significant costs for networking 23 Jan 2017 CWP - San Diego

What is the GreenITCube? A new DC recently built at GSI in preparation for the FAIR project needs 6 story steel frame building (27x30x20m) with space for 128 racks per floor Separate building for cooling infrastructure and cooling towers 2N power distribution up to 12 MW Primary and secondary cooling circuits with N+1 redundancy Passive water cooled racks for up to 35 kW/rack Design of cooling and power distribution for pairs of floors, i.e. for 4 MW Recovered heat used for heating of an adjacent building Construction started in December 2014 and opening ceremony in January 2016 with first servers running in March 2016 Is being reproduced in other locations 23 Jan 2017 CWP - San Diego

GSI Experience CERN Directorate: PUE is extremely good < 1.1 (1.05 achieved during commissioning) Cost Currently equipped for 4 MW at a cost of 12M€ and foreseen cost of 16M€ for full 12MW Timescale: the full construction took just over 1 year Potential capacity 12 MW is ~3 times what we have in the CERN DC Capacity increase available in steps of 4MW CERN Directorate: Decision made to provisionally go ahead with an open tender for a Turnkey DC, based on a high level Functional Specification, but with strict qualification criteria Final decision expected shortly 23 Jan 2017 CWP - San Diego

CERN cloud procurements Since ~2014, series of short CERN procurement projects of increasing scale and complexity 23 Jan 2017 CWP - San Diego

2nd 6th 1st 23rd 20th Mar. Nov. Aug. End: 18th of Dec. 2015 End: 31st of March 2015 ATLAS simulation jobs Single core VMs Up to 3k VMs for 45 days 1st Cloud Procurement End: 18th of Dec. 2015 Target all VOs, simulation jobs 4-core VMs, O(1000) instances 2nd Cloud Procurement End: 30th of Nov. 2016 Provided by OTC IaaS 4-core VMs, O(1000) instances 500TB of central storage (DPM) 1k public IPs through GÉANT 3rd Cloud Procurement 2015 2016 Agreement between IBM and CERN CERN PoC to evaluate: Resource provisioning Network configurations Compute performance Transparent extension of CERN’s T0 End: 13th of May 2016 Sponsored Account “evaluation of Azure as an IaaS” Any VO, any workload Targeting multiple DCs: Iowa, Dublin and Amsterdam End: 30th of Nov. 2015 23rd Mar. 20th Nov. 23 Jan 2017 CWP - San Diego

Recent activity: T-Systems Running Cores Batch resources fully loaded shared among VOs WAN largely used Sometimes even saturated Mixture of “CPU-intensive” and “network-intensive” tasks MC workloads easier to manage Workloads CPU intensive workloads successfully run With some supplier issues I/O intensive workloads now being tested Network remains a concern for bandwidth and control Some suppliers have no public IPv4 addresses!

HELIX NEBULA The Science Cloud Procurers: CERN, CNRS, DESY, EMBL-EBI, ESRF, IFAE, INFN, KIT, SURFSara, STFC Procurers have committed funds (>1.6M€), manpower, use-cases with applications & data, in-house IT resources Experts: Trust-IT & EGI.eu Objective: procure innovative IaaS level cloud services Fully and seamlessly integrating commercial cloud (Iaas) resources with in-house resources and European e-Infrastructures To form a hybrid cloud platform for science Services will be made available to end-users from many research communities: High-energy physics, astronomy, life sciences, neutron/photon sciences, long tail of science Co-funded via H2020 (Jan’16-Jun’18) as a Pre-Commercial Procurement (PCP) project: Grant Agreement 687614, total procurement volume: >5M€ R&D is a very important part – represents > 50% of contract volume 23 Jan 2017 CWP - San Diego

HNSciCloud – Challenges Compute Mostly HTC, integrating some HPC requirements Full support for containers at scale Storage Caching at provider’s site, if possible automatically (avoid managed storage) Network Connection via GÉANT Support of identity federation (eduGAIN) for IT managers Procurement Match of cloud providers’ business model with public procurement rules 23 Jan 2017 CWP - San Diego

HNSciCloud – Design Phase Contractors T-Systems, Huawei, Cyfronet, Divia IBM RHEA Group, T-Systems, exoscale, SixSq Indra, HPE, Advania, SixSq Other major players not interested or dropped out just before tender submission 23 Jan 2017 CWP - San Diego

Provisioning services Moving towards Elastic Hybrid IaaS model: In house resources at full occupation Elastic use of commercial & public clouds Assume “spot-market” style pricing OpenStack Resource Provisioning (>1 physical data centre) HTCondor Public Cloud VMs Containers Bare Metal and HPC (LSF) Volunteer Computing IT & Experiment Services End Users CI/CD APIs CLIs GUIs Experiment Pilot Factories 23 Jan 2017 CWP - San Diego

CERN Facilities – longer term CERN Meyrin + Prevessin data centres Aim to keep fully occupied Option: “Wigner-like” or hosting for business continuity Could be a cloud-like solution too 2nd Network hub Elasticity from cloud/commercial resources Use as required – within cost envelope Extended with other opportunistic resources HPC, <lhcexperiment>@home, etc. Relative weight will be driven (mostly) by cost 23 Jan 2017 CWP - San Diego