1. Evolution of CERN Facilities
Ian Bird, Helge Meinhard – CERN
CWP Workshop
Sam Diego, 23/01/2017
23 Jan 2017
CWP - San Diego

2. CERN Facilities today 2017-18/19 Upgrade internal networking capacity
Refresh tape infrastructure
2017:
225k cores  325k
150 PB raw  250 PB
2018:
???
23 Jan 2017
CWP - San Diego

3. Under construction – 2nd network hub
Provide redundancy to external connectivity (GEANT, LHCOPN, LHCONE, ESNet, etc.)
Physically separate from main CC – in Prévessin
Project started some time ago – originally conceived as part of Wigner project to ensure business continuity
Delayed by lack of funding
Hopefully to be delivered in 2017
23 Jan 2017
CWP - San Diego

4. CERN Prévessin Data Centre?
Drivers:
Optimise the overall cost of LHC computing for CERN over the long-term (~20 years)
”online” and “offline”
Political sensitivity to many large (multi-MW) data centres in the Pays de Gex
Replacement of Wigner DC needed circa 2019 (+ 1 or 2)
Significantly increased demands on CERN computing for Run3, Run4
And others: neutrino platforms, HPC uses, design for FCC, etc., etc.
Consideration that software trigger systems/HLT could be part of a shared facility to enable synergies, operations costs, resource re-use
Commercial resources are not likely to be cost-effective/practical for many of these needs
Guaranteed availability (e.g. for HLT, Tier 0, calibration, data quality)
Bandwidth/latency for HLT
Reasonably scaled local facility for the fixed part of the likely load will still be optimal cost
Elastic/on-demand needs should be separate from this
23 Jan 2017
CWP - San Diego

5. New DC – study
A study group, mandated by the CERN Directorate, was set up
to investigate whether a new DC on the Prévessin site might be a better solution than multiple smaller DCs, and other alternatives for additional capacity
to look at the feasibility: technical, cost and schedule as very high bandwidth would be required from pits to new DC
Assumed starting point to be a copy/paste of the GreenITCube design of GSI/FAIR
Reported end July
Essentially a positive recommendation on feasibility, although significant costs for networking
23 Jan 2017
CWP - San Diego

6. What is the GreenITCube?
A new DC recently built at GSI in preparation for the FAIR project needs
6 story steel frame building (27x30x20m) with space for 128 racks per floor
Separate building for cooling infrastructure and cooling towers
2N power distribution up to 12 MW
Primary and secondary cooling circuits with N+1 redundancy
Passive water cooled racks for up to 35 kW/rack
Design of cooling and power distribution for pairs of floors, i.e. for 4 MW
Recovered heat used for heating of an adjacent building
Construction started in December 2014 and opening ceremony in January 2016 with first servers running in March 2016
Is being reproduced in other locations
23 Jan 2017
CWP - San Diego

7. GSI Experience CERN Directorate:
PUE is extremely good
< 1.1 (1.05 achieved during commissioning)
Cost
Currently equipped for 4 MW at a cost of 12M€ and foreseen cost of 16M€ for full 12MW
Timescale: the full construction took just over 1 year
Potential capacity
12 MW is ~3 times what we have in the CERN DC
Capacity increase available in steps of 4MW
CERN Directorate:
Decision made to provisionally go ahead with an open tender for a Turnkey DC, based on a high level Functional Specification, but with strict qualification criteria
Final decision expected shortly
23 Jan 2017
CWP - San Diego

8. CERN cloud procurements
Since ~2014, series of short CERN procurement projects of increasing scale and complexity
23 Jan 2017
CWP - San Diego

9. 2nd 6th 1st 23rd 20th Mar. Nov. Aug. End: 18th of Dec. 2015
End: 31st of March 2015
ATLAS simulation jobs
Single core VMs
Up to 3k VMs for 45 days
1st Cloud Procurement
End: 18th of Dec. 2015
Target all VOs, simulation jobs
4-core VMs, O(1000) instances
2nd Cloud Procurement
End: 30th of Nov. 2016
Provided by OTC IaaS
4-core VMs, O(1000) instances
500TB of central storage (DPM)
1k public IPs through GÉANT
3rd Cloud Procurement
2015
2016
Agreement between IBM and CERN
CERN PoC to evaluate:
Resource provisioning
Network configurations
Compute performance
Transparent extension of CERN’s T0
End: 13th of May 2016
Sponsored Account
“evaluation of Azure as an IaaS”
Any VO, any workload
Targeting multiple DCs:
Iowa, Dublin and Amsterdam
End: 30th of Nov. 2015
23rd
Mar.
20th
Nov.
23 Jan 2017
CWP - San Diego

10. Recent activity: T-Systems
Running Cores
Batch resources fully loaded
shared among VOs
WAN largely used
Sometimes even saturated
Mixture of “CPU-intensive” and “network-intensive” tasks
MC workloads easier to manage
Workloads
CPU intensive workloads successfully run
With some supplier issues
I/O intensive workloads now being tested
Network remains a concern for bandwidth and control
Some suppliers have no public IPv4 addresses!

11. HELIX NEBULA The Science Cloud
Procurers: CERN, CNRS, DESY, EMBL-EBI, ESRF, IFAE, INFN, KIT, SURFSara, STFC
Procurers have committed funds (>1.6M€), manpower, use-cases with applications & data, in-house IT resources
Experts: Trust-IT & EGI.eu
Objective: procure innovative IaaS level cloud services
Fully and seamlessly integrating commercial cloud (Iaas) resources with in-house resources and European e-Infrastructures
To form a hybrid cloud platform for science
Services will be made available to end-users from many research communities: High-energy physics, astronomy, life sciences, neutron/photon sciences, long tail of science
Co-funded via H2020 (Jan’16-Jun’18) as a Pre-Commercial Procurement (PCP) project: Grant Agreement , total procurement volume: >5M€
R&D is a very important part – represents > 50% of contract volume
23 Jan 2017
CWP - San Diego

12. HNSciCloud – Challenges
Compute
Mostly HTC, integrating some HPC requirements
Full support for containers at scale
Storage
Caching at provider’s site, if possible automatically (avoid managed storage)
Network
Connection via GÉANT
Support of identity federation (eduGAIN) for IT managers
Procurement
Match of cloud providers’ business model with public procurement rules
23 Jan 2017
CWP - San Diego

13. HNSciCloud – Design Phase Contractors
T-Systems, Huawei, Cyfronet, Divia
IBM
RHEA Group, T-Systems, exoscale, SixSq
Indra, HPE, Advania, SixSq
Other major players not interested or dropped out just before tender submission
23 Jan 2017
CWP - San Diego

14. Provisioning services
Moving towards Elastic Hybrid IaaS model:
In house resources at full occupation
Elastic use of commercial & public clouds
Assume “spot-market” style pricing
OpenStack Resource Provisioning
(>1 physical data centre)
HTCondor
Public Cloud
VMs
Containers
Bare Metal and HPC
(LSF)
Volunteer Computing
IT & Experiment Services
End Users
CI/CD
APIs
CLIs
GUIs
Experiment Pilot Factories
23 Jan 2017
CWP - San Diego

15. CERN Facilities – longer term
CERN Meyrin + Prevessin data centres
Aim to keep fully occupied
Option: “Wigner-like” or hosting for business continuity
Could be a cloud-like solution too
2nd Network hub
Elasticity from cloud/commercial resources
Use as required – within cost envelope
Extended with other opportunistic resources
HPC, etc.
Relative weight will be driven (mostly) by cost
23 Jan 2017
CWP - San Diego