1. Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF CERN Computer Centre Consolidation Project Vincent Doré IT Technical Forum - 10 th June 2011

2. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Overview Current configuration of the CC (excluding B613) Current issues Consolidation project – Goals – Technical choices – Plans and views – New IT room – Main dates On-going work outside the project Questions Computer Center Consolidation – VD - 100611 - 2

3. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Current configuration - 1 Designed and built in early 1970s Fully refurbished around 2000 2 types of equipment both on UPS: – Physics (single power feed, no diesel backup) – Critical (dual power feed and diesel backup) Power capacity: – Nominal current capacity: 2.5MW (including 240kW of critical power) – Extended (usable) capacity: 2.9MW (including 340kW of critical power) forfeiting redundancy for all UPS systems 2 main IT rooms (R-050 -1400m² & Vault - 1200m²). Several dedicated labs (telecom) Computer Center Consolidation – VD - 100611 - 3

4. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Current configuration - 2 3 physics UPS, each with 4*400kvA modules 1 critical UPS with 2*300kVA modules Air cooling installations: for IT equipment including telecoms (capacity up to ~2.5MW), technical areas... and offices – (3+1) Chillers not secured by UPS nor diesel (water at ~5C) – Pumps and ventilation units secured by diesel – Water-tank for stored cooling capacity Water cooling installations: for IT physics equipment (capacity up to 800kW) – (1+1) Chillers not secured by UPS nor diesel (water at ~14C) – Pumps secured by (physics) UPS – 80 water-cooled racks secured by physics UPS – Water-tank for stored cooling capacity Computer Center Consolidation – VD - 100611 - 4

5. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Current issues - 1 Cooling: – Insufficient cooling for critical UPS room – Impossible to improve the cooling in critical UPS room – CC not cooled when running on UPS without diesel – Insufficient cooling when running on diesel IT critical equipment running Pumps & Ventilation Units running No chiller and insufficient stored cold water – Offices in B31 cooled by the CC Computer Center Consolidation – VD - 100611 - 5

6. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Current issues - 2 Power – Insufficient critical power available – No redundancy for critical UPS (> 240 kW) – No redundancy for physics UPS (> 2.7 MW) Other – Limited fire protection in B513 – Critical areas and physics areas strongly coupled Share the same locations, cooling infrastructure and fire risks) Improving critical infrastructure implies improving physics one... at an expensive price Computer Center Consolidation – VD - 100611 - 6

7. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Consolidation Project Main goals – Secure the cooling and fire-protection of Critical IT equipment... Which should be able to run as long as power is available (for ever on diesel) – Increase the available critical power for IT equipment – Restore N+1 redundancy for both critical and physics UPS – Decouple the cooling of B31 from the CC Main points on the wish list – Increase the physics power (if cost-effective) – Secure the cooling of physics equipment  Computer Center Consolidation – VD - 100611 - 7

8. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Technical choices - 1 Upgrade of the electrical infrastructure – Total capacity 3.5 MW of IT equipment including 600kW of critical equipment – N+1 redundancy for critical and physics UPS Separation of the critical and physics infrastructure – 3 critical areas : new IT room (450kW) extended telecom labs (50kW) legacy network area in R-050 (100kW)  – New dedicated cooling installation for the critical areas (dual power feed, diesel) Computer Center Consolidation – VD - 100611 - 8

9. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Technical choices - 2 Construction of a new partially sunken building dedicated to the cooling of the critical areas – (2+1) new chillers – 1 new storage tank – Pumps – New ventilation system for the critical UPS room Conversion of the Barn area to house – 2 Electrical rooms (batteries, UPS and switchboards) – An IT room with water cooled racks for IT critical equipment (up to 480kW) – New ventilation system for the legacy critical network area in R-050 and for new electrical rooms Computer Center Consolidation – VD - 100611 - 9

10. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Technical choices - 3 Modification of the telecom area – Extension into adjacent labs – Fire protected walls – New ventilation systems Modification of the technical rooms (basement) – Move of the critical UPS (from S-435 to S-437) – New cooling control (in S-400, S-401 and S-402) Addition of 2 new electrical transformers (adjacent to the existing ones) Addition of 1 new water tank for physics equipment (adjacent to the existing one) Computer Center Consolidation – VD - 100611 - 10

11. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF 3D view from the car park Computer Center Consolidation – VD - 100611 - 11

12. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF 3D view of the ground floor Computer Center Consolidation – VD - 100611 - 12

13. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Ventilation in the Barn Computer Center Consolidation – VD - 100611 - 13

14. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Telecom critical area Computer Center Consolidation – VD - 100611 - 14

15. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF 3D view of the basement Computer Center Consolidation – VD - 100611 - 15

16. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF The new IT room Critical IT equipment only (450kW, room designed for 480kW) Cooling of IT equipment still being discussed, most likely water-cooled racks with passive heat exchangers – No power consumption – Cheap (compared to other cooling solutions) – No fan problems – No cold/hot corridors, cold room – Cooling issues for heterogeneous equipment??? Very likely 90 racks (70cm*110cm*47u), average power density 5kW/rack Computer Center Consolidation – VD - 100611 - 16

17. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Views of a passive rack Computer Center Consolidation – VD - 100611 - 17

18. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Plan of the new IT room (draft) Computer Center Consolidation – VD - 100611 - 18

19. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Important dates Autumn 2009 : serious discussions with EN May 2010 : approval of the project October 2010 : barn ready for dismantling January 2011 : preliminary electrical works May 2011 : start of the main civil engineering works Summer 2011 : MS and IT for the water-cooled racks November 2011 : end of the main civil engineering works November 2011 : start of the main cooling works November 2011 : start of the main electrical works May 2012 : start of the works for the additional water tank July 2012 : end of the main cooling works July 2012 : end of the works for the additional water tank August 2012 : end of the main electrical works October 2012 : end of the works Computer Center Consolidation – VD - 100611 - 19

20. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Work outside the project - 1 Problematic cooling of the Physics equipment – When running on UPS without diesel power : IT physics servers up... without cooling – No easy solution. We need power (500kW) for the pumps and ventilation in R-050 and the Vault (>500kW) – No agreement reached when the consolidation project was approved – However... on-going discussions/investigations and a good chance to power the ventilation units from the UPS Computer Center Consolidation – VD - 100611 - 20

21. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Work outside the project - 2 Improvement of the cooling efficiency – To be able to cool more IT equipment and to save on cooling power – Main steps already achieved Cold corridors (from 2008) Temperature increase of blown air in R-050 from 14C to 16C (September 2010) (More) use of outside fresh air (February 2011, net immediate gain 200kW) Blown air in R-050 from 16C to 18C (2 May 2011) Blown air in R-050 from 18C to 19C (11 May 2011) Computer Center Consolidation – VD - 100611 - 21

22. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Work outside the project - 3 Improvement of the cooling efficiency – Future steps Continue to increase the temperature of the air blown in R- 050 (up to 22? While ensuring air < or = 27C for all IT equipment) Increase the temperature of the Vault and possibly of some UPS rooms. Discussion starting with EN/CV Better understand and improve the air flow from the main ventilation units to the servers in R-050 Continue to improve cold air confinement where relevant Increase the temperature of the chilled water? Computer Center Consolidation – VD - 100611 - 22

23. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Work outside the project - 4 Improvement of the monitoring of the infrastructure of the Computer Centre – Important work already done to better understand the power flows in the CC and store/display them in Lemon... Still some work to do, in particular on the alarms – Work to be done to include temperatures and related alarms in Lemon – Progressive implementation of intelligent power bars in IT racks (end-2011 installations & new IT room) Computer Center Consolidation – VD - 100611 - 23

24. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Video on the outside work To upload Computer Center Consolidation – VD - 100611 - 24

25. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Questions? See also http://cern.ch/it/cc2012http://cern.ch/it/cc2012 Computer Center Consolidation – VD - 100611 - 25