1. Bernard Boutherin boutherin@lpsc.in2p3.fr 1 Free-Cooling@LPSC Feedback after four years of operation

2. Bernard Boutherin boutherin@lpsc.in2p3.fr 2 Outline Before the free-cooling Description of the solution Experience feedback on the implementation of the free-cooling Risks : quality of the operating environment of the servers in terms of filtration, temperature and humidity Advantages : level of service and availability, working costs

3. Bernard Boutherin boutherin@lpsc.in2p3.fr 3 Before the free-cooling

4. Bernard Boutherin boutherin@lpsc.in2p3.fr 4 2007 : more and more computing needs up to the deadlock Local computing resource needs are increasing LHC (Large Hadron Collider at Cern) computing for ATLAS and ALICE experiment, tier2 LHC computing Grid. Reactor physics and theoretical physics (Lattice QCD) Limits of the air conditioning system has been reach, all the installation must be redesigned Working costs were exploding 8 racks 42U  100kW  60 000€/year Cooling cost can double the electricity bill! Before the free-cooling

5. Bernard Boutherin boutherin@lpsc.in2p3.fr 5 Meteorological environment Before the free-cooling

6. Bernard Boutherin boutherin@lpsc.in2p3.fr 6 Description of the solution

7. Bernard Boutherin boutherin@lpsc.in2p3.fr 7 How does it work Description of the solution Joseph Piarulli

8. Bernard Boutherin boutherin@lpsc.in2p3.fr 8 Operating modes Description of the solution HeatweaveT°outside > T°rear << 1% of the time (never happen) Temperature setup = 25°C Water to air exchanger running T° setup = 25°C regulation by the water flow Recycling register open Inlet register close Output register close Cold mode T° outside <= 13°C 40% of the time Temperature setup = 13°C Water to air exchanger stopped T° setup = 13°C regulation by the recycling register Recycling register partially open Inlet register partially open Output register partially open Normal mode 13° < T° outside <= 25°C 45% of the time Temperature setup = outside temperature Water to air exchanger stopped T° setup = T° outside Recycling register close Inlet register open Output register open Warm mode 25° <T° outside < T rear 15% of the time Temperature setup = 25°C Water to air exchanger running T° setup = 25°C regulation by the water flow Recycling register close Inlet register open Output register open

9. Bernard Boutherin boutherin@lpsc.in2p3.fr 9 Feedback of the operation of the free- cooling, risks and advantages Risks for the servers filtration, temperature and humidity

10. Bernard Boutherin boutherin@lpsc.in2p3.fr 10 Experience feedback Need to control the air fluxes To evacuate the maximum amount of heat to outside Do not mix warm air with cold air which feeds the servers. The temperature in the cold corridor must be uniform => Need to control the air fluxes - Blank panel when a server is missing - Foam strips between racks Feedback

11. Bernard Boutherin boutherin@lpsc.in2p3.fr 11 Risk /air filtration Characteristics of the air filters G4 rather than F7 F7 Pressure drop higher => air flow cost is doubled. Very much more expensive than G4 must be replaced more often. G4 Replaced 4 times a year => 1 000€/an Soiling of servers is more important. Risks : filtration

12. Bernard Boutherin boutherin@lpsc.in2p3.fr 12 Temperature and humidity risks Psychrometric diagram

13. Bernard Boutherin boutherin@lpsc.in2p3.fr 13 Risk analysis / high humidity In which context humidity is high? During summer outside air can be warm and humid. When we cool it its humidity increases. Humidity can reach the saturation at the level of the water-air exchanger. What consequences? When air passes through the servers, its temperature rises while its humidity decreases. There is no risk of condensation in the servers. The high humidity is a risk for oxidation of the servers What actions has to be taken? Recuperate the condensate on the exchanger Risks / Humidity

14. Bernard Boutherin boutherin@lpsc.in2p3.fr 14 Risk analysis / low humidity In which context humidity is low? In winter outside air is cold and dry. When it goes through the servers its temperature grows up and its humidity decreases. Its humidity also decreases due to the effect of recycling warm air. What are the problems with low humidity? Electrostatic risk May lead to unexpected reboot or breakdown due to electrostatic discharges. We run into no problem of this type during four years of operation! Actions which can be taken Ground carefully the racks and the servers and install an antistatic floor. Risks / Humidity

15. Bernard Boutherin boutherin@lpsc.in2p3.fr 15 Measured temperature and humidity / ASHRAE and Manufacturer’s recommendations ASHRAE : American Society of Heating, Refrigerating and Air-Conditioning Engineers

16. Bernard Boutherin boutherin@lpsc.in2p3.fr 16 Feedback of the operation of the free-cooling, risks and advantages Advantages of the solution Level of service and availability Working cost and ecological impact

17. Bernard Boutherin boutherin@lpsc.in2p3.fr 17 Level of service and availability During four years no stop of the production linked with the cooling! This is not by chance but is linked with the robustness of direct air free-cooling At the beginning of the project 2 month (March and April) delay delivery of the chiller : neither cooling nor ventilation. During August 2010 the inlet water pipe was cut by an excavator. => No impact on the production Availability and level of service

18. Bernard Boutherin boutherin@lpsc.in2p3.fr 18 Level of service and availability Consequences of a cooling problem Outside temperature <= 25°C, no impact; 85% of the time Outside temperature between 25°C and 35°C, small impact Inside the manufacturer requirement >>14% of the time Outside temperature >35°C, large impact Out of the requirement bounds, Breakdown rate increases <<1% of the time Availability and level of service Free-cooling/classic cooling : Compared evolution of the temperature in case of a breakdown of the cooling

19. Bernard Boutherin boutherin@lpsc.in2p3.fr 19 Working cooling cost Measured IT consumption 60 kW Ventilation 100 % of the time 2 kW (airflow 16 000 m3/h for 60kW IT) Pumping When using water to air exchanger 15% of the time 6 kW (water flow 20m3/h maximum, 10m3/h average) Average cooling consumption 2 kW x 100% + 6 kW x 15% = 2,9 kW Cooling impact on the Power Usage Effectiveness P.U.E. PUE = (60 + 2.9) / 60 = 1.05 (PUE = 1.6 is considered good) Cooling is only 5% of the IT consumption! Benefits in terms of working cost

20. Bernard Boutherin boutherin@lpsc.in2p3.fr 20 Conclusion Reliability of direct air free-cooling is very good Benefits in terms of working costs and eco-responsibility : 90% reduction in consumption of the cooling justify to assume the risks associated with the use of free-cooling HPC manufacturers requirements less constraining => Same components as for the general public Toward zero energy for cooling datacenters?

21. Bernard Boutherin boutherin@lpsc.in2p3.fr 21 Extra slides Power calculation Costs

22. Bernard Boutherin boutherin@lpsc.in2p3.fr 22 Power of the installation in the mode “direct air free cooling” Power dissipated by an air flow P (W) = W (J) / t (s) W (J) = CcalAir (J / g / °C). m (g). DeltaTair (°C) m (g) = Vair (m3 / h). 1190 (g/m3) / 3600 (s / h). t (s) P = CcalAir(J/g /°C). Vair(m3/h). 1190(g/m3) / 3600 (s/h).DeltaTair (°C) P(W) = Vair (m3 / h) x 0.33 x DeltaTair (°C) At the LPSC DeltaTair 13°C (6 months average) Maximum airflow 23 000 m3/h Power evacuated with “direct air” mode : P = 98 kW

23. Bernard Boutherin boutherin@lpsc.in2p3.fr 23 Power of the installation in the mode “water to air exchanger” Power evacuated as a function of the water flow P = CcalEau (J /g/°C). DeltaTeau (°C). m(g) / t(s) m(g) = Veau (m3/h). 10 6 (g/m3) / 3600(s/h). t(s) P(W) = Veau (m3 / h). 1161. DeltaTeau (°C) At the LPSC Maximum water flow : 20 m3/h Water deltaT 5°C Maximum power evacuated in this mode P = 116 kW

24. Bernard Boutherin boutherin@lpsc.in2p3.fr 24 Cost of installation Chiller 40K€ Air handling unit Input/output/recycling registers Ancillary works 20K€ Ventilation ducts Water intake pipes and plumbing Carpentry (partition) Description of the solution

25. Bernard Boutherin boutherin@lpsc.in2p3.fr 25 Production of cold water Context LPSC is located at the intersection of two rivers. We have the possibility to use groundwater because it was historically used for the cooling of an accelerator. Groundwater is pumped at a temperature of 16°C and is rejected at 21°C. Classic cooling (compressor based) can be used to produce cold water. This cooling will be used only 15% of the time, electricity costs will be divided by 6.6. You will have 85% of the time free for maintenance operations on the cooling system The cooling system will last longer as it is used only 15% of the time.

26. Bernard Boutherin boutherin@lpsc.in2p3.fr 26 Cold water, but not too cold Several ways to achieve a temperature of 20° C, humidity <80% Absolute humidity of air 10 gH2O/kg Cold water >14 ° C no condensation Cold water <14 ° C condensation, air discharge water air warm up by mixing with ambient air => a lot of energy lost (heat of vaporization of water)