OCP Cold Plate Work Session Jessica Gullbrand Cold Plate Work Stream Lead Advanced Cooling Solutions

ACS Cold Plate Focus Focus group working on ACS Cold Plate Requirements Technology Cooling System (TCS) fluid loop - Cooling loop from CDU through rack manifold and IT equipment and back Provide information/education, classification, and minimal requirement for OCP Authors: - Jessica Gullbrand, Intel - Nigel Gore, Iceotope - Elizabeth Langer, CPC https://docs.google.com/document/d/1f7supR2NAMiCMZVeT9mmS4-DogFta-cG_He8HRBWsM8/edit

ACS Cold Plate Requirements Focus Technology Cooling System (TCS) Examples In-Rack CDU Row Level CDU Virtual CDU Use the Facility Cooling Liquid in TCS loop Figure by Intel, C. Winkel

ACS Cold Plate Definition Hybrid Cooling – both liquid and air cooling IT Equipment/Rack (with or without rear-door heat exchanger, RDHX) Full Liquid Cooling – all heat is rejected to liquid Rack/IT Equipment (with or without RDHX)

ACS Cold Plate Definition Cold Plates – Heat exchangers or heat sinks with internal tubing or channels to allow cooling liquid to flow through Coolant Distribution Unit (CDU) – Circulates the cooling liquid to the IT equipment and transfer the heat to the facility cooling liquid Rack Manifold – Distributes cooling liquid within the rack Quick Disconnect Couplings – Disconnects the IT equipment or its components from the TCS loop for serviceability

Cooling Liquid Considerations Cooling Liquids Included (Single- and Two-Phase Liquids): Water with additives Glycol based liquids Dielectric liquids Refrigerants Material compatibility required between wetted materials and cooling liquid What cooling liquids are covered/considered in DCF?

Liquid Cooling Requirements Pressure Safety Requirements The liquid cooling loop, its ingredients, and connections must be tested to the highest pressure of the safety standards: The IEC 60950-1 For EU and US, this standard is outgoing and is expected to be withdrawn on December 20, 2020 in favor of the IEC 62368-1 standard. The IEC 62368-1 is the new safety standard requiring leak tests at 3x under normal operating pressure and 2x under abnormal and single fault conditions ASME B31.3 “Process Piping” (2018) requires leak test at the 1.5x design pressure for interconnecting piping What pressure requirements are being considered by DCF? Standards valid on facility level? On TCS level?

Liquid Cooling Classifications IT Equipment Classifications Hybrid Basic: CPU/GPU cold plates Hybrid Intermediate: CPU/GPU & DIMMs cold plates Hybrid Advanced: CPU/GPU & DIMMs & Additional (specify) cold plates Full Liquid: cold plates on nearly all components

Liquid Cooling Classifications Liquid Rack Classifications Hybrid Liquid Cooled Ready Rack without RDHX* Hybrid Liquid Cooled Ready Rack with RDHX* Full Liquid Rack Cooling without RDHX* Full Liquid Rack Cooling with RDHX* *RDHX = Read-Door Heat Exchanger

Performance Metric

Performance Metric PUE versus TUE for liquid cooling PUE = Power usage effectiveness TUE = Total PUE, “PUE” type metric ITUE = PUE for the IT equipment

Performance Metric https://eehpcwg.llnl.gov/assets/as_tue_a_new_look_at_pue.pdf

Performance Metric

Performance Metric

Performance Metric

Performance Metric g i

Performance Metric

Performance Metric

Future Topics of Importance Manifold Design Connector Specification Leakage Detection & Mitigation Performance Metrics Testing/Metrology/Requirements

Backup

ACS Cold Plate Purpose Purpose: Generate an open specification and supporting documents focusing on standardization and definitions of liquid cooled solutions without preventing innovation Note: Cold Plate work stream is focusing on liquid cooled solutions using cold plates

Liquid Cooling Requirements Certification Markings Each liquid cooled technology needs to comply with all certification regulations, which are valid for the geographic location where the liquid cooled system is used. Different regions have different requirements. Examples: UL and FCC markings in the US CE certification in Europe

Cooling Liquid Considerations Water with Additives Water with Additives Pros Cons Good heat transfer properties (high conductivity and specific heat) Additives needed for reducing corrosion risk and bacterial growth Ongoing regular maintenance

Cooling Liquid Considerations Glycol Based Liquids Glycol Based Liquids Pros Cons >25% glycol no bacterial growth Changes to viscosity => changes pump power conditions including power needed Improved maintenance schedule Lower conductivity and specific heat with increasing glycol level

Cooling Liquid Considerations Dielectric Liquids Dielectric Liquids Pros Cons No short circuit of electronics during potential leaks Adds liquid weight Higher Global Warming Potential (GWP) Higher cost Limited supplier availability Potential for flow instabilities and maldistribution in micro-channel cold plate using 2-phase

Cooling Liquid Considerations Refrigerants Refrigerants Pros Cons Increase in thermal performance compared to single-phase Environmental - Ozone Depletion Potential (ODP) and Global Warming Potential (GWP) Potential for no and lower pumping power compared to single-phase Potential loss of liquid under maintenance conditions Specifically newer refrigerants can have lower GWP than dielectric 2-phase liquid Higher operating pressure than single-phase Can be inert: nontoxic, nonflammable, nonelectrically conductive Potential for flow instabilities and maldistribution in micro-channel cold plate using 2-phase Specialist knowledge required for commissioning, handling, and maintenance