Cooling: Best Practices and Economizers 4/21/2017 Cooling: Best Practices and Economizers Randall Poet A C Systems

Ideal Situation Server Load = CRAC Capacity Server Airflow = CRAC Airflow

Agenda The Conventional Approach – Base Case Raise the Return / Supply Air Temp Contain the Cold Aisle Add Intelligent Control The Set-up Summary

Start with Best Practices Hot aisle / Cold aisle Blanking panels in racks Blanking panels between racks Blanking Panels between racks and floor Cable cut-outs covered Relatively clean underfloor area Proper location of CRAC units Vapor barrier around space

Base Case – 75F Return Air Control 20 Racks 8 kW each 35F delta T 4 CRACs (N+1) 54 kW each 21F delta T design

Base Case – 75F Return Air Control 20 Racks 8 kW each 35F delta T 3 CRACs Running 54 kW each 21F delta T design

Base Case Operating Scenario 75F 89F 54F CFM 100% Unit Airflow – 24,000 CFM Rack Airflow – 14,600 CFM Bypass Airflow – 9,400 CFM

The Airflow Pattern

Typical Air Cooled DX Energy Consumption Condenser Fan Motors Evap Fan Motor Compressors

Typical Energy Consumption System kW 75F RA Compressors 35.4 Evap Fans 8.7 Condenser Fans 5.4 Total 49.5 Condenser Fan Motors Evap Fan Motor Compressors Compressors run at full capacity Fans run at full speed

Agenda The Conventional Approach – Base Case Raise the Return / Supply Air Temp Contain the Cold Aisle Add Intelligent Control The Set-up Summary

New Operating Scenario 85F 99F 64F CFM 100% Unit Airflow – 24,000 CFM Rack Airflow – 14,600 CFM Bypass Airflow – 9,400 CFM

Increased Capacity at Full Airflow 4/21/2017 Increased Capacity at Full Airflow

Increased Capacity at Full Airflow 4/21/2017 Increased Capacity at Full Airflow

Operating Systems Comparison System kW 75F RA 85F RA Compressors 35.4 24.9 Evap Fans 8.7 Condenser Fans 5.4 Total 49.5 39.0 Base 78.8% Compressors run at reduced capacity or unloaded Fan motors run at full speed

Sanity Check – 100 dF OAT, 6400’ ASL 4/21/2017 Sanity Check – 100 dF OAT, 6400’ ASL

Sanity Check

Agenda The Conventional Approach – Base Case Raise the Return / Supply Air Temp Contain the Cold Aisle Add Intelligent Control The Set-up Summary

Contain the Cold Aisle Unit Airflow – 21,600 CFM Rack Airflow – 15,800 CFM Bypass Airflow – 5,800 CFM

Operating Systems Comparison System kW 75F RA 85F RA Contained Compressors 35.4 24.9 25.8 Evap Fans 8.7 7.5 Condenser Fans 5.4 Total 49.5 39.0 38.7 Base 78.8% 78.2% At the higher RA temperature, the contained system has very similar operating costs as the non- contained Fan motors run at full speed but at a reduced CFM and HP due to the higher static pressure Compressors run at reduced capacity or unloaded but slightly higher than the non-contained

Why use Containment?? System operating costs are similar Containment partitions and doors cost $$

Hot Spots in Racks due to Wrap-Around

No Leakage into the Cold AIsle

Higher Temperatures without Containment

Containment of Cold Aisle

Issues to Consider Fire Detection / Suppression Installation 4/21/2017 Issues to Consider Fire Detection / Suppression Wide variation between municipalities If local Fire Inspector involved early, typically goes well Curtains usually eliminate this issue Installation Will be site specificy Irregular row length/height, gaps, etc. Site conditions critical, one size does NOT fit all

4/21/2017 Issues To Consider ADA CAC space is for Service Personnel (Section 4.1.1) What about cooling for components in rest of room? Best solution today may be ducted return from hot aisle Perf tiles near other equipment requiring cooling (eg. UPS) 85⁰ Room temperature?

HAC vs CAC Main Purpose of Cooling in Data Center? 4/21/2017 HAC vs CAC Main Purpose of Cooling in Data Center? Cool the equipment Data Centers commonly on raised floor CAC allows current investment to be used HAC typically requires in row cooling So, refrigerant or chilled water and condensate intermingled with IT equipment and racks Which aisle does majority of work take place in? CAC hot aisle likely in 85⁰F range Can use perf tiles in other space HAC hot aisle likely in 100⁰F range

Agenda The Conventional Approach – Base Case Raise the Return / Supply Air Temp Contain the Cold Aisle Add Intelligent Control The Set-up Summary

Closer to Matching the Load to the Cooling 92F 97F 62F CFM 70% Variable Capacity Compressors Variable Speed Fans Intelligent Control Unit Airflow – 16,800 CFM Rack Airflow – 14,600 CFM Bypass Airflow – 2,200 CFM

Operating Systems Comparison System kW 75F RA 85F RA Contained 92F RA Contained & Controlled Compressors 35.4 24.9 25.8 25.5 Evap Fans 8.7 7.5 3.0 Condenser Fans 5.4 Total 49.5 39.0 38.7 33.9 Base 78.8% 78.2% 68.5% The airflows and capacities/loads are more closely matched Fan motors run at reduced speed, CFM and HP based on the demand in the contained area Compressors run at reduced capacity or unloaded but slightly higher than the non-contained

Agenda The Conventional Approach – Base Case Raise the Return / Supply Air Temp Contain the Cold Aisle Add Intelligent Control (Creating SmartAisle) The Set-up Summary slide

Sensor Location Server centric solution, meaning that it focuses on the inlet temperature to the servers Self adapting to environment changes due to server utilization, equipment location changes and outside variables Can adapt to situations with no containment, end containment, and full containment

Rack Sensors Rack Sensors without doors can be mounted on the frame of the racks. Temperature differences were within .5°F

Cold Aisle Sensors Sensors can also be mounted at the top of cold aisles when rack mounting is unavailable

Supply Compensation Lower Supply Compensation is the magical link between controlling sensors The controller evaluates the Rack Sensors and Fan Speed If the fan is operating at 100% and remains above the cold aisle set point Then the supply temperature set point will slowly lower to drive the correct cold aisle temperature Fan Speed 100% Lower Supply

Controller and CRAC Operation Unit ON CW Valve Open % Fan Speed % 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Increase in kW Supply Temperature The increase in fan speed will result in a warmer supply air temperature which is detected by the supply air sensor that will increase cooling to maintain supply air setpoint Cold Aisle Containment iCOM will automatically adjust to changes that result in a temperature increase or decrease Fan Speed Increases So that cold aisle temperature is maintained at customer temperature Setpoint IT Load Increases Rack Temperature Sensor detects inlet rack temperature

Controller and CRAC Operation Advanced freeze protection routine Allow all units to reduce fan speed to 60% Fan speed and compressor capacity (or CW valve) managed for best unit efficiency and performance Multiple remote sensors Controller can use averaged and maximum/minimum values to individually control multiple CRAC systems CRAC systems work as a team All remote sensors used Increase capacity of other applicable adjacent units if one is at maximum and unable to handle the load Automatically adjust for units not in service

Agenda The Conventional Approach – Base Case Raise the Return / Supply Air Temp Contain the Cold Aisle Add Intelligent Control The Set-up Summary

Issues to Consider Fire Detection / Suppression Installation Wide variation between municipalities If local Fire Inspector involved early, typically goes well Curtain eliminates this issue Installation Will be a local responsibility You need to develop relationship with local Cable Contractor or similar company for installation Irregular row length/height Site Survey critical Most expensive component

Issues To Consider ADA Consultant raised issue at BAIS project Successfully defended CAC space is for Service Personnel (Section 4.1.1) What about cooling for components in rest of room? Best solution today is ducted return from hot aisle perf tile by IT component 85⁰ Room temperature

Issues To Consider Purpose of Cooling in Data Center? Control equipment inlet temps Data Centers often on raised floor CAC allows current investment to be used Which aisle does majority of work take place in? CAC hot aisle likely in 85⁰F range Could use perf tile in other space HAC hot aisle likely in 100⁰F range

Summary Best practices are a must if improved efficiency is a goal Running warmer temperatures in the space will improve the cooling system operating efficiency Containment improves the availability of the servers by eliminating hot spots Intelligently controlling fan speed and compressor capacity “balances” the system to operate at it’s most efficient level

Summary Efficiency is the story, not containment 4/21/2017 Summary Efficiency is the story, not containment Containment can be done in several ways, none necessarily fit all situations Each critical space is unique and merits individual planning Dynamic Control is (currently )the final element

…Shifting on to Economizers and Equipment Considerations…

90.1 Economizer Map

Definitions of Cooling Efficiency EER (Energy Efficiency Ratio) Total Cooling Capacity (BTUH)/Total power input (watts) Full-load value on 95F design day Commercial return air conditions of 80F/50% SCOP (Sensible Coefficient of Performance) Sensible Cooling Capacity (kW)/Total power input (kW) Data Center return air conditions of 75F/45% These are two common terms to describe the energy efficiency of cooling units: EER (energy efficiency ratio) is expressed at Total cooling BTUhs divided by the total power input (in watts). It is also based on one specific operating point, that the cooling unit is likely never to operate because of cooling redundancy that is usually designed into the data center. It is also based on 80F air temperature back to the cooling unit. SCOP (sensible coefficient of performance) is a net cooling capacity, with fan motor heat removed, and is expressed as kW cooling divided by kW power input. Again, it is based on a full-load condition on a 95F day. Data center conditions of 75F are used. ASHRAE 90.1, reference in most state energy codes, now includes minimum requirements for data center coolin.

ASCOP – Efficiency Metric Annualized Sensible Coefficient of Performance (location specific) Bin efficiency x bin hours/total annual hours Factors hours at each efficiency operating point Factors part-load efficiency Factors economizer hours

Single System – Outside Air Economizer

Large or Multiple Systems – Outside Air Economizer

Single System with Glycol Economizer

Precooler on Chilled Water System

Chilled Water with Cooling Tower Economizer

What if we could do a refrigerant based economizer cycle?

Energy Consuming Components Air-Cooled System Compressor energy reduction is low-hanging fruit

Digital Scroll Technology Continuous variable capacity compressor technology without inverter drive from 10% to 100% of capacity In commercial use since 2004 Available in single compressor and tandem compressor configurations High reliability versus Inverter drive systems No electrical harmonics introduced

EC fans in unit

EC fans in raised floor

EconoPhase Pumped Refrigerant Economizer Outdoor Ambient 95°F KW @ 70% Load 24.1 Summer operation EconoPhase Condenser Check Valve Refrigerant Pumps OFF Solenoid Valve Check Valve Evaporator Check Valve Circuit 2 Electronic expansion valve Circuit 1 Compressors ON

EconoPhase – Partial Economization Outdoor Ambient 65°F KW @ 70% Load 15.1 Fall / Evening operation EconoPhase Condenser Check Valve ON Refrigerant Pump Solenoid Valve OFF Check Valve As ambient temperatures get cooler in the evenings or with the change of seasons, the Econophase will act as a “pre-cooler” to reduce compressor operation. In this case one circuit runs on compressor operation and the second circuit runs on the refrigerant pump. This is where the staged A coil comes into play. It pre-cools the refrigerant to reduce the compressor workload. Note in this case my system energy draw has decreased by 37% to 15 kW. ON Evaporator Check Valve Circuit 2 Electronic expansion valve OFF Circuit 1 Compressor

EconoPhase - Full Economization Outdoor Ambient 25°F KW @ 70% Load 3.7 Winter operation EconoPhase Condenser Check Valve ON ON Solenoid Valve Refrigerant Pump Check Valve In full economization mode – when temperatures are below 35 degrees – both compressors are off and both refrigerant pumps circulate the refrigerant through their respective circuits. Energy consumption is now only 3.7 kW for the entire system! And I get full capacity from the system. This is an 85% reduction in the total system energy draw – and that’s because the refrigerant pump draws about 1/20 the energy of a compressor! OFF Evaporator Check Valve Circuit 2 Electronic expansion valve Circuit 1 OFF Compressor

Refrigerant Economizer Technology Electronic Expansion Valve EC Plug Fans Tandem compressors w/ Digital Scroll New Evaporator Coil design Staged “A” Coil Greater surface area Microchannel Condenser Indoor Unit Communicates to Condenser Refrigerant Economizer The DSE is loaded with new technology and Liebert firsts. Its features include electronic expansion valves, ec plug fans, and features new to the precision cooling market: tandem compressors in each circuit with digital scroll compressors, a larger, staged “A” coil optimized to the compressor, and a microchannel condenser that even communicates with the indoor unit for optimized performance.

Advantages of Refrigerant Economizer No additional heat exchangers No dust or contamination concerns No need for added ductwork or structural changes to building for economizer No dampers to maintain Quick changeover between compressor mode and economizer mode No water usage No water freeze or coil freeze concerns

Liebert DSE Full-Load Efficiency SCOP (kW/kW) @ 85F return air Liebert DSE DA125A @ 50% load +115% +92% Liebert DSE DA125A @ 100% load Efficiency Plateau Traditional Refrigeration +45% ASHRAE 90.1 @ 75F

With Refrigerant Economizer Maximum capacity, max airflow 4/21/2017 With Refrigerant Economizer Maximum capacity, max airflow Ambient, ºF

No water use for cooling MERV 13 filters No outside air contamination Designing for LEED Energy efficiency No water use for cooling MERV 13 filters No outside air contamination R-410A refrigerant/lower charge Utility Rebates

Condenser Improvements

Air Cooled Condenser Improvements Microchannel vs. Fin and Tube Variable Speed Fans Control Based on Refrigerant Pressure

Microchannel Coil Aluminum Fins & Tubes Developed & used in Automotive A/C for 25+yrs

Microchannel Coil Advantage 1” Thin coil replaces 2” to 6” fin/tube coil Lower airside ∆P Reduced maximum fan wattage for heat transfer Reduced refirgerant quantity 6” 1”

EC Axial Fan Technology EC (Electronically Commutated) Motor Every fan/motor is designed for variable speed. Superior part load efficiency 3-Phase AC power feed, but efficiency of a DC motor 20% reduction of fan speed = 49% savings in energy

Sound Quiet fan blade design

Sound Level Factors & Options Sound varies with fan speed Fan speed is proportional to % compressor load and outdoor ambient temperature Will you ever hear a lightly loaded system condenser? Even more quiet options Upsizing condenser will keep fan speed in nearly “inaudible” range!

Why do they cool things with air, anyway??

Why indeed? How about pumped refrigerant? Refrigerant piping anifold supplies pumped refrigerant Cam levers activates engagement and disengagement of server Refrigerant connection points at top of the rack on left side. All plumbing runs along left side.

Cooling Architecture Each server is cooled by an individual cold plate Cold plates and insertion mechanism is designed for use of standard 1U servers Circulating refrigerant is the cooling medium Rack Server Heat exchanger & circulation unit Cold Plate To/From Chiller Cold Plate Highly flexible Backed with pressure plate Forms to the server lid

In-Rack Cooling with Pumped Refrigerant The rack is 45U tall by 800mm wide by 1200mm deep. 36 1U server slots with cold plate Three non-cooled server slots for “cool” devices like switches, etc. Can be configured for a cooling capacity of 20kW or 40kW.

4/21/2017 Thank You. Questions?