1. Cooling: Best Practices and Economizers
4/21/2017
Cooling: Best Practices and Economizers
Randall Poet
A C Systems

2. Ideal Situation Server Load = CRAC Capacity
Server Airflow = CRAC Airflow

3. Agenda The Conventional Approach – Base Case
Raise the Return / Supply Air Temp
Contain the Cold Aisle
Add Intelligent Control
The Set-up
Summary

4. Start with Best Practices
Hot aisle / Cold aisle
Blanking panels in racks
Blanking panels between racks
Blanking Panels between racks and floor
Cable cut-outs covered
Relatively clean underfloor area
Proper location of CRAC units
Vapor barrier around space

5. Base Case – 75F Return Air Control
20 Racks
8 kW each
35F delta T
4 CRACs (N+1)
54 kW each
21F delta T design

6. Base Case – 75F Return Air Control
20 Racks
8 kW each
35F delta T
3 CRACs Running
54 kW each
21F delta T design

7. Base Case Operating Scenario
75F
89F
54F
CFM 100%
Unit Airflow – 24,000 CFM
Rack Airflow – 14,600 CFM
Bypass Airflow – 9,400 CFM

8. The Airflow Pattern

9. Typical Air Cooled DX Energy Consumption
Condenser Fan Motors
Evap Fan Motor
Compressors

10. Typical Energy Consumption
System kW
75F RA
Compressors
35.4
Evap Fans
8.7
Condenser Fans
5.4
Total
49.5
Condenser Fan Motors
Evap Fan Motor
Compressors
Compressors run at full capacity
Fans run at full speed

11. Agenda The Conventional Approach – Base Case
Raise the Return / Supply Air Temp
Contain the Cold Aisle
Add Intelligent Control
The Set-up
Summary

12. New Operating Scenario
85F
99F
64F
CFM 100%
Unit Airflow – 24,000 CFM
Rack Airflow – 14,600 CFM
Bypass Airflow – 9,400 CFM

13. Increased Capacity at Full Airflow
4/21/2017
Increased Capacity at Full Airflow

14. Increased Capacity at Full Airflow
4/21/2017
Increased Capacity at Full Airflow

15. Operating Systems Comparison
System kW
75F RA
85F RA
Compressors
35.4
24.9
Evap Fans
8.7
Condenser Fans
5.4
Total
49.5
39.0
Base
78.8%
Compressors run at reduced capacity or unloaded
Fan motors run at full speed

16. Sanity Check – 100 dF OAT, 6400’ ASL
4/21/2017
Sanity Check – 100 dF OAT, 6400’ ASL

17. Sanity Check

18. Agenda The Conventional Approach – Base Case
Raise the Return / Supply Air Temp
Contain the Cold Aisle
Add Intelligent Control
The Set-up
Summary

19. Contain the Cold Aisle Unit Airflow – 21,600 CFM
Rack Airflow – 15,800 CFM
Bypass Airflow – 5,800 CFM

20. Operating Systems Comparison
System kW
75F RA
85F RA
Contained
Compressors
35.4
24.9
25.8
Evap Fans
8.7
7.5
Condenser Fans
5.4
Total
49.5
39.0
38.7
Base
78.8%
78.2%
At the higher RA temperature, the contained system has very similar operating costs as the non- contained
Fan motors run at full speed but at a reduced CFM and HP due to the higher static pressure
Compressors run at reduced capacity or unloaded but slightly higher than the non-contained

21. Why use Containment?? System operating costs are similar
Containment partitions and doors cost $$

22. Hot Spots in Racks due to Wrap-Around

23. No Leakage into the Cold AIsle

24. Higher Temperatures without Containment

25. Containment of Cold Aisle

26. Issues to Consider Fire Detection / Suppression Installation
4/21/2017
Issues to Consider
Fire Detection / Suppression
Wide variation between municipalities
If local Fire Inspector involved early, typically goes well
Curtains usually eliminate this issue
Installation
Will be site specificy
Irregular row length/height, gaps, etc.
Site conditions critical, one size does NOT fit all

27. 4/21/2017
Issues To Consider
ADA
CAC space is for Service Personnel (Section 4.1.1)
What about cooling for components in rest of room?
Best solution today may be ducted return from hot aisle
Perf tiles near other equipment requiring cooling (eg. UPS)
85⁰ Room temperature?

28. HAC vs CAC Main Purpose of Cooling in Data Center?
4/21/2017
HAC vs CAC
Main Purpose of Cooling in Data Center?
Cool the equipment
Data Centers commonly on raised floor
CAC allows current investment to be used
HAC typically requires in row cooling
So, refrigerant or chilled water and condensate intermingled with IT equipment and racks
Which aisle does majority of work take place in?
CAC hot aisle likely in 85⁰F range
Can use perf tiles in other space
HAC hot aisle likely in 100⁰F range

29. Agenda The Conventional Approach – Base Case
Raise the Return / Supply Air Temp
Contain the Cold Aisle
Add Intelligent Control
The Set-up
Summary

30. Closer to Matching the Load to the Cooling
92F
97F
62F
CFM 70%
Variable Capacity Compressors
Variable Speed Fans
Intelligent Control
Unit Airflow – 16,800 CFM
Rack Airflow – 14,600 CFM
Bypass Airflow – 2,200 CFM

31. Operating Systems Comparison
System kW
75F RA
85F RA
Contained
92F RA Contained & Controlled
Compressors
35.4
24.9
25.8
25.5
Evap Fans
8.7
7.5
3.0
Condenser Fans
5.4
Total
49.5
39.0
38.7
33.9
Base
78.8%
78.2%
68.5%
The airflows and capacities/loads are more closely matched
Fan motors run at reduced speed, CFM and HP based on the demand in the contained area
Compressors run at reduced capacity or unloaded but slightly higher than the non-contained

32. Agenda The Conventional Approach – Base Case
Raise the Return / Supply Air Temp
Contain the Cold Aisle
Add Intelligent Control (Creating SmartAisle)
The Set-up
Summary
slide

33. Sensor Location
Server centric solution, meaning that it focuses on the inlet temperature to the servers
Self adapting to environment changes due to server utilization, equipment location changes and outside variables
Can adapt to situations with no containment, end containment, and full containment

34. Rack Sensors
Rack Sensors without doors can be mounted on the frame of the racks.
Temperature differences were within .5°F

35. Cold Aisle Sensors
Sensors can also be mounted at the top of cold aisles when rack mounting is unavailable

36. Supply Compensation Lower Supply
Compensation is the magical link between controlling sensors
The controller evaluates the Rack Sensors and Fan Speed
If the fan is operating at 100% and remains above the cold aisle set point
Then the supply temperature set point will slowly lower to drive the correct cold aisle temperature
Fan Speed
100%
Lower Supply

37. Controller and CRAC Operation
Unit ON
CW Valve Open %
Fan Speed %
100%
90%
80%
70%
60%
50%
40%
30%
20%
10% 0%
Increase in kW
Supply Temperature
The increase in fan speed will result in a warmer supply air temperature which is detected by the supply air sensor that will increase cooling to maintain supply air setpoint
Cold Aisle Containment
iCOM will automatically adjust to changes that result in a temperature increase or decrease
Fan Speed Increases
So that cold aisle temperature is maintained at customer temperature Setpoint
IT Load Increases
Rack Temperature Sensor detects inlet rack temperature

38. Controller and CRAC Operation
Advanced freeze protection routine
Allow all units to reduce fan speed to 60%
Fan speed and compressor capacity (or CW valve) managed for best unit efficiency and performance
Multiple remote sensors
Controller can use averaged and maximum/minimum values to individually control multiple CRAC systems
CRAC systems work as a team
All remote sensors used
Increase capacity of other applicable adjacent units if one is at maximum and unable to handle the load
Automatically adjust for units not in service

39. Agenda The Conventional Approach – Base Case
Raise the Return / Supply Air Temp
Contain the Cold Aisle
Add Intelligent Control
The Set-up
Summary

40. Issues to Consider Fire Detection / Suppression Installation
Wide variation between municipalities
If local Fire Inspector involved early, typically goes well
Curtain eliminates this issue
Installation
Will be a local responsibility
You need to develop relationship with local Cable Contractor or similar company for installation
Irregular row length/height
Site Survey critical
Most expensive component

41. Issues To Consider
ADA
Consultant raised issue at BAIS project
Successfully defended CAC space is for Service Personnel (Section 4.1.1)
What about cooling for components in rest of room?
Best solution today is ducted return from hot aisle
perf tile by IT component
85⁰ Room temperature

42. Issues To Consider Purpose of Cooling in Data Center?
Control equipment inlet temps
Data Centers often on raised floor
CAC allows current investment to be used
Which aisle does majority of work take place in?
CAC hot aisle likely in 85⁰F range
Could use perf tile in other space
HAC hot aisle likely in 100⁰F range

43. Summary Best practices are a must if improved efficiency is a goal
Running warmer temperatures in the space will improve the cooling system operating efficiency
Containment improves the availability of the servers by eliminating hot spots
Intelligently controlling fan speed and compressor capacity “balances” the system to operate at it’s most efficient level

44. Summary Efficiency is the story, not containment
4/21/2017
Summary
Efficiency is the story, not containment
Containment can be done in several ways, none necessarily fit all situations
Each critical space is unique and merits individual planning
Dynamic Control is (currently )the final element

45. …Shifting on to Economizers and Equipment Considerations…

46. 90.1 Economizer Map

47. Definitions of Cooling Efficiency
EER (Energy Efficiency Ratio)
Total Cooling Capacity (BTUH)/Total power input (watts)
Full-load value on 95F design day
Commercial return air conditions of 80F/50%
SCOP (Sensible Coefficient of Performance)
Sensible Cooling Capacity (kW)/Total power input (kW)
Data Center return air conditions of 75F/45%
These are two common terms to describe the energy efficiency of cooling units:
EER (energy efficiency ratio) is expressed at Total cooling BTUhs divided by the total power input (in watts). It is also based on one specific operating point, that the cooling unit is likely never to operate because of cooling redundancy that is usually designed into the data center. It is also based on 80F air temperature back to the cooling unit.
SCOP (sensible coefficient of performance) is a net cooling capacity, with fan motor heat removed, and is expressed as kW cooling divided by kW power input. Again, it is based on a full-load condition on a 95F day. Data center conditions of 75F are used. ASHRAE 90.1, reference in most state energy codes, now includes minimum requirements for data center coolin.

48. ASCOP – Efficiency Metric
Annualized Sensible Coefficient of Performance (location specific)
Bin efficiency x bin hours/total annual hours
Factors hours at each efficiency operating point
Factors part-load efficiency
Factors economizer hours

49. Single System – Outside Air Economizer

50. Large or Multiple Systems – Outside Air Economizer

51. Single System with Glycol Economizer

52. Precooler on Chilled Water System

53. Chilled Water with Cooling Tower Economizer

54. What if we could do a refrigerant based economizer cycle?

55. Energy Consuming Components Air-Cooled System
Compressor energy reduction is low-hanging fruit

56. Digital Scroll Technology
Continuous variable capacity compressor technology without inverter drive from 10% to 100% of capacity
In commercial use since 2004
Available in single compressor and tandem compressor configurations
High reliability versus Inverter drive systems
No electrical harmonics introduced

57. EC fans in unit

58. EC fans in raised floor

59. EconoPhase Pumped Refrigerant Economizer
Outdoor Ambient 95°F
70% Load 24.1
Summer operation
EconoPhase
Condenser
Check Valve
Refrigerant Pumps OFF
Solenoid Valve
Check Valve
Evaporator
Check Valve
Circuit 2
Electronic expansion valve
Circuit 1
Compressors ON

60. EconoPhase – Partial Economization
Outdoor Ambient 65°F
70% Load 15.1
Fall / Evening operation
EconoPhase
Condenser
Check Valve ON
Refrigerant Pump
Solenoid Valve
OFF
Check Valve
As ambient temperatures get cooler in the evenings or with the change of seasons, the Econophase will act as a “pre-cooler” to reduce compressor operation. In this case one circuit runs on compressor operation and the second circuit runs on the refrigerant pump. This is where the staged A coil comes into play. It pre-cools the refrigerant to reduce the compressor workload.
Note in this case my system energy draw has decreased by 37% to 15 kW. ON
Evaporator
Check Valve
Circuit 2
Electronic expansion valve
OFF
Circuit 1
Compressor

61. EconoPhase - Full Economization
Outdoor Ambient 25°F
70% Load 3.7
Winter operation
EconoPhase
Condenser
Check Valve ON ON
Solenoid Valve
Refrigerant Pump
Check Valve
In full economization mode – when temperatures are below 35 degrees – both compressors are off and both refrigerant pumps circulate the refrigerant through their respective circuits. Energy consumption is now only 3.7 kW for the entire system! And I get full capacity from the system. This is an 85% reduction in the total system energy draw – and that’s because the refrigerant pump draws about 1/20 the energy of a compressor!
OFF
Evaporator
Check Valve
Circuit 2
Electronic expansion valve
Circuit 1
OFF
Compressor

62. Refrigerant Economizer
Technology
Electronic Expansion Valve
EC Plug Fans
Tandem compressors w/ Digital Scroll
New Evaporator Coil design
Staged “A” Coil
Greater surface area
Microchannel Condenser
Indoor Unit Communicates to Condenser
Refrigerant Economizer
The DSE is loaded with new technology and Liebert firsts.
Its features include electronic expansion valves, ec plug fans, and features new to the precision cooling market: tandem compressors in each circuit with digital scroll compressors, a larger, staged “A” coil optimized to the compressor, and a microchannel condenser that even communicates with the indoor unit for optimized performance.

63. Advantages of Refrigerant Economizer
No additional heat exchangers
No dust or contamination concerns
No need for added ductwork or structural changes to building for economizer
No dampers to maintain
Quick changeover between compressor mode and economizer mode
No water usage
No water freeze or coil freeze concerns

64. Liebert DSE Full-Load Efficiency SCOP (kW/kW) @ 85F return air
Liebert DSE 50% load
+115%
+92%
Liebert DSE 100% load
Efficiency Plateau
Traditional Refrigeration
+45%
ASHRAE 75F

65. With Refrigerant Economizer Maximum capacity, max airflow
4/21/2017
With Refrigerant Economizer Maximum capacity, max airflow
Ambient, ºF

66. No water use for cooling MERV 13 filters No outside air contamination
Designing for LEED
Energy efficiency
No water use for cooling
MERV 13 filters
No outside air contamination
R-410A refrigerant/lower charge
Utility Rebates

67. Condenser Improvements

68. Air Cooled Condenser Improvements
Microchannel vs. Fin and Tube
Variable Speed Fans
Control Based on Refrigerant Pressure

69. Microchannel Coil Aluminum Fins & Tubes
Developed & used in Automotive A/C for 25+yrs

70. Microchannel Coil Advantage
1” Thin coil replaces 2” to 6” fin/tube coil
Lower airside ∆P
Reduced maximum fan wattage for heat transfer
Reduced refirgerant quantity 6” 1”

71. EC Axial Fan Technology
EC (Electronically Commutated) Motor
Every fan/motor is designed for variable speed.
Superior part load efficiency
3-Phase AC power feed, but efficiency of a DC motor
20% reduction of fan speed = 49% savings in energy

72. Sound
Quiet fan blade design

73. Sound Level Factors & Options
Sound varies with fan speed
Fan speed is proportional to % compressor load and outdoor ambient temperature
Will you ever hear a lightly loaded system condenser?
Even more quiet options
Upsizing condenser will keep fan speed in nearly “inaudible” range!

74. Why do they cool things with air, anyway??

75. Why indeed? How about pumped refrigerant?
Refrigerant piping anifold supplies
pumped refrigerant
Cam levers activates engagement
and disengagement of server
Refrigerant connection points
at top of the rack on left side.
All plumbing runs along left side.

76. Cooling Architecture Each server is cooled by an individual cold plate
Cold plates and insertion mechanism is designed for use of standard 1U servers
Circulating refrigerant is the cooling medium
Rack
Server
Heat exchanger
&
circulation unit
Cold Plate
To/From Chiller
Cold Plate
Highly flexible
Backed with pressure plate
Forms to the server lid

77. In-Rack Cooling with Pumped Refrigerant
The rack is 45U tall by 800mm wide by 1200mm deep.
36 1U server slots with cold plate
Three non-cooled server slots for “cool” devices like switches, etc.
Can be configured for a cooling capacity of 20kW or 40kW.

78. 4/21/2017
Thank You.
Questions?