Closing the Gap to Free Cooling in the Data Center Ralph Wescott Data Center Manager May 30, 2012 PNNL-SA-86058 (2)

PNNL A multiprogram research laboratory FY 2011 business volume $1.105B 4,600 staff 10,000 networked devices Unique laboratory equipment Scientific supercomputing PNNL has a unique operating contract with DOE that enables private research. EMSL is a unique user facility.

ISB2/1 Perimeter CRAC Cooling 3,000 ft sq typical perimeter CRAC cooled with 2 x 30 ton water-side economizer CRACs and 4 x 20 ton dry coolers. This layout being established to help enclose the Cold aisle to improve the PUE even further.

CSF/1811 Floorplan 10,000 ft sq data center

Chilled Door Exploded View Source: Motivair Corporation

RDHx with doors closed

RDHx with doors open

Top view of the Rear Door Heat Exchangers in the open position Top view of the Rear Door Heat Exchangers in the open position. Each flow 13.6gpm and remove up to 20kW of heat. Air leaving the doors is room neutral.

CSF/1811 Cooling Flow Cooling Flow and Sensor points, details of sensors on next page along with measurements averaged over 24 hr period on 5/22/2012.

Energy Share Calculations Sensor Readings Description of Readings averaged over 24 hrs on 5/22/2012 1 62.2 kW Chiller/Compressor 1 2 0.0 kW Chiller/Compressor 2 3 53.6 gpm Chilled Water Fow to CRACs 4 64.54F Chilled CRAC Water Return Temp 5 45.62F Chilled CRAC Water Supply Temp 6 3.0 kW Chilled Water Pump 1 7 1.38 kW Chilled Water Pump 2 8 815.69 gpm Condenser (Closed) Loop Water Flow 9 74.0F RDHx Loop Return Temp 10 63.67F RDHx Loop Supply Temp 11 111.88 gpm RDHx Loop Water Flow Rate 12 3.83 kW Condenser (Closed) Loop Pump 1 13 2.4 kW Condenser (Closed) Loop Pump 2 14 1.8 kW RDHx Loop Pump 3 (dedicated) 15 RDHx Loop Pump 4 (dedicated) 16 70.69F Condenser (Closed) Loop Return Temp 17 63.38F Condenser (Closed) Loop Supply Temp 18 58.97F Outside Air Temp (average 24 hrs) 19 22.23 kW Well Pump 1 20 17.8 kW Well Pump 2 21 Well Pump 3 22 21.84 kW Well Pump 4 23 63.83F Condenser Chiller 1 Intake Temp 24 70.64F Condenser Chiller 1 Exit Temp 25 624 gpm Condenser Chiller 1 Cooling Flow 26 n/a Condenser Chiller 2 Intake Temp 27 Condenser Chiller 2 Exit Temp 28 0.0 gpm Condenser Chiller 2 Cooling Flow Readings on the left fed calculations on the right to get the fair share of the cooling costs (in kW) that the RDHx and the CRACs have used. Power consumption by the lights was ignored as trivial for this comparison. Consumption by transformers and UPS’s is included in the measuring points.

Contrast/Comparison ISB2/1 GP Data Center CSF/1811 Computer Lab CRACs 99.3 kW to operate 181.9 kW heat removed Efficiency Ratio = 1.82 ISB2/1 Overall PUE = 1.54 (IT + CRACs)/IT (181.9+99.3)/181.9=1.54 CSF/1811 Computer Lab RDHx 19.5 kW to operate 169.3 kW heat removed Efficiency Ratio = 8.7 CRACs 42.5 kW to operate 148.6 kW heat removed Efficiency Ratio = 3.5 CSF Overall PUE = 1.18 (IT + RDHx + CRAC)/IT (336+19.5+42.5)/336=1.18 Wasted energy from UPS’s and transformers included. Lights are not included. Of note in our application: Groundwater cooled CRACs are 2 times more efficient than air-cooled CRACs RDHx are 2.5 times more efficient than groundwater cooled CRACs RDHx are 5 times more efficient that air-cooled CRACs Efficiency Ratio = How many kW of heat removed per kW of energy applied.

Capturing and tracking environmental data will become more important to maximize efficiency and help in choosing the most effective cooling techniques for your locality. This is an example of data available from modern building management software which can be used to feed DCIM Applications to help measure Data Center efficiency and evaluate energy efficient improvements. We are enclosing our cold aisle in ISB2/1 (June 2012) and will be able to immediately compare cooling costs before and after to give us an immediate idea of the ROI for this project. This delta will give us our carbon savings in a verifiable and easy to calculate manner.

DCIM Software in ISB2/1 Random screen shot of ISB2/1 on 5/23/2012. Cooling kW and PUE graphs are saw-toothed due to compressors going on and off. Using air as a heat sink, efficiency is better in the winter, worse in the summer. We are updating this chart to reflect the recent removal of a 300 kVA transformer and older UPS’s which eliminated the 19.98 kW energy loss due to that equipment.

Final Thoughts Groundwater Benefits: Immediate plans: Very efficient for CRACs and especially RDHx Avoid super-hot data center Avoid free-air contamination and security issues Immediate plans: Centralize building controls data base for ease of access Expand DCIM applications to CSF (June 2012) Increase metering to lights and CRACs for CSF Explore “touch cooling” for CPU’s and RAM Uses existing piping installed for RDHx at much reduced flow Returns 120F to building operators With current flow rates, achieve six times the kW cooled PNNL is standardizing on Alerton Controls for all new buildings and converting over for older buildings. This will enable a much more robust SQL centralized database for this information. Current RDHx rise is 10F, with touch cooling rising to 120F, same flow will yield 6 times the heat dissipated and the option of reusing that 120F water for per-heating labs during winter. Wet labs require 6 full air exchanges per hour by code, very expensive to heat/cool that much air all day long.

Ralph Wescott ralph.wescott@pnnl.gov 509-372-6901