Effect of Rack Server Population on Temperatures in Data Centers CEETHERM Data Center Laboratory G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA Rajat Ghosh, Vikneshan Sundaralingam, Yogendra Joshi Collaborators: Pramod kumar, Vaibhav Arghode, Steven Isaacs.

Outline Problem Statement. Methodology –Experiments. –Computational fluid dynamics (CFD) analysis. Results. Conclusions. May 30-June 1 Ghosh, Joshi: ITherm

Problem Statements Characterize rack-level air temperatures for a full-capacity server rack. Estimate effect of rack server population on air temperatures in a data center. Estimate effect of server location on its CPU temperature and average fan speed. May 30-June 1 Ghosh, Joshi: ITherm

Experimental Setup May 30-June 1 Ghosh, Joshi: ITherm Measurement Uncertainty: –Temperature: –Flowrate: –Length: A data center with 10 server racks arranged in a 5x2 architecture. Raised floor plenum supply and overhead plenum return. Alternating cold-aisle/ hot-aisle. CRAC-1 is only active CRAC –

Test Rack May 30-June 1 Ghosh, Joshi: ITherm Server rack (consists of 42 1-U server and headnode) with heterogeneous heat load ranging from 240 W to 0 W.

Containment System May 30-June 1 Ghosh, Joshi: ITherm Isolate airflow into the test rack. Cooling Air from Plenum Test Rack Hot Aisle Containment Cold Aisle Containment Perforated Tile Hot Exhaust Air Exhaust Plane

Thermocouple Grid May 30-June 1 Ghosh, Joshi: ITherm x z x y Steel Frame Thermocouple Tube Grid: 21 T-type copper- constantan thermocouples made from 28 gauge (0.9 mm diameter) wire. Response time: 28 ms x-axis: Parallel to rack width. y-axis: Parallel to tiles. z-axis: parallel to rack height.

Experimental Procedure 1.Deploy the rack-level containment. 2.Vary the server population in the test rack (N=42, 32, 22,12) and measure air temperatures in cold and hot aisles. 3.Vary the location of a server stack and measure temperatures of CPUs and speeds of fans inside servers. May 30-June 1 Ghosh, Joshi: ITherm

Server Population as Parameter May 30-June 1 Ghosh, Joshi: ITherm

Varying Position of a Server Stack May 30-June 1 Ghosh, Joshi: ITherm

CFD Simulation May 30-June 1 Ghosh, Joshi: ITherm x-dimension: 2.46 m y-dimension: 0.60 m Z-dimension: 1.95 m 1-U server: 1m x 0.6 m x0.4 m Server fan: Tile: Velocity inlet with 0.8 m/s to match 639 CFM (0.3 m 3 /s) supply Exhaust: Pressure outlet Server inlet and outlet: Porous jump Grid number: 1.4 millions for grid- independent solution

Transient CRAC Supply Air Temperature May 30-June 1 Ghosh, Joshi: ITherm CRAC-1 has return air temperature control. Variable supply air temperature –Mean= C. Std. Dev.=0.9 0 C.

Cold Aisle Temperature Variation May 30-June 1 Ghosh, Joshi: ITherm With height in the cold aisle, average temperature varies irregularly. N=42

Hot Aisle Temperature Variation May 30-June 1 Ghosh, Joshi: ITherm With height in the hot aisle, average temperature varies irregularly. N=42

CFD-predicted Airflow May 30-June 1 Ghosh, Joshi: ITherm Recirculation in the airflow explains irregular pattern of air temperatures in the cold and hot aisles. N=42

Temperature Difference Variation May 30-June 1 Ghosh, Joshi: ITherm Temperature difference increases with height. N=42

Average Temperature in Exhaust Plane May 30-June 1 Ghosh, Joshi: ITherm Average temperature in the exhaust plane increases with server population.

Effect of Server Population on Temperature Difference May 30-June 1 Ghosh, Joshi: ITherm For all heights, temperature difference increases with server number.

Effect of Containment May 30-June 1 Ghosh, Joshi: ITherm The containment system reduces average temperature in the cold aisle - Blocks hot air recirculating from other parts of the room.

Effect of Server Location May 30-June 1 Ghosh, Joshi: ITherm Keeping server stack near the highest possible location is more energy- efficient practice in this case -Lowest CPU temperature. -Lowest average server fan speed.

Conclusion Rack-level temperature field (average temperatures in cold and hot aisles; and average temperature difference between cold and hot aisles) is characterized for a full capacity server rack (N=42). –Air recirculation thorough the void affects convective temperature field. Rack server population has a significant impact on air temperatures –Temperature difference across the rack increases with the server population in the test rack. Recommended best practice for filling out a server stack in an empty stack –Sever stack should be placed at the highest possible location. May 30-June 1 Ghosh, Joshi: ITherm

Acknowledgement The authors acknowledge support for this work from IBM Corporation, with Dr. Hendrik Hamann as the Technical Monitor. Acknowledgements are also due to the United States Department of Energy as the source of primary funds. Additional support from the National Science Foundation award CRI enabled the acquisition of some of the test equipment utilized. May 30-June 1 Ghosh, Joshi: ITherm

Questions? May 30-June 1 Ghosh, Joshi: ITherm