The CDCE BNL HEPIX – LBL October 28, 2009 Tony Chan - BNL

Background Rapid growth in the last few years caused space, power and cooling problems Rapid growth in the last few years caused space, power and cooling problems Increasing capacity for RHIC/ATLAS and other activities cannot be accommodated with current facility infrastructure Increasing capacity for RHIC/ATLAS and other activities cannot be accommodated with current facility infrastructure Search for additional data center space began in 2007 Search for additional data center space began in 2007 Update of talk originally given at HEPIX in St. Louis (Nov. 2007) Update of talk originally given at HEPIX in St. Louis (Nov. 2007)

Vital Statistics Currently housing 165 racks of equipment (disk storage, cpu, network, etc) + 9 robotic silos Currently housing 165 racks of equipment (disk storage, cpu, network, etc) + 9 robotic silos Approximately 35 PB of tape storage, 9 PB of disk storage capacity and 10,200 computing cores Approximately 35 PB of tape storage, 9 PB of disk storage capacity and 10,200 computing cores Average power usage ~ 650 kW (~60% of maximum UPS capacity) with peak load ~ 790 kW Average power usage ~ 650 kW (~60% of maximum UPS capacity) with peak load ~ 790 kW Cooling capacity for a maximum of ~1000 kW Cooling capacity for a maximum of ~1000 kW

The Growth of Computing

Total Distributed Storage Capacity

Evolution of Space Usage Capacity of old data center Intel dual and quad-core deployed

Evolution of Power Usage Existing UPS Capacity

The Search For Solutions (1) Engaged Lab management – Spring 2007 Engaged Lab management – Spring 2007 Discussion on possible solutions – Summer 2007 Discussion on possible solutions – Summer 2007 CostCost TimeTime LocationLocation Recommendation to Lab management – Fall 2007/Winter 2008 Recommendation to Lab management – Fall 2007/Winter 2008 Two-phase solution to meet cost and time constraintsTwo-phase solution to meet cost and time constraints Identify and renovate existing space to meet short-term requirementsIdentify and renovate existing space to meet short-term requirements New building to meet long-term requirementsNew building to meet long-term requirements Funding for two construction projects approved – Spring 2008 Funding for two construction projects approved – Spring 2008

The Search for Solutions (2) Renovate existing floor space (US $0.6 million) Renovate existing floor space (US $0.6 million) Tender award – April 2008Tender award – April 2008 Renovations begin – June 2008Renovations begin – June 2008 Renovations end -- October 2008Renovations end -- October 2008 Occupancy – November 2008Occupancy – November 2008 New building (US $5 million) New building (US $5 million) Finalize design – May 2008Finalize design – May 2008 Tender award – June 2008Tender award – June 2008 Construction starts – August 2008Construction starts – August 2008 Construction ends – August 2009Construction ends – August 2009 Occupancy – October 2009Occupancy – October 2009 From first proposal to occupancy took 2½ years From first proposal to occupancy took 2½ years

Facility Development Timeline Recent Past ( ) Recent Past ( ) More efficient use of facility resourcesMore efficient use of facility resources Supplemental cooling system in existing facilitySupplemental cooling system in existing facility Near-Term (2008 to present) Near-Term (2008 to present) Renovation of (2000 ft 2 ) 185 m 2 of unused floor with 300 kW of powerRenovation of (2000 ft 2 ) 185 m 2 of unused floor with 300 kW of power New building with (6600 ft 2 ) 622 m 2 and 1.0 MW of powerNew building with (6600 ft 2 ) 622 m 2 and 1.0 MW of power Mission-specific facility (redundant cooling, deep raised floors, etc)Mission-specific facility (redundant cooling, deep raised floors, etc) Room for ~ 150 racks and 7 robotic silosRoom for ~ 150 racks and 7 robotic silos Long-Term (2017 and beyond) Long-Term (2017 and beyond) New BNL data center with ft 2 (2300 m 2 ) after 2018New BNL data center with ft 2 (2300 m 2 ) after 2018

Rack-Top Cooling Units

Rear Door Heat Exchanger

Data Center Layout in 2007

Data Center Layout in 2009

DATA CENTER EXPANSION – PART 1 September 12, 2008

Data Center Expansion – Part 1 July 15, 2009

Data Center Expansion – Part 2 January 14, 2009

Data Center Expansion – Part 2 October 10, 2009

Data Center Expansion – Part 2 October 10, 2009

Data Center Expansion – Part 2 October 10, 2009

New Data Center Layout

Where We Are Today (1) Data Center Expansion (part 1) is similar to existing facility Data Center Expansion (part 1) is similar to existing facility 12-in (30.48 cm) raised floor12-in (30.48 cm) raised floor Redundant cooling capacityRedundant cooling capacity No support for racks > 10 kWNo support for racks > 10 kW No support for supplemental coolingNo support for supplemental cooling Cable trays for power and networkCable trays for power and network Data Center Expansion (part 2) was designed for high density equipment Data Center Expansion (part 2) was designed for high density equipment 30-in (76.2 cm) raised floor30-in (76.2 cm) raised floor Support for racks > 10 kW (blades, half-depths, etc)Support for racks > 10 kW (blades, half-depths, etc) Redundant cooling capacityRedundant cooling capacity Support for racks > 2,500 lbs (1,135 kg)Support for racks > 2,500 lbs (1,135 kg) 13-ft ceiling (4 m) for high-profile racks13-ft ceiling (4 m) for high-profile racks Cable trays for power and networkCable trays for power and network Support for supplemental coolingSupport for supplemental cooling Environmentally-friendly buildingEnvironmentally-friendly building

Where We Are Today (2) Facility expanded from 5000 ft 2 (465 m 2 ) to 13,600 ft 2 (1260 m 2 ) of floor space Facility expanded from 5000 ft 2 (465 m 2 ) to 13,600 ft 2 (1260 m 2 ) of floor space Equipment capacity Equipment capacity from ~150 to ~300 racksfrom ~150 to ~300 racks from 6 to 13 robotic silosfrom 6 to 13 robotic silos Infrastructure support Infrastructure support from ~1 to ~2.0 MW of UPS-backed power (up to 4 MW capacity)from ~1 to ~2.0 MW of UPS-backed power (up to 4 MW capacity) Cooling capacity grew from ~1 to ~ 2 MW (up to 4 MW capacity)Cooling capacity grew from ~1 to ~ 2 MW (up to 4 MW capacity) 3 robotic silos and 6 racks of worker nodes first occupants of CDCE (October 2009) 3 robotic silos and 6 racks of worker nodes first occupants of CDCE (October 2009) Is this sufficient until 2018? Is this sufficient until 2018?

Unresolved Issues Insufficient funds to: Insufficient funds to: add 2 nd flywheel UPS for CDCEadd 2 nd flywheel UPS for CDCE diesel generator to support additional flywheel UPS unitsdiesel generator to support additional flywheel UPS units install additional 1 MW of cooling capacity (equipment already purchased)install additional 1 MW of cooling capacity (equipment already purchased) Estimated cost is additional US $2-3 million Estimated cost is additional US $2-3 million Estimate CDCE will exceed 2 MW of UPS power and cooling by 2012Estimate CDCE will exceed 2 MW of UPS power and cooling by 2012 Lead time to approve funds and pre-installation is 12 months  decision by 2011 Lead time to approve funds and pre-installation is 12 months  decision by 2011

Reason for Optimism? Multi-core cpu’s and increasing storage density have helped restrain a feared unsustainable growth in power and space needs Multi-core cpu’s and increasing storage density have helped restrain a feared unsustainable growth in power and space needs Rack counts have not increased at the same rate as computing and storage deployments Rack counts have not increased at the same rate as computing and storage deployments Somewhat hopeful that continued technological gains will further restrain data center growth Somewhat hopeful that continued technological gains will further restrain data center growth

Summary Facility footprint nearly tripled since 2007 Facility footprint nearly tripled since 2007 Applied lessons learned in design of data center expansion (part 2) Applied lessons learned in design of data center expansion (part 2) Must increase cooling efficiency with new technologies Must increase cooling efficiency with new technologies Rack-top cooling unitsRack-top cooling units Rear-door heat exchangerRear-door heat exchanger Hot aisle containmentHot aisle containment Significant increases in power efficiency and technology (power supply, multi-core cpu, etc) is a positive development, but some unresolved issues remain Significant increases in power efficiency and technology (power supply, multi-core cpu, etc) is a positive development, but some unresolved issues remain