1. A New Building Data Center Upgrade capacity and technology step 2011 – May the 4 th– Hepix spring meeting Darmstadt (de) Pascal Trouvé (Facility Manager CC IN2P3)

2. Summary : Latest improvements of the current computing room New computing room - Project scope New Building – New architecture Technical commissioning –Virtual tour –Technical data & figures On going projects –Technical infrastructures phase II –Electrical power supply challenge Future development project 2Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

3. Latest improvements of the computing room POWER –New transformers have been installed (01/2010) : 3 X 1 MVA –Diesel generator (1.1 MVA) –Second electrical low voltage master distribution panel installed (one dedicated for Operation power [racks and cooling], one dedicated to infrastructure power) –Reactive energy compensation panel ( 4 capacitors banks) - savings –4 Power distribution panels replaced + Source Transfer Switch for critical Network racks installed –News batteries farm – sealed lead battery – cell watch system 3Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011v

4. Last improvements of the computing room COOLING : –A third cooling unit (600 kW) has been installed on roof top,1800 kW available –Cooling capacity is now maximum complying with environmental regulation HAZARDS –News fire safety detection system and monitoring ( SSI/CMSI) –New extinguishers fleet and reviewed lay out INFRASTRUCTURE –High density confined racks standardization –InRow air-conditioning racks –PDUs suited to equipment specifications, and monitored 4Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

5. IMPROVMENT INPUTS POWER No power distribution problems occurred over the last year Better batteries back up capacity coupled with the GE COOLING Still not fully redundant (short of 1600 kW) on the existing computing room No cooling issues over the last summer, redundancy will be reached with the new computing room commissioning HAZARDS Better fire safety with modern fire detection system Decrease of battery’s Hydrogen Hazard with sealed technology INFRASTRUCTURE Better global efficiency and sustainability Higher general density 5Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

6. Power exemplification 6 New aeros on roof top 2 New chillers unit Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

7. Confinement and Water cooled racks 7  confinement InRaws cooled racks All the new hardware is installed in “confined racks”, including disks Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

8. Project scope New computing room The identified needs :  ~ 60 users groups increased demand  Meet LHC’s requirements  To meet the planned upgrade of LHC  Important identified request for « astroparticules” research activity 8Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

9. Project scope - Evolution plan – new The new computing room is an extension of the existing computing room (3 MW power + cooling in 2011) Target plan: 2011 50 racks 0.6 MW 2011 50 racks 0.6 MW 2015 125 racks 1.5 MW 2015 125 racks 1.5 MW 2019 216 racks 3.2 MW 2019 216 racks 3.2 MW 9 Total power 3 MW Total power 3 MW Total power 6 MW Total power 6 MW Total power 9 MW Total power 9 MW Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

10. A careful hardware selection 10 IBM iDataplex Intel 5430 LV – 2.66 GHz – 50 W 16 GB IBM iDataplex Intel 5430 LV – 2.66 GHz – 50 W 16 GB DELL 1U 1950b 3 GHz 16 GB DELL 1U 1950b 3 GHz 16 GB Blades DELL E5540 (2.53 Ghz) 48 GB Blades DELL E5540 (2.53 Ghz) 48 GB Dell C6100 X5650 – 2.66 GHz 72 GB Dell C6100 X5650 – 2.66 GHz 72 GB Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

11. New Building – New architecture 11Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 5 th 2011

12. 31 mars 2011 Chiller units Cold Water pumps stations Special construction design – heavy load roof structure Heavy duty Elevator 3 tons load 12Dominique Boutigny Chilled water tank 30 m3

13. Infrastructure design scope 13 Fully confined architecture All the water pipes, electrical distribution and other services are routed from the ceiling Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmsatdt - Mai 4 th 2011

14. New computing room  3 levels of 850 Sq. Meters each [ machine room for 850 sq. meters and 2 technical levels including the roof top] + heavy load elevator  We adopted a simplified “Design and build procedure” (single company contracted) delivered the building within18 months  Due to budget constraints the new computer room start with a limited technical infrastructure, but has a modular design that allowed scalable upgrades  First years, room dedicated to worker nodes only  Chilled water and electricity distribution and back up, technical rooms are designed for the 2019 target value 14Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 5 th 2011

15. The New Data Center Building 15Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011 Setting up of a new building live ! 12 months construction work >= 630 tons of concrete >= 70 tons of steel framework Building work started end of April 2010 Targeted delivery date targeted February 1st 2011 Final delivery date : March 14 th 2011 Technical commissioning : undergoing Go live production ~ Mai 24 th 2011

16. 16 Virtual Tour 24 APC racks : InRaw 20 KW each Confined + workers + 2 Penduit racks with Nexus + switch Cisco To be completed by end of year : 26 APC racks InRaw 20 KW and fully confined area Still 600 sqr meters available …… Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011 Computing room phase I

17. 17 Technical equipment and installation on roof top 2 Chillers units 800 KW each Chilled water tank 30 tons – 30 min back up Insulated piping and electrical tracing Spare place for future upgrades Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 5 th 2011

18. 18 Power Distribution 2 UPS Galaxy 7000 – 600 KVa Main Switch board 2 MW Redundant Air cooling unit Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstdt - Mai 4 th 2011 Fire protection “Vesda” permanent detection Conventional fire detection system level one Sprinkler (automatic water extinguisher system) level 2 Monitoring

19. Power distribution structure 19Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

20. 20 Phase II Additional Equipment & facilities Additional power transformer Third chiller unit 400 KW Second Batteries farm Piping and water pumps Additional power switch boards Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011 Project launched to be delivered by end of October 2011 A “ready to plug in” for 50 more racks units …… On going projects

21. Main Electrical Supply redundancy project 21 2013 – 1,95/ 4 MW 1,95 MW will be backed up by the power supplier 2016 -2019 - 4 / 9MW 4 MW will be guaranteed by the power supplier 2011 - 1,7/ 6 MW Final decision taken to face a new challenge Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 4 th 2011 In the first phase the electrical redundancy will be minimal On going projects

22. Future Development project 22 Amphitheatre accessible to the public Multimedia center Outreach center Dedicated Training center A “Hub” for fundamental research, University and public information An office space was part of the original extension project but dropped due to budget limitation Project Re-formatted with new functionalities added Very well perceived by the Region and the city community (Grand Lyon) Pascal.trouve@cc.inp3.fr Hepix Spring Meeting - Darmstadt - Mai 5 th 2011