1. Diamond Computing Nick Rees et al.

2. Storage and Computing Requirements

3. Storage – Initial requirements 200TBytes usable disk space 20 Mbytes/sec/Tbyte scalable aggregated throughput. –4 GBytes/sec aggregated throughput for 200 Tbytes. 100 MBytes/sec transfer rate for individual 1 Gbit clients 400 MBytes/sec for individual 10 Gbit clients Highly resilient Support for Linux clients (RHEL4U6 and RHEL5 or later) POSIX with extended attributes and ACLs Groups for file access control based (> 256 groups/user) Ethernet Clients. Extendable by at least 200TB for future growth.

4. Storage - solution Based on Data Direct Networks S2A9900 system Up to 5.7 GB/s throughput Fault-tolerant architecture Runs Lustre file system –Open source file system recently acquired by Sun –Most popular file system in top 100 supercomputers

5. Storage - solution 10 Disk trays 60 hot-swap disks/tray 2 redundant trays.

6. Storage - solution 60 tiers of 10 disks 2 parity disks/tier 2 controllers – both controllers can access all data. FPGA based - no loss of performance when running in degraded mode.

7. Computing – Initial requirements 1500 SPECfp_rate2006 total. x86 based processors with EM64T extensions 2GBytes RAM per CPU core Gigabit Ethernet Remote management PCI-E 16x slot for GPU accelerator Price includes 3 years power price (50% of total)

8. Computing - solution 15 Viglen HX2225i, 1U Twin Motherboard Systems Each 1U system comprises: –Supermicro X7DWT Motherboard –Intel 5400 (Seaburg) Chipset –Dual Intel Xeon E5420 (Quad Core, 2.5GHz) Processors –16GB DDR2-667 ECC Registered FBDIMM –Memory Installed as 4x4GB FBDIMMs per motherboard –160GB SATA Hard Disk –Single x16 PCI-Express slot (for GPU extension) Total 20 systems, 16 cores/system, 320 cores.

9. Computing – solution 1U twin motherboard systems Each 1U motherboard is: –Supermicro X7DWT –Intel 5400 (Seaburg) Chipset –Dual Intel Xeon E5420 (Quad Core, 2.5GHz) Processors –16GB DDR2-667 ECC RAM –Installed as 4x4GB FBDIMMs –160GB SATA Hard Disk –1x16 PCI-Express slot (for GPU extension) Total 20 systems, 16 cores/system, 320 cores.

10. Network - current layout

11. Network – new layout Two core switches, one in each computer room Each beamline connected to both switches – so either 2GBit or 20 GBit peak bandwidth available –halved if one core switch fails Cluster switch connected with 2 x 10 GBit to each switch. Storage system connected directly into core. Formally looking at a goal of >99.95% reliability on network infrastructure during beam time. –Corresponds to 3 hours outage/year –Doesn’t include reliability of associated computer services.

12. Network - new layout

13. The interesting specifications Each disk tray: –Consumes up to 1750 W –Weighs up to 105 kg –Is 4U high (10 trays in one rack) Each compute node –Consumes up to 600 W –Weighs 18 kg –Is 1U high (38 nodes in each rack) So: –10 disk trays in a rack are 17.5 kW and weigh 1050 kg –38 computers in a rack are 22.8 kW and weigh 684 kg

14. Where do we put the systems? We need a computer room which provides: –~20kW/rack. –> 1 ton rack load –24 hrs/day operation, even during power and water maintenance. Or there is the CERN alternative: –currently fill their racks with 6 1U servers and then they are “full”.

15. New computer room Room has up to 350 kW redundant power, (A and B) from two separate sub-stations. Power from A is UPS and generator backed up. Room is provided with up to 350 kW cooling water. Primary cooling is from site chilled water, 220 kW standby chiller in case of problems.

16. Computer room - racks 10 racks initially – 2 network, 8 computer. Cooling by in-rack cooling units to cope with the high power density. Low power network racks also have cooling units to provide N+1 resiliancy. Computers exhaust into a shared hot-air plenum which can only escape through one of the cooling units. Each rack has four 11 kW 3-phase power strips (two from each sub-station). Status of system continuously monitored and available through browser and SNMP.

17. Contained cold aisle 19” server / equipment 20 kW in row air to water hear exchanger Double doors Network racks 4,375 mm 800 mm 4,000 mm 1,200 mm Racks - cooling

18. Racks - layout

19. Current situation Water pipes Cable Tray

20. Racks Front view Rear view 20 kW air handling unit 2 x 11 kW power strips

21. Some calculations Heat capacity of air = 0.0013 J/cm^3/K = 1.3 kJ/m^3/K Volume of 1 rack = 0.6x0.8x2 m 3 = 0.96 m 3 Typical Delta T for computers = 20 C So, air change time for 20 kW rack=20*0.96/(1.3*20) = 0.74 s Want 8 server racks = 160 kW. Volume of cold air in racks = 1.2x2x4.5 = 10.8 m 3 Temperature rise rate of air = 160/(10.8*1.3) = 11.4 deg/sec The room volume = 16*6.5*2.7 = 280 m 3. Temperature rise rate of air = 160/(280*1.3) = 0.44 deg/sec Need safety system in case water stops flowing!

22. The Services Crisis Server compute performance has been increasing by a factor of three every two years. Computer energy efficiency is only doubles every two years. Computer spend is staying relatively constant. –Computational requirements appear to scale with the compute performance. So, while power consumption per computational unit has dropped dramatically, the power consumption per rack has risen dramatically.

23. Bottom line Reprinted with permission of The Uptime Institute Inc from a White Paper titled “The Invisible Crisis in the Data Center: The Economic Meltdown of Moore’s Law” $1000 also buys you 190 W over 3 years @ $0.20/kWh

24. Rack loads Floor loads are approaching 1.5 tons/m 2 Architects had said there were no problems When I looked the floor spec was 350 kg/m 2

25. Floor strengthening

26. Conclusions Computing is now all about providing services We are fighting hard against the heat capacity of air – its down to very basic physics. Costs are dominated by services – £1,000k gives: –£45k computing –£300k storage –£105k network –£200k racks –£100k mechanical services (water pipes) –£100k electrical services –£100k floor strengthening –£50k (largely wasted) architectural services

27. Introduction Over the past year we have spent a lot of time thinking about how to provide the computing resources for Phase 2. Started as a project to just look at tomography, but broadened to all of Phase 2 computing. Taken input from beamlines, data acquisition group, other synchrotrons and management. Had proposals for capital projects in the system since June 2007. Visited SLS, CERN and ESRF in November 2007. Finally had Gerd signoff late February 2008.

28. Other requirements Some beamline load is relatively constant (e.g. MX), but others are heavily peaked (Tomography). Total demand can rise rapidly with new detectors (e.g. Pilatus). We need to maintain consistency across beamlines to keep our overheads low. We need a balanced system with no major bottlenecks We need power and cooling infrastructure – beamlines are not designed for the spiraling computing demands.

29. Typical data analysis Detector Image processing Align Image generation Remove Instrumental signature Display Archive and off-site copy

30. 10 Gbit/s Physical system implementation Disks 1 Gbit/s 2 Gbit/s Phase 1 Beamlines Beamline Switch 0.5 Gbit/s 80 Gbit/s Phase 2 Beamlines Disks Beamline Switch Cluster Switch 1 Gbit/s 10 Gbit/s 40x1 Gbit/s Central Switch Cluster 60 Gbit/s 40 Gbit/s

31. Other bottlenecks The disk bottleneck is not the only possibility, it is just the major one we encountered in Phase 1 Other bottlenecks include –Latencies as well as throughputs –Network –Backplane –Memory –CPU A balanced system is required CPU is easiest and cheapest to address, but often doesn’t solve the problem Software design changes are often the most effective and cheapest.

32. Other considerations This is just raw power cost. –Our tenders included the 3 year power can cooling cost as part of the tendered price. Rack power density is doubling every two years. Rack cooling load is doubling every two years. The capital cost of providing the room services is significant.

33. The snowball effect Initial task: –Provide some computers for tomography for a number of beamlines. Result: –Need to provide network to that room Need to ensure network won’t bottleneck Ensure it keeps working, even during maintenance. –Need to provide disk storage that won’t bottleneck –Need to provide a room to house computers Need to provide cooling for that room Need to provide power for that room Ensure it keeps working even during maintenance. –Need to build in room for expansion –We’ve done so much, why don’t provide enough for all Phase 2 beamlines? –…and, by the way, buy a few compute nodes as well.

34. Result We are building a new high density computer room with resilient power and cooling. We will upgrade the network to increase performance and provide resiliency. We have purchased a storage system about 200 TBytes in size, with 6 GBytes/sec performance. –Can be expanded to nearly 500 TBytes We have bought a 240 core compute cluster for the next MX beamlines and Tomography. –Each dual quad-core motherboard has a PCI-Express x16 slot for NVidia expansion.

35. Philosophy Beamlines share highly resilient, high performance, central computing facilities We ensure the network is always available and the computing and storage is always there. Beamlines have local work stations, control computers, media stations printers etc. Design relies on a balance of network, storage and computing performance. –Computing cost £45k –Network cost £105k –Storage cost £300k –Racks, services, and building costs £500k

36. Layout of Beamline Computing Resources Science Router Beamline Firewall/ filter Cluster Switch Machine Switch Diamond House Compute Cluster RAID Storage cluster Computer Room Cluster Firewall/ filter 1 GBit/s Ethernet 4 GBit/s Fibre Channel 10 GBit/s Ethernet Beamline Switch Detector Server Media Station Consoles Detector Local detector storage Beamline User USB and eSATA drives CA Gateway File Servers

37. Computer room Rooms 1.152-1.155 in Zone 10. Selected because it was not suitable for offices because the outside windows are being blocked off by the I12 build. Possibility of extending it counter-clockwise, if necessary.

38. Computer room - location

39. I12

40. Computer room - location I12 Standby Chiller Chilled water distribution: – provides cooling directly to racks

41. New Computer Room A new computer room is being constructed in Zone 10 above I12. Initial computer suite is 8 racks: –Each rack has ~20 kW of cooling –Each rack has ~22 kW of redundant power –Backup chiller is available if site cooling is unavailable. Initial computing fills about 2 racks: –200 TBytes of disk with 6 Gbytes/sec performance Phase 1 disks are bottlenecked at ~50 Mbytes/sec –240 cores of computing. Network is also upgraded: –Core network is two independent stars. We expect the redundancy to be used more to enable maintenance to take place without interrupting services, rather than because of actual equipment failure.

42. Current situation Water pipes Cable Tray

43. Future expansion Current systems fill 2.5 racks out of 8. Room services allow a doubling in power and cooling capacity Room area allows more than doubling of number of racks. Disk trays only have 25 disks with a capacity of 60 –Note that adding disks does not increase available bandwidth. Compute nodes can have NVidia accelerators

44. Programme End March 2008: –Design finalised (PB) –Tenders ready for racks, cooling, network, storage and computing End April 2008: –Prime contractor selected and on board –Electrical enabling works complete May-July 2008 –Build work –Suppliers for racks, cooling etc selected and orders placed. Late July 2008 –Build work completed. –In room fitout starts Will actually start late August Late September 2008 –Systems available for use Will be November at least

45. Conclusions We are working on an ambitious upgrade to Diamonds computing resources We have a tight timescale We believe we can meet it, but will be needing more staff (and are currently hiring…)

46. Any Questions?