1. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Options for Expanding CERN’s Computing Capacity Without A New Building Medium term plans to handle growth of CERN’s computing needs HEPiX NERSC Alan Silverman ( with input by T.Cass, L.Pregernig and W.Salter) 29 th October 2009

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Outline Background Options – hosting, containers, new centre Features of Hosting Features of Containers IT’s plans Current status

3. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Limits of Building 513 Capacity Building 513 arriving to maximum capacity for power and cooling while demands to expand LCG and infrastructure services continue to grow. Limit probably now reached with respect to “critical” systems – systems whose loss would mean major disruption (corporate data and physics meta-data, mail servers, network devices, etc). Plans being made to accommodate more critical services but short term measures needed for next round of database servers among remaining 2009 purchases. IT Department – 29 October 2009 – HEPiX NERSC- 3

4. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Long Term – A New Building? Plans and budget exist for a Tier 0 centre – a major new building, probably on Prevessin site. Uncertainty created by - –Uncertain physics needs in advance of real LHC data –Ongoing debate of cost-effectiveness of one-time new building versus incrementally adding containers –Unknown costing of different options –Our unfamiliarity with these options Offer from Norway to host a remote centre - –costing unclear –remote operations support also unclear Prefer to take time to understand these various options before coming to a decision. IT Department – 29 October 2009 – HEPiX NERSC- 4

5. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Short and Medium term Short-term: Price Enquiry issued to host offsite some “critical” systems to be acquired in the next round of purchases. Medium-term: Call for Tender to purchase (or rent) a number of Containers to install on the CERN site. Should be enough for the coming 2-5 years to give us time - –to get some experience of the Container option –to decide long term future –if a new building is chosen, to get it designed and built. IT Department – 29 October 2009 – HEPiX NERSC- 5

6. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Hosting – choices - 1 Rent space and use our own racks or rent space in their racks? If their racks, does our choice of PC fit with the specifications we select from our PC suppliers? Do they support the power rating we need per rack? Options for cooling the racks Do we rent space now for possible expansion or add this later? (Risk of renting non- contiguous spaces – does this even matter?) IT Department – 29 October– HEPiX- 6

7. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Hosting – choices - 2 Dual power supply? UPS cover? Networking facilities in the Hosting Centre and between the Hosting Centre and CERN? Local (near CERN) so we can install our own systems and manage them; or remote in which case what service is available? Affect on cost (Geneva prices)? Any software licences needed which may be tied to CERN site? Maintenance contracts for PCs still valid? Data protection laws if personnel or financial data stored offsite? IT Department – 29 October– HEPiX- 7

8. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Status of Hosting Plans - 1 Stay close to CERN so that we can get some experience with minimal risk. Supplier identified after Price Enquiry. One attractive supplier did not bid because his current installation is full and it would have cost too much for him/us to equip the next part of his building in time for our needs. Contract under negotiation with target of February start. Data protection laws in Switzerland are rather solid (fall-out from Swiss banking) Need 10GB network to CERN – under negotiation IT Department – 29 October– HEPiX- 8

9. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Status of Hosting Plans - 2 We have decided to use our own racks (because they are cheaper and we can use our manpower for installation) We will probably only rent space for immediate use and take the risk of extra space if we need it later being in a distant part of the hosting site?? No software licence issue but some critical servers can never be off-site for this reason. Since the hosting site is in Geneva, no maintenance issue this time. IT Department – 29 October– HEPiX- 9

10. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Container Features Size typically 3 by 12 metres. Can be stacked but then you need infrastructure to work on upper boxes. Can be in open air (e.g. at SLAC) but easier if indoors (e.g. Microsoft’s new supercentre in Chicago – space for 56 containers times 2000 servers per container) - –safer access when raining/snowing –covered area to unpack equipment to install. If indoors, need crane or access for mobile crane to install the boxes. No special access needs on Building, already on the Containers themselves. IT Department – 29 October 2009 – HEPiX NERSC- 10

11. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Container Features - 2 Need access to - –power –water (even if we buy chillers we need to fill them) –technical galleries (for the power and high-speed networking). Need 400V, 3 phase power - –typically up to 15-20 racks times 15-20Kw/rack = ~300Kw per unit for computers (depends on PCs installed) –150Kw per unit for cooling. IT Department – 29 October 2009 – HEPiX NERSC- 11

12. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t IT’s Plans for containers Initially buy (or rent?) 1 unit for “critical power” systems. Improves IT’s Business Continuity - different power source from 513, distance from 513. Plus 1 unit for LCG physics. Plan on 1 additional unit for physics per year for up to 4 years. Two options (in fact decided on second):- –Buy/rent containers and use Site Services for all power and cooling (then need power and cooling redundancy for critical power container plus UPS for physics units) –Buy/rent a complete service which implies an extra unit to provide safe power and external chillers for the containers. IT Department – 29 October 2009 – HEPiX NERSC- 12

13. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t IT’s needs for containers Ideally a building (Prevessin site? Meyrin West Area?) with capacity as needed Prevessin site offers better “business continuity”. This was chosen. Open ground (recommended to be concreted) with foreseen capacity for 4 years growth; ideally one area but not essential. This is the likely choice. IT Department – 29 October 2009 – HEPiX NERSC- 13

14. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Status of Container Plans - 1 Outdoor site (with a nearby small building) identified for initial set of containers. Near CCC so good networking close by. EN agree to supply 400V power from a nearby 18KV line and a basic water supply but no cooling facilities (so we need to acquire the chillers as well as backup diesels and UPS) CERN’s complicated purchase procedure has started; target installation Oct 2010 IT Department – 29 October 2009 – HEPiX NERSC- 14

15. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Status of Container Plans - 2 Initial installation is planned to be – One container for critical services – needs UPS in addition to dual power feeds; One container for physics – needs only dual power feeds; Each container will need a second container for diesel backup and chillers (and UPS for critical services container). Possibility to add one physics container (probably plus services container) per year for 2-3 years. Open question – buy or rent? What is the rental amortisation period? IT Department – 29 October– HEPiX- 15