Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland t CF CERN IT Facility Planning and Procurement HEPiX Fall 2010 Workshop Olof B ärring, CERN IT/CF CERN IT Facility Planning and Procurement

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Outline Team & Tasks IT facility Hardware inventory Procurement Conclusions CERN IT Facility Planning and Procurement - 2

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Team & Tasks IT re-org in effect from Jan’10 Computing Facility group (CF) “ Management and operation of the CERN Computer Centre and associated Computing Facilities.” –Automated Services for Infrastructure (ASI) –Facility Planning and Procurement (FPP) –System Administration and Operation (SAO) CERN IT Facility Planning and Procurement - 3

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Team & Tasks IT re-org in effect from Jan’10 Computing Facility group (CF) “ Management and operation of the CERN Computer Centre and associated Computing Facilities.” –Automated Services for Infrastructure (ASI) –Facility Planning and Procurement (FPP) 3 staff: Andras Horvath, Eric Bonfillou + me 2 students: Michael Scott, Imre Szebenyi (fellow) –System Administration and Operation (SAO) CERN IT Facility Planning and Procurement - 4

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Team & Tasks Technical responsible for procurement of servers and storage –Planning –Technical spec and questionnaire –Validation of bids –Liaise with purchasing and accounts payable services Hardware expertise –Follow-up systematic issues –Liaise with vendors –Maintain BIT suites and assist benchmarking Asset management –Maintain up-to-date inventory –Systematic tracking of FW, BIOS – plan and coordinate deployment campaigns –Track warranty service performance Vendor contacts –Regular warranty service reviews –Organize information/roadmap meetings CERN IT Facility Planning and Procurement - 5

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Planning and coordinatecommercial procedures (tenders) Facility Planning & Procurement CERN IT Facility Planning and Procurement - 6 FPP IT resource mgr, Service responsible Users IT operations & sysadmins Vendors DPO, Purchasing Planning h/w resource provisioning Installation/removal planning

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Outline Team & Tasks IT facility Hardware inventory Procurement Conclusions CERN IT Facility Planning and Procurement - 7

CERN IT Department CH-1211 Geneva 23 Switzerland t CF IT facility CERN IT Facility Planning and Procurement - 8

CERN IT Department CH-1211 Geneva 23 Switzerland t CF x2 Installation & removal CERN IT Facility Planning and Procurement - 9

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Installation & removal CERN IT Facility Planning and Procurement - 10 x 63

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Machine room considerations Machine room –Total available electrical power: 2.9MW –Power distribution in main machine room: 50kW per PDU feeding row of 15 racks  average 3-4kW/rack –Power distribution in vault allow for ~10kW/rack in water cooled racks (WCR) Available power committed –Cannot install without a prior remove Example: remove 388 servers for installation of 350 disk servers (12PB raw) –Consolidate CPU workers to WCR –Disk (DAS) servers in air cooled racks 8 x 4U CERN IT Facility Planning and Procurement - 11

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Outline Team & Tasks IT facility Hardware inventory Procurement Conclusions CERN IT Facility Planning and Procurement - 12

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Inventory CERN IT Facility Planning and Procurement - 13 Number of racks: 841 Number of systems: 11,320 Number of processors: 15,321 Number of cores: 57,651 SPEC CPU2006: 404,279 Raw HDD capacity (TiB): 61,137 Number of HDD's: 62,608 Memory capacity (TiB): 142 Memory modules: 50,775 Number of RAID controllers: 3,871 Current power consumption (kW): 2,486 Current power consumption (kVA): 2,622

CERN IT Department CH-1211 Geneva 23 Switzerland t CF SLS CERN IT Facility Planning and Procurement - 14

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Inventory  trends CERN IT Facility Planning and Procurement - 15 Woodcrest Harpertown Nehalem Westmere

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Inventory  trends CERN IT Facility Planning and Procurement - 16

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Inventory  tracking Serial numbers recorded in Cdb since 2009 Can be used for tracking repairs –Component change  new SN No Lemon sensor (yet) but daily cron-job –Dumps a snapshot of all SNs from Cdb –Compare with dump from previous day CERN IT Facility Planning and Procurement - 17

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Inventory  tracking CERN IT Facility Planning and Procurement - 18 Vendor replaced full batch of 1680 disks

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Inventory  tracking CERN IT Facility Planning and Procurement - 19

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Inventory  tracking CERN IT Facility Planning and Procurement - 20 HDD failures/week:30 Hours/week: 168  hrs/failure: 5.6 drives in the centre: 62,608  MTTF = 350,604 hrs <<1.2Mhrs!? 

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Outline Team & Tasks IT facility Hardware inventory Procurement Conclusions CERN IT Facility Planning and Procurement - 21

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Procurements in 2010 Sustained the 2009 ‘rate’ up to August –CERN wide budget cuts Impact of budget cuts –Postponed EOY tenders to kHS06 and 6PB (12PB raw) –Extend warranty where appropriate/applicable Opening for some competition –Last CPU tender split 2U Twin 2 dual Intel (L5640), 36GB RAM, 141HS06 1U quad AMD (6164HE) batch, 96GB RAM, 319HS06 –Also some promising bids with alternative chassis CERN IT Facility Planning and Procurement - 22

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Procurement – issues Early failures (or infant mortality) –Better burn-in tests  less problems in prod –But … acceptance time increases Resources unusable 3-4 months delayed prod deployment not unusual Waste of electrical- and man-power  –High risk even with split adjudication Degrading warranty service –30 – 50% missed repair target not unusual –Monitored but lack commercial means to put real pressure Low margin bidders can go bankrupt –Elonex (CH) in 2008 –Melrow (NL) in 2008 –CPI (DE) this year CERN IT Facility Planning and Procurement - 23 From Wikipedia, the free encyclopedia: Commodity computing is computing done in commodity computers as opposed to supermicro computers or boutique computers. Commodity computers are computer systems manufactured by multiple vendors, incorporating components based on open standards.

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Conclusions Installation/removal planning & IT are intimately coupled through electrical power constraints Up-to-date inventory underpins an optimal management of deployed assets –Trending and issue tracking –Identify and eliminate inefficient (Watts/capacity) resource Need to learn to cope with undesired side effects of strong competitive tendering CERN IT Facility Planning and Procurement - 24