S.Jarp CERN openlab CERN openlab Total Cost of Ownership 11 November 2003 Sverre Jarp

S.Jarp CERN openlab Agenda Day 1 (IT Amphitheatre) Introductory talks: 09:00 – 09:30 Welcome. (W. von Rüden) 09:30 – 10:15 Setting the scene: A Global View of the plans for the LHC Computing Grid – (L.Robertson) 10:15 – 10:45 Coffee break Part 2: 10:45 – 11:15 Review of calculated cost summary for 2003 (S. Jarp) 11:15 – 12:00 Estimated LCG Personnel and Materials costs (2004 – 2008) (B.Panzer) 12:30 – 14:00 Lunch Part 3 14:00 – 14:45 Planning the LCG Fabric at CERN (T. Cass) 14:45 – 15:30 External Testimony: Michael Levine (Pittsburgh Supercomputer Center) 15:30 – 16:00 Coffee break 16:00 – Discussions and conclusion of day 1 (All) Day 2 (IT Amphitheatre MORNING, afternoon) Discussions with HP: 08:45 – 10:45 One-on-one with HP Discussions with Intel: 11:00 – 13:00 One-on-one with Intel Discussions with IBM: 14:00 – 16:00 One-on-one with IBM

S.Jarp CERN openlab AIM Understand Total Cost of Ownership “Linux fabric” in the Computer Centre CPU servers Disk servers, Tape servers + Silos To a “sufficient” level of detail 2003 (July/August) Budget situation Rather than actual sums across all cost centres Personnel + Materials No direct measurement of quality Nor the fraction of capacity actually in use at a given time

S.Jarp CERN openlab Cost of manpower Complicated issue Approach chosen: Take IT salary budget for staff and fellows Including direct benefits (pension fund, health care, etc.) Without counting management overhead Neither inside IT nor CERN Actual number: KCHF Excluded: Associates + Students In general, paid from the materials budget Associates: Some do not necessarily spend most of their time at CERN Students: Avoid very short stays (Summer students)

S.Jarp CERN openlab IT budget categories In the TCO exercise the total IT budgets were initially split into 9 categories: Computer Centre Services: Cost of the Physics Fabric and closely related services, such as internal and external networking, etc. Grid Development/Deployment Openlab Computing Infrastructure (non-physics oriented): Windows, Mail, Web services, Phone exchange, etc. Product Support (mainly for Engineers): Digital electronics, CAD, license office, Solaris support. etc. Controls for physics experiment so-called “on-line computing” Database Services and Applications Communications Infrastructure for the LHC Accelerator Divisional Management

S.Jarp CERN openlab IT Division (total) Materials (KCHF) FTE% of IT total (Materials + FTEs) (Basic) Computer Centre Services12, Grid Development and Deployment Openlab Infrastructure (non-physics) 4, Product Support 1, Controls Databases and Applications 1, Accelerator Communications 3, Divisional Management Total25,

S.Jarp CERN openlab Box count Central Linux Batch “white box” servers1490 Central IDE Disk servers 250 Central PC Tape servers 60 Linux systems in the Computer Centre1800 Linux systems outside the computer centre 1600 Central Windows servers 150 Central Solaris servers 50 IP devices on Campus:28K

S.Jarp CERN openlab Capacity Numbers Other relevant capacity numbers are: Total installed PC CPU capacity: ~1.2 Million SpecInt2000 (SI2K) CPU capacity purchased by August: 400 * 1.6 KSI2K = 640 KSI2K Total disk space: 200 TB Disk space acquired by August: 50 TB at required performance (actually: 65 TB) Tape silos installed: 10 (with 6000 slots each) Tape drives: 50 STK9940B (+ 20 STK9840 with low usage)

S.Jarp CERN openlab Cost of Central Services Batch/Interactive services: Acquisition costs for batch/interactive servers, software licenses and maintenance Managed storage: Tape and disk servers (purchase and maintenance) Computer Centre Infrastructure: Various contract staff contracts; Monitoring and Supervision; Remedy Fabrics: The budget line only covers the CC upgrade this year. The two "Operations" lines: Cover costs associated with the various groups (travel, associates/consultants, workstation purchases, etc.)

S.Jarp CERN openlab Cost of Central Services

S.Jarp CERN openlab Cost of Fabric

S.Jarp CERN openlab Costs beyond the fabric In many cases non-fabric-related costs dominate Campus Network: 2991 KCHF User Support and Quality Insurance: 1265 KCHF Computer security: 741 KCHF Etc. In other words: The fabric benefits from large demands on other services

S.Jarp CERN openlab CPU part of fabric

S.Jarp CERN openlab Disk/tape part of fabric

S.Jarp CERN openlab Summary All-in-all: IT estimates (on average) that it has spent ~(1 + 1) MCHF on basic hardware across CPU servers Disk servers + Tape servers Our calculation shows: Linux fabric in total: 18.6 MCHF CPU part: 10.5 MCHF Disk/tape part: 6.3 MCHF External networking: 1.8 MCHF Cost on top of basic hardware: 14.8 / 2 = 7.4

S.Jarp CERN openlab Backup

S.Jarp CERN openlab (replacement) cost IT main entities Acquisition (CHF) Installation, Maintenance (CHF) Quantity (per unit) CERN totalAcquisition (so far) 2003 CPU server2, SI2K1.2 MSI2K400 Disk server10,0002,0001 TB (1.3)200 TB50 Tape cartridge GB Tape silo150,000 +6,000 slots10- Tape drives (9940B/9840) 30,000/N.A /20 MB/s AFS home/project 80,0001 TB4 TB AFS scratch10,0002,0001 TB