The Worldwide LHC Computing Grid Frédéric Hemmer IT Department Head Visit of INTEL ISEF CERN Special Award Winners 2012 Thursday, 21 st June 2012

CERN IT Department CH-1211 Genève 23 Switzerland The LHC Data Challenge The accelerator will run for 20 years Experiments are producing about 20 Million Gigabytes of data each year (about 3 million DVDs – 700 years of movies!) LHC data analysis requires a computing power equivalent to ~100,000 of today's fastest PC processors Requires many cooperating computer centres, as CERN can only provide ~20% of the capacity June Frédéric Hemmer 2

A distributed computing infrastructure to provide the production and analysis environments for the LHC experiments Managed and operated by a worldwide collaboration between the experiments and the participating computer centres The resources are distributed – for funding and sociological reasons Our task was to make use of the resources available to us – no matter where they are located June Frédéric Hemmer WLCG – what and why? Tier-0 (CERN): Data recording Initial data reconstruction Data distribution Tier-1 (11 centres): Permanent storage Re-processing Analysis Tier-2 (~130 centres): Simulation End-user analysis 3

CERN IT Department CH-1211 Genève 23 Switzerland Global Lambda Integrated Facility June Frédéric Hemmer 4

CERN IT Department CH-1211 Genève 23 Switzerland Data acquired in Data written: Total 9.4 PB to end May >3 PB in May (cf 2 PB/month in 2011) 2012 Data written: Total 9.4 PB to end May >3 PB in May (cf 2 PB/month in 2011) Data accessed from tape, 2012 June Frédéric Hemmer5

CERN IT Department CH-1211 Genève 23 Switzerland Data transfers Global transfers > 10 GB/s (1 day) Global transfers (last month) CERN  Tier 1s (last 2 weeks) June Frédéric Hemmer6

WLCG – no stop for computing Activity on 3 rd Jan

Problem - Technology Explosion with NGS 8

Problem - Data Growth & Storage Costs Tiered storage (2x disk, 1x tape) Invest: + 40% p.a. Disk Price: - 20% p.a. New Storage: 2x each 15 Month 9

Sequence Production & IT Infrastructure at EMBL Compute Power: CPU Cores, 6+ TB RAM Storage: 1+ PB High Performance Disk 4 x Ilumina HiSeq TB data each week 2 x Ilumina GAIIx 10

NGS - The Big Picture ~ 8.7 million species in the world (estimate) ~ 7 billion people Sequencers exist in both large centres & small research groups > 200 Ilumina HiSeq sequencers in Europe alone => capacity to sequence 1600 human genomes / month Largest centre: Beijing Genomics Institute (BGI) 167 sequencers, 130 HiSeq 2,000 human genomes / day Hiseq devices worldwide today 3-6 PB /day 1.1 – 2.2 Exabytes / year 11

World Map of High-throughput Sequencers 12

CERN IT Department CH-1211 Genève 23 Switzerland The CERN Data Centre in Numbers Data Centre Operations (Tier 0) – 24x7 operator support and System Administration services to support 24x7 operation of all IT services. – Hardware installation & retirement ~7,000 hardware movements/year; ~1800 disk failures/year – Management and Automation framework for large scale Linux clusters High Speed Routers (640 Mbps → 2.4 Tbps) 24 Ethernet Switches Gbps ports2000 Switching Capacity4.8 Tbps 1 Gbps ports16, Gbps ports558 Racks828 Servers8938 Processors15,694 Cores64,238 HEPSpec06482,507 Disks64,109 Raw disk capacity (TiB)63,289 Memory modules56,014 Memory capacity (TiB)158 RAID controllers3,749 Tape Drives160 Tape Cartridges45000 Tape slots56000 Tape Capacity (TiB)34000 IT Power Consumption2456 KW Total Power Consumption3890 KW June Frédéric Hemmer 13

CERN IT Department CH-1211 Genève 23 Switzerland Scaling CERN Data Center(s) to anticipated needs CERN Data Center dates back to the 70’s – Now optimizing the current facility (cooling automation, temperatures, infrastructure) Renovation of the “barn” for accommodating 450 KW of “critical” IT loads – an EN, FP, GS, HSE, IT joint venture June Frédéric Hemmer Exploitation of 100 KW of remote facility down town – Understanding costs, remote dynamic management, ensure business continuity Exploitation of a remote Data center in Hungary – 100 Gbps connections – Agile infrastructure – virtualization 14

CERN IT Department CH-1211 Genève 23 Switzerland 15 June Frédéric Hemmer

CERN IT Department CH-1211 Genève 23 Switzerland June Frédéric Hemmer 16