1. The Worldwide LHC Computing Grid Frédéric Hemmer IT Department Head Visit of INTEL ISEF CERN Special Award Winners 2012 Thursday, 21 st June 2012

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it The LHC Data Challenge The accelerator will run for 20 years Experiments are producing about 20 Million Gigabytes of data each year (about 3 million DVDs – 700 years of movies!) LHC data analysis requires a computing power equivalent to ~100,000 of today's fastest PC processors Requires many cooperating computer centres, as CERN can only provide ~20% of the capacity June 2012 - Frédéric Hemmer 2

3. A distributed computing infrastructure to provide the production and analysis environments for the LHC experiments Managed and operated by a worldwide collaboration between the experiments and the participating computer centres The resources are distributed – for funding and sociological reasons Our task was to make use of the resources available to us – no matter where they are located June 2012 - Frédéric Hemmer WLCG – what and why? Tier-0 (CERN): Data recording Initial data reconstruction Data distribution Tier-1 (11 centres): Permanent storage Re-processing Analysis Tier-2 (~130 centres): Simulation End-user analysis 3

4. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Global Lambda Integrated Facility June 2012 - Frédéric Hemmer 4

5. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Data acquired in 2012 2012 Data written: Total 9.4 PB to end May >3 PB in May (cf 2 PB/month in 2011) 2012 Data written: Total 9.4 PB to end May >3 PB in May (cf 2 PB/month in 2011) Data accessed from tape, 2012 June 2012 - Frédéric Hemmer5

6. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Data transfers Global transfers > 10 GB/s (1 day) Global transfers (last month) CERN  Tier 1s (last 2 weeks) June 2012 - Frédéric Hemmer6

7. WLCG – no stop for computing Activity on 3 rd Jan

8. Problem - Technology Explosion with NGS 8

9. Problem - Data Growth & Storage Costs Tiered storage (2x disk, 1x tape) Invest: + 40% p.a. Disk Price: - 20% p.a. New Storage: 2x each 15 Month 9

10. Sequence Production & IT Infrastructure at EMBL Compute Power: 2000+ CPU Cores, 6+ TB RAM Storage: 1+ PB High Performance Disk 4 x Ilumina HiSeq2000 25 TB data each week 2 x Ilumina GAIIx 10

11. NGS - The Big Picture ~ 8.7 million species in the world (estimate) ~ 7 billion people Sequencers exist in both large centres & small research groups > 200 Ilumina HiSeq sequencers in Europe alone => capacity to sequence 1600 human genomes / month Largest centre: Beijing Genomics Institute (BGI) 167 sequencers, 130 HiSeq 2,000 human genomes / day 500-1000 Hiseq devices worldwide today 3-6 PB /day 1.1 – 2.2 Exabytes / year 11

12. World Map of High-throughput Sequencers 12

13. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it The CERN Data Centre in Numbers Data Centre Operations (Tier 0) – 24x7 operator support and System Administration services to support 24x7 operation of all IT services. – Hardware installation & retirement ~7,000 hardware movements/year; ~1800 disk failures/year – Management and Automation framework for large scale Linux clusters High Speed Routers (640 Mbps → 2.4 Tbps) 24 Ethernet Switches350 10 Gbps ports2000 Switching Capacity4.8 Tbps 1 Gbps ports16,939 10 Gbps ports558 Racks828 Servers8938 Processors15,694 Cores64,238 HEPSpec06482,507 Disks64,109 Raw disk capacity (TiB)63,289 Memory modules56,014 Memory capacity (TiB)158 RAID controllers3,749 Tape Drives160 Tape Cartridges45000 Tape slots56000 Tape Capacity (TiB)34000 IT Power Consumption2456 KW Total Power Consumption3890 KW June 2012 - Frédéric Hemmer 13

14. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Scaling CERN Data Center(s) to anticipated needs CERN Data Center dates back to the 70’s – Now optimizing the current facility (cooling automation, temperatures, infrastructure) Renovation of the “barn” for accommodating 450 KW of “critical” IT loads – an EN, FP, GS, HSE, IT joint venture June 2012 - Frédéric Hemmer Exploitation of 100 KW of remote facility down town – Understanding costs, remote dynamic management, ensure business continuity Exploitation of a remote Data center in Hungary – 100 Gbps connections – Agile infrastructure – virtualization 14

15. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it 15 June 2012 - Frédéric Hemmer

16. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it June 2012 - Frédéric Hemmer 16