CC-IN2P3: A High Performance Data Center for Research Dominique Boutigny February 2011 Toward a future cooperation with Israel

February 2011 CC-IN2P3 in the French scientific framework Main scientific organization in France National Institute for Nuclear Physics and Particle Physics Dedicated Computing Center Irfu CC-IN2P3 federates the main computing resources For : High energy physics Nuclear physics Astroparticle physics Atomic Energy Commission + some opening to other sciences Manpower: 85 people Budget 2010: 10.8 M€ with salaries 2Dominique Boutigny

February 2011 A diversity of scientific applications CC-IN2P3 is part of a worldwide network of datacenters for High Energy Physics 3Dominique Boutigny

February 2011 The Large Hadron Collider (LHC): the today’s challenge 4 experiments with a level of complexity never reached before 10 billions collisions recorded each year ATLAS CMS Up to 15 PB/year of raw data 4Dominique Boutigny

February 2011 A global grid infrastructure CC-IN2P3 FZK PIC NDGF NIKHEF ASCC Brookhaven Fermilab TRIUMF RAL CNAF T1 (11) T0 T2 (~60) Île de France Clermont Nantes Grenoble Lyon CC-IN2P3 Annecy T3 (many) Strasbourg Marseille 5Dominique Boutigny

February 20116Dominique Boutigny

CC-IN2P3 February 2011 CC-IN2P3 T1 T2 7Dominique Boutigny

The French LCG sites February 2011Dominique Boutigny Tier-2: LPC AF: CC-IN2P3 Tier-2: LAPP Tier-2: IPHC Lyon Clermont-Ferrand Ile de France Marseille Nantes Strasbourg Annecy Tier-3: IPNL Tier-2: CPPM Tier-2: Subatech Tier-2: GRIF CEA/IRFU LAL LLR LPNHE IPNO Tier-2: GRIF CEA/IRFU LAL LLR LPNHE IPNO Tier-1: CC-IN2P3 Grenoble Tier-3: LPSC Sept connection to remote T2: Beijing – Tokyo – Romania - Belgium

France Grilles en 2010 February 2011 CC-IN2P3 is also playing a central role in the French NGI  See Vincent’s presentation 9Dominique Boutigny

Grid CC-IN2P3 February 2011 CC-IN2P3 is the main node of a French multidisciplinary grid (France Grille) which is part of the European infrastructure EGI Operation Support Training Interoperability tools and portals based on standards Distributed data management tools a.k.a iRODS 10Dominique Boutigny

February 2011 Credit: R. Lemrani FP 7 design study for PLT 11Dominique Boutigny

Some highlights about the services provided by CC-IN2P3 February 2011 Main data repository site for HESS and AUGER 1100 cores for Fermi simulations Main CC after SLAC Will increase significantly this year 1100 cores for Fermi simulations Main CC after SLAC Will increase significantly this year Operate the central Oracle database for Opera – up to 120 TB Image processing and analysis for Supernovae detection projects Provide data storage and operate data distribution for all experiments 12Dominique Boutigny First remote computing site for BaBar. Full backup of the data (1.6 PB to be transferred) First remote computing site for BaBar. Full backup of the data (1.6 PB to be transferred) The main remote production site for D0 We also provide dedicated network links to isolated French sites

The HEP model has been implemented for other scientific applications February 2011Dominique Boutigny Biomedical imagery Embryogenesis Biology Humanities 13 Big usage of the grid storage middleware iRODS Allow to store and distribute data and metadata in an intelligent and flexible way

CC-IN2P3 current status February 2011 Today, computing power at CC-IN2P3 is dominated by LHC But several other experiments ( >40) relies on CC-IN2P3 for their computing cores 10 PB of disk storage 4 robotic silos (up to 40 Pbytes capacity) 14Dominique Boutigny

Network February 2011 CC-IN2P3 is hosting a RENATER PoP Excellent connections within France and to the international network Several dedicated 10 Gb/s connections Successful test of a 100 Gb/s connection between CERN and CC-IN2P3 15Dominique Boutigny

Infrastructure evolution February 2011 The very fast increase of the computing equipments installed at CC-IN2P3 creates a huge load on the infrastructure 16Dominique Boutigny

The infrastructure is at its limit February 2011 A lot of money invested since 6 years But not enough to fulfill our long term scientific commitments 17Dominique Boutigny

 Build a new datacenter February 2011 The design parameters used to define the size of the new datacenter has been the following:  Serve ~40 experiments with "standards needs"  Fulfill LHC computing commitments and provide first class analysis capability  Expect a very significant growth of the Astroparticle community needs: LSST, EUCLID)  Add some capacity for network, services, etc. ~ today’s cores 18Dominique Boutigny

New datacenter February 2011  Assume Moore law is still valid  Extrapolate up to 2019 On top of the existing computer room (1 MW) indicated power is for computing only power for cooling has to be added Due to budget constraints the new computer room will start with a limited power (600 kW)  Modular design – Easily scalable Chilled water and electricity distribution has been designed for the 2019 target value Equipment: transformers, chillers, UPS etc… will be added later End up with: racks 600 kW racks 600 kW racks 1.5 MW racks 1.5 MW racks 3.2 MW racks 3.2 MW 19Dominique Boutigny

The new datacenter February Dominique Boutigny

February 2011 All the cooling equipment is on the roof Connection between the 2 computer rooms + elevator Electrical equipment in the basement Computer room in the 2 nd floor (850 m 2 ) 21Dominique Boutigny

The new computer room will be available early March An upgrade plan do exist to bring it to full power by 2019 (4.2 MW total capacity for computing equipment) Plenty of room, cooling and power February Dominique Boutigny

TIDRA : A regional initiative February 2011Dominique Boutigny23 Annecy Grenoble Lyon The idea is to federate Grid resources and expertise in the Rhône-Alpes region 5 sites (4 HEP including CC-IN2P3 + 1 Bio) Open to the industrial world Contacts with image generation industries

Associated International laboratories February 2011Dominique Boutigny24 CC-IN2P3 is participating to 4 Associated International Laboratories : China – Japan – Korea and Vietnam Interoperability tools based on OGF standards Data management tools General computing expertise sharing Setup and maintain a French-Asian VO – support for AIL scientific activities  1 publication related to LQCD Support mainly GEANT4 and LQCD applications

February 2011Dominique Boutigny25