Dominique Boutigny December 12, 2006 CC-IN2P3 a Tier-1 for W-LCG 1 st Chinese – French Workshop on LHC Physics and associated Grid Computing IHEP - Beijing

12/12/2006D. Boutigny2 dapnia CC-IN2P3 presentation (1) CC-IN2P3 is providing computing resources for the whole French community working on:  Particle Physics  Nuclear Physics  Astro-Particle Physics CC-IN2P3 is located in Villeurbanne near Lyon

12/12/2006D. Boutigny3 CC-IN2P3 presentation (2)  ~70 groups are using CC-IN2P3 resources  ~10 are very active  This is more than 2500 users  3700 jobs running in //  ~ jobs in waiting state  jobs completed each day Batch system load Waiting Running CC-IN2P3 is operating 24/24 – 7/7

12/12/2006D. Boutigny4 CPU resources (1) 2.5 M SpecInt2000 CPU Consumed / Available ~ % Sept 2004 – Sept 2006: CPU × ~ 6

12/12/2006D. Boutigny5 CPU resources (2) In 2006 we bought a new rack mountable IBM computer system In 2006 we bought a new rack mountable IBM computer system –Dual CPU / Dual core Opteron machines –265 computers  1060 cores –Total power: 1.7 M SpecInt2000 according to IBM1.7 M SpecInt2000 according to IBM 1.2 M SpecInt2000 measured by us1.2 M SpecInt2000 measured by us –Cost is ~ 0.59 € / SI2K This system is in the list of the top 500 world fastest computers (exact rank is 483) This system is in the list of the top 500 world fastest computers (exact rank is 483)

12/12/2006D. Boutigny6 CPU sharing between experiments Nuclear 11%

12/12/2006D. Boutigny7 Storage resources (1) TB Mass Storage Disk Start of LHC ramp up

12/12/2006D. Boutigny8 Storage resources (2) In 2006 we bought a 400 TB disk storage system from SUN In 2006 we bought a 400 TB disk storage system from SUN –1 high end server for 24 TB –Cost is ~ 1.5 € / GB A new tape silo is currently under purchase A new tape silo is currently under purchase –Very likely SUN/STK or IBM – slots –500 GB tape drives Will evolve to 1 TB tape drives next yearWill evolve to 1 TB tape drives next year  10 PB capacity  10 PB capacity

12/12/2006D. Boutigny9 The CC-IN2P3 Tier-1 and Analysis facility CC-IN2P3 has started to build a Tier-1 and an Analysis Facility for the 4 LHC experiments CC-IN2P3 has started to build a Tier-1 and an Analysis Facility for the 4 LHC experiments –CC-IN2P3 will provide 12 to 15% of the whole T1 computing Also need to continue to provide resources for other experiments Also need to continue to provide resources for other experiments –Astroparticles is needing more and more resources The plan is to keep 20% of the resources available for non LHC experiments in 2008 The plan is to keep 20% of the resources available for non LHC experiments in 2008

12/12/2006D. Boutigny10 CPU sharing between LHC experiments ATLAS LHC-b ALICE CMS In principle the sharing should have been: 45% ATLAS 45% ATLAS 25% CMS 25% CMS 15% ALICE 15% ALICE 15% LHC-b 15% LHC-b In 2006

12/12/2006D. Boutigny11 LHC computing ramp up ~500 kSI2k.month

12/12/2006D. Boutigny12 Planned resources (1) CPU Tapes Tier-1 component Analysis Facility 22 M SI2k Disk 8.5 PB 10 PB  Numbers have been revised according to the new LHC schedule

12/12/2006D. Boutigny13 Planned resources (2) The purchase process is a long operation (6-8 months) The purchase process is a long operation (6-8 months) We need to anticipate buying in order to be ready in time We need to anticipate buying in order to be ready in time We plan to buy 40% of year N during year N-1 with a 2 step delivery We plan to buy 40% of year N during year N-1 with a 2 step delivery –In 2007 we will buy: 5.7 M SI2k5.7 M SI2k 2.1 PB of disk2.1 PB of disk 4 PB of tapes4 PB of tapes ~100% of % of non-LHC This is ~ 5000 today's cores ! 2007 CC-IN2P3 budget will be ~ 9.2 M€ (without salaries)

12/12/2006D. Boutigny14 Network The 10 Gbps optical connection (LHC-OPN) with CERN is up and running since the beginning of the year The 10 Gbps optical connection (LHC-OPN) with CERN is up and running since the beginning of the year Another 10 Gbps will be setup with Karlsruhe (FZK) (Backup T1 site) Another 10 Gbps will be setup with Karlsruhe (FZK) (Backup T1 site) 2 x 1 Gbps dedicated connection between CC-IN2P3 and FNAL is up and running 2 x 1 Gbps dedicated connection between CC-IN2P3 and FNAL is up and running –Done within the framework of a research project on grid interoperability and massive data transfer –10 months to setup with an impressive coordination between many actors

12/12/2006D. Boutigny15 Required network bandwidth for LHC In (Gb/s) Out (Gb/s) T0 – T T1 – T1 55 T1 – T2 (France) T1 – T2 (Int'l)

12/12/2006D. Boutigny16 T1-T2 connections T1-T2 connections are under active discussion at CERN (C. Eck) T1-T2 connections are under active discussion at CERN (C. Eck) –Situation is still… confused ! Connections to : Connections to : –All French T2/T3 –Belgium (CMS) –Romania (ATLAS) –Korea (ALICE) South Africa (ALICE) – SPAIN (ALICE)South Africa (ALICE) – SPAIN (ALICE) –China and Japan are mentioned to be connected to both CC-IN2P3 and ASGC (Taipei)

12/12/2006D. Boutigny17 Manpower and Operation Running a Tier-1 requires manpower and a strong organization Running a Tier-1 requires manpower and a strong organization –In 2007 large chunks of new computing equipment will arrive every 2-3 months Manpower Manpower –Total CC-IN2P3 manpower is 65 FTE –3 computing engineer hired in 2006 –Will continue to hire 3 to 4 engineers / year up to 2008 Operation Operation –Grid operation is mainly done by people hired under EGEE contracts  12 EGEE people at CC-IN2P3  ~5 FTE dedicated to Grid operation –Very strong involvement in LCG worldwide operation framework User support is also crucial User support is also crucial –We will put 1 engineer in support for each LHC experiment –At the moment we have 2.5 FTE

12/12/2006D. Boutigny18 Infrastructure (1) The exponential increase of the computing resources has a significant impact on the computing centre infrastructure The exponential increase of the computing resources has a significant impact on the computing centre infrastructure CC-IN2P3 average electrical power in kW An important work is going on in order to upgrade the computer room Electrical distribution Electrical distribution Cooling Cooling Uninterruptible Power Supply Uninterruptible Power Supply  Up to 1.6 MW of computing equipment + cooling ?

12/12/2006D. Boutigny19 Infrastructure (2) The computer room upgrade is not enough to receive all the computing hardware The computer room upgrade is not enough to receive all the computing hardware A project to build a new building with a new computer room is already started A project to build a new building with a new computer room is already started –800 m 2 new computer room –Up to 2.5 MW of computing equipment (on top of the existing 1 MW)

12/12/2006D. Boutigny20 Conclusions CC-IN2P3 is building up its Tier-1 + Analysis Facility CC-IN2P3 is building up its Tier-1 + Analysis Facility Substantial budget has been allocated for 2007 Substantial budget has been allocated for 2007 –Clearly a strong priority for IN2P3 and CEA/DAPNIA Impact on infrastructure is huge Impact on infrastructure is huge An efficient collaboration between China and CC- IN2P3 on computing matters requires to setup a good network connection now An efficient collaboration between China and CC- IN2P3 on computing matters requires to setup a good network connection now