W.A.Wojcik/CCIN2P3, May Running the multi-platform, multi-experiment cluster at CCIN2P3 Wojciech A. Wojcik IN2P3 Computing Center URL:

W.A.Wojcik/CCIN2P3, May IN2P3 Computer Center  Provides the computing and data services for the French high energy and nuclear physicists: IN2P3 – 18 physics labs (in all big towns in France) CEA/DAPNIA  French groups are involved in 35 experiments at CERN, SLAC, FNAL, BNL, DESY and other sites (also astrophysics).  Specific situation: our CC is not directly connected to experimental facilities, like CERN, FNAL, SLAC, DESY, BNL.

W.A.Wojcik/CCIN2P3, May General rules  All groups/experiments share the same interactive and batch (BQS) clusters and other type of services (disk servers, tapes, HPSS and networking). Some exceptions later …  /usr/bin and lib (OS and compilers) are local  /usr/local/* on AFS, specific for each platform  /scratch – local tmp disk space  System, group and user profiles define the proper environment

W.A.Wojcik/CCIN2P3, May General rules  User has the AFS account with access to the following AFS disk spaces: HOME - backup by CC THRONG_DIR (up to 2GB) - backup by CC GROUP_DIR (n * 2GB), no – backup  Data are on: disks (GROUP_DIR, Objectivity), tapes (xtage system) or in HPSS  Data exchange on the following media: DLT, 9480 Network (bbftp)  ssh/ssf - access to/from external domains recommended.

W.A.Wojcik/CCIN2P3, May Supported platforms  Supported platforms: 1. Linux (RedHat 6.1, kernel smp) with different egcs compilers (gcc , gcc with patch for Objy 5.2, gcc – installed on /usr/local), requested by different experiments 2. Solaris 2.6, 2.7 soon 3. AIX HP-UX – end of this service already announced

W.A.Wojcik/CCIN2P3, May Support for experiments  About 35 different High Energy, Astrophysics and Nuclear Physics experiments.  LHC experiments: CMS, Atlas, Alice and LHCb.  Big non-CERN experiments: BaBar, D0, STAR, PHENIX, AUGER, EROS II.

W.A.Wojcik/CCIN2P3, May

8

9 Disk space  Need to make the disk storage independent of the operating system.  Disk servers based on: A3500 from Sun with 3.4 TB VSS from IBM with 2.2 TB ESS from IBM with 7.2 TB 9960 from Hitachi with 21.0 TB

W.A.Wojcik/CCIN2P3, May Mass storage  Supported medias (all in the STK robots): 3490 DLT4000/ (Eagles) Limited support for Redwood  HPSS – local developments: Interface with RFIO: – API: C, Fortran (via cfio from CERNLIB) – API: C++ (iostream) bbftp – secure parallel ftp using RFIO interface

W.A.Wojcik/CCIN2P3, May Mass storage  HPSS – test and production services $HPSS_TEST_SERVER:/hpsstest/in2p3.fr/… $HPSS_SERVER:/hpss/in2p3.fr/…  HPSS – usage: BaBar - usage via ams/oofs and RFIO EROS II – already 1.6 TB in HPSS AUGER, D0, ATLAS, LHCb Other experiments on tests: SNovae, DELPHI, ALICE, PHENIX, CMS

W.A.Wojcik/CCIN2P3, May Networking - LAN  Fast Ethernet (100 Mb full duplex) --> to interactive and batch services  Giga Ethernet (1 Gb full duplex) --> to disk servers and Objectivity/DB server

W.A.Wojcik/CCIN2P3, May Networking - WAN  Academic public network “Renater 2” based on virtual networking (ATM) with guaranteed bandwidth (VPN on ATM)  Lyon  CERN at 34Mb (155 Mb in June 2001)  Lyon  US is going through CERN  Lyon  Esnet (via STAR TAP), Mb, reserved for the traffic to/from ESnet, except FNAL.

W.A.Wojcik/CCIN2P3, May BAHIA - interactive front-end Based on multi-processors:  Linux (RedHat 6.1) -> 10 PentiumII PentiumIII1GHz (2 processors)  Solaris 2.6 -> 4 Ultra-4/E450  Solaris 2.7 -> 2 Ultra-4/E450  AIX > 6 F40  HP-UX > 7 HP9000/780/J282

W.A.Wojcik/CCIN2P3, May Batch system - BQS Batch based on BQS (CCIN2P3 product)  In constant development, used since 7 years  Posix compliant, platform independent (portable)  Possibilities to define the resources for the job (the class of job is calculated by scheduler as a function of): CPU time, memory CPU bound or I/O bound Platform(s) System resources: local scratch disk, stdin/out size User resources (switches, counters)

W.A.Wojcik/CCIN2P3, May Batch system - BQS  Scheduler takes into account: Targets for groups (declared twice a year for the big production runs) Consumption of cpu time in last periods: month, week, day for user and group Proper aging and interleave in the class queues  Possibility to open the worker for any combination of classes.

W.A.Wojcik/CCIN2P3, May Batch system - configuration  Linux (RedHat 6.1) -> 96 dual PIII 750MHz dual PIII1GHz  Solaris 2.6 -> 25 * Ultra60  Solaris 2.7 -> 2 * Ultra60 (test service)  AIX > 29 * RS * 43P-B50  HP-UX > 52 * HP9000/780

W.A.Wojcik/CCIN2P3, May Batch system – cpu usage

W.A.Wojcik/CCIN2P3, May Batch system – Linux cluster

W.A.Wojcik/CCIN2P3, May Regional Center for:  EROS II (Expérience de Recherches d’Objets Sombres par effet de lentilles gravitationnelles)  BaBar  Auger (PAO)  D0

W.A.Wojcik/CCIN2P3, May EROS II  Raw data (from ESO site in Chili) on DLTs (tar format).  Restructuring of the data from DLT to 3490 or 9480, creation of metadata on Oracle DB.  Data server (on development) - 7TB of data actually, 20TB at the end of experiment – using HPSS + WEB server.

W.A.Wojcik/CCIN2P3, May BaBar  AIX and HP-UX not supported by BaBar, Solaris 2.6 with Workshop 4.2 and Linux (RedHat 6.1). Solaris 2.7 in preparation.  Data are stored in ObjectivityDB, import/export of data is done using bbftp. The import/export on the tapes has been abandoned.  Objectivity (ams/oofs) servers (dedicated only to BaBar) have been installed (10 servers).  Usage of HPSS for staging the ObjectivityDB files.

W.A.Wojcik/CCIN2P3, May Experiment PAO

W.A.Wojcik/CCIN2P3, May PAO - sites

W.A.Wojcik/CCIN2P3, May PAO - AUGER  CCIN2P3 is acting as AECC (AUGER European CC).  Access granted to all AUGER users (AFS accounts provided).  CVS repository for AUGER software has been installed at CCIN2P3, access from AFS (from the local and non-local cells) and from non-AFS environment using ssh.  Linux is the preferred platform.  Simulation software based on Fortran programs.

W.A.Wojcik/CCIN2P3, May D0  Linux is one of D0 supported platforms and is available at CCIN2P3.  D0 software is using the KAI C++ compiler  Import/export of D0 data (using internal Enstore format) is a complicated work. We will try to use the bbftp as a file transfer program.

W.A.Wojcik/CCIN2P3, May CCIN2P3 Import/export CERN CASTOR HPSS SLAC HPSS FNAL ENSTORE SAM BNL HPSS ?? ? ?

W.A.Wojcik/CCIN2P3, May Problems  To add the new Objy servers (for other experiments) is very complicated. It needs the new separate machines, with modified port numbers in /etc/services. Under development for CMS.  The OS system versions and levels  The compilers versions (mainly for Objy for different experiments).  Solutions?

W.A.Wojcik/CCIN2P3, May Conclusions  The data exchange should be done using the standards (e.g. files or tapes) and common access interfaces (bbftp and rfio are the good examples).  Needs for better coordination and similar requirements on supported system and compiler levels between experiments.  The choice of the CASE technologie is out of the control of our CC acting as Regional Computer Center .  GRID will require more uniform configuration of the distributed elements.  Who can help? HEPCCC? HEPiX? GRID?