Perspectives of JINR Grid-infrastructure V.V. Ivanov, Gh. Adam, V.V. Korenkov, T.A. Strizh, P.V. Zrelov Laboratory of Information Technologies Joint Institute for Nuclear Research

V. Ivanov (LIT, JINR) Participating in LHC, RHIC, TEVATRON… In future: FAIR, ILC … LHC-2008FAIR-2015ILC – after 2020 (?) The JINR participation in the LHC experiments and in other large-scale projects asked for a substantial increase of its networking and information resources as well as the deployment of a large volume of work toward the development of the JINR Grid-segment and its integration in the Russian Grid- infrastructure RDIG (Russian Data Intensive Grid).

V. Ivanov (LIT, JINR) The Laboratory of Information Technologies (LIT) is responsible for the computing infrastructure development in JINR and hosts the Central Information and Computing Complex (CICC) of the Institute. Further perspective tasks follow from the decision of the JINR Committee of the Plenipotentiary Representatives, concerning the radical improvement of the JINR computer telecommunication links with major partner laboratories in the JINR Member States during the years and the development of a unified Grid-environment of the JINR Member States. This is a cornerstone task of the JINR seven-year plan on 2010–2016 within the direction ”Networks. Computing. Computational Physics”.

V. Ivanov (LIT, JINR) Development of the JINR Grid-environment – Three levels within the JINR Grid- environment can be distinguished: network, resource, and applied. NETWORK RESOURCE APPLIED

V. Ivanov (LIT, JINR) Cooperation Collaborators from the JINR Member States (protocols of cooperation with INRNE (Bulgaria), ArmeSFo (Armenia), FZK Karlsruhe GmbH (Germany), IHEPI TSU (Georgia), NC PHEP BSU (Belarus), KFTI NASU (Ukraine), etc.). Grants of Plenipotentiaries of the JINR Member States: Poland, Bulgaria, Slovakia, Romania. Agreement on Joint Research Programme “Meshcheryakov-Hulubei” LIT-Romania. BMBF grant “JINR Network Security, Failover Measures, Network monitoring”. CERN-JINR Cooperation Agreement on several topics, and JINR in the LCG Project (participation of JINR in the LCG project). JINR-South Africa Cooperation Agreement - project “Development of Grid segment for the LHC experiments”. Participation in joint projects: Enabling Grids for E-sciencE (EGEE -a project co-funded by the European Commission through the Sixth Framework Programme, CERN-INTAS project and "SKIF-GRID“ project of the Program of the Belarusian-Russian Union State as a part of joint propositions of NASB and Federal Agency of Science and Innovations of Russian Federation “The development and use of hardware and software in GRID-technologies and advanced supercomputer systems SKIF in (SKIF-GRID)”. Joint research grants from RFBR and Ministry of Science and Education with MSU and SINP MSU, Saratov University, IM SB RAS, IMM RAS, TU Gomel (Belorussia).

V. Ivanov (LIT, JINR) June two important projects completed Two large-scale very important projects have been completed: -- JINR - Moscow 20Gbps telecommunication channel JINR CentralInformation and Computing Complex major upgrade. -- JINR Central Information and Computing Complex major upgrade. Comprehensive presentation was done at the meeting of the Programme Advisory Committee for Particle Physics, on June 10. All organizations involved in the project realizations, together with the representatives of Ministry of Communications, Federal Agency for Science and Innovation, attended the Presentation. Our community colleagues and prospective users - residents of the special economical zone - also took part in the event.

V. Ivanov (LIT, JINR) Collaboration with RSCC, RIPN, MSK-IX, JET Infosystems, Nortel JINR - Moscow 20Gbps telecommunication channel In perspective, the used DWDM and 10Gb Ethernet technologies allow the creation of up to 80 channels of 10 Gbps each, resulting in a throughput of up to 800 Gbps.

V. Ivanov (LIT, JINR) JINR Central Information and Computing Complex In 2008, the total CICC performance was 1400 kSI2K, the disk storage capacity 100 TB At present, the CICC performance equals 2300 kSI2K and the disk storage capacity 500 TB CPU(kSI2K) Disk(TB) Tapes(TB) PLAN

V. Ivanov (LIT, JINR) JINR Local Area Network Backbone (LAN) Comprises 6579 computers and nodes, Users – 3593, IP – 7633, Remote VPN users (Lanpolis, Contact, TelecomMPK) ; High-speed transport (1Gbps) (Min. 100 Mbps to each PC); Controlled-access (Cisco PIX- 525 firewall) at network entrance; Partially isolated local traffic (8 divisions have own subnetworks with Cisco Catalyst 3550 as gateways); General network authorization system involves many services (AFS, batch systems, Grid, JINR LAN remote access, etc.) Comprises 6579 computers and nodes, Users – 3593, IP – 7633, Remote VPN users (Lanpolis, Contact, TelecomMPK) ; High-speed transport (1Gbps) (Min. 100 Mbps to each PC); Controlled-access (Cisco PIX- 525 firewall) at network entrance; Partially isolated local traffic (8 divisions have own subnetworks with Cisco Catalyst 3550 as gateways); General network authorization system involves many services (AFS, batch systems, Grid, JINR LAN remote access, etc.) Plans: Step-by-step modernization of the JINR Backbone – transfer to 10 Gbps Development and modernization of the control system of the JINR highway network

V. Ivanov (LIT, JINR) upgrade of the JINR backbone to a data transfer rate of 10 Gbps and gradual connection to it of various JINR laboratories; increase of the performance of the LAN central telecommunication node kernel; increase of the LAN protection at the hardware level; wireless access to the LAN within the JINR territory; implementation of new solutions and methods for the traffic control, information security, development of effective tools for control and the quality of service of JINR LAN; development of the system of protection of the JINR LAN resources from non-authorized access. The plan for the JINR LAN development for the years foresees several general purpose tasks as well as specific site-related ones. A selective enumeration points to the: First stage JINR Laboratories

V. Ivanov (LIT, JINR) JINR Central Information and Computing Complex

V. Ivanov (LIT, JINR) JINR CICC structural scheme

V. Ivanov (LIT, JINR) The protocol between CERN, Russia and JINR on participation in LCG Project was approved in MoU on Worldwide LHC Computing Grid (WLCG) was signed by JINR in October 2007 The tasks of the JINR in the WLCG: WLCG-infrastructure support and development at JINR;WLCG-infrastructure support and development at JINR; participation in WLCG middleware testing/evaluation, participation in WLCG middleware testing/evaluation, participation in Service/Data Challenges, participation in Service/Data Challenges, grid monitoring and accounting tools development; grid monitoring and accounting tools development; FTS-monitoring and testing; FTS-monitoring and testing; participation in ARDA activities in coordination with experiments; participation in ARDA activities in coordination with experiments; JINR LCG portal support and development; JINR LCG portal support and development; HEP applications; HEP applications; MCDB development; MCDB development; support of JINR Member States in the WLCG activities; support of JINR Member States in the WLCG activities; User & Administrator training and education. User & Administrator training and education. LHC Computing Grid Project

V. Ivanov (LIT, JINR) support and development WLCG/EGEE infrastructure; participation in middleware testing/evaluation, participation in Data and Service Challenges, grid monitoring and accounting system development; FTS-monitoring and testing MCDB development; Participation in ARDA activities in coordination with experiments; HEP applications; User & Administrator Training and Education support of JINR member states in the GRID activities. JINR in the LCG/EGEE

V. Ivanov (LIT, JINR) CICC comprises: 65 servers 4 interactive nodes 960 computing nodes, Xeon 5150, 8GB RAM (GEthernet) (160 computing nodes, Xeon X5450, 16GB RAM, InfiniBand). Site name: JINR-LCG2 Internal CICC network – 1Gbit/sec Operating system -Scientific Linux CERN 4.6 (transfer to Scientific Linux SL release 5.2.) ; Middleware version GLITE-3.1 File Systems – AFS (the Andrew File System) for user Software and home directories is a world-wide distributed file system. AFS permits to share easily files in an heterogeneous distributed environment (UNIXes, NT) with a unique authentication scheme (Kerberos). dCache- for data. User registration system – Kerberos 5 ( AFS use Kerberos 5 for authentication ) JINR WLCG infrastructure

V. Ivanov (LIT, JINR) JINR provides the following services in the WLCG environment: Basic services: - Berkley DB Information Index (top level BDII); - site BDII; - 2 x Computing Element (CE); - Proxy Server (PX); - 2 x Workload Management System (WMS); - Logging&Bookkeeping Service (LB); - RGMA-based monitoring system collector server (MON-box); - LCG File Catalog (LFC); - Storage Element (SE), dCache 400 TB, 4 x gridftp door, 14 x pool; - 4 x User Interface (UI), installed in AFS. Special Services - VO boxes for ALICE and for CMS; ROCMON; PPS and testing infrastructure - Pre-production gLite version; Software for VOs: dCache xrootd door, AliROOT, ROOT, GEANT packages for ALICE; ATLAS packages; CMSSW packages for CMS and DaVinchi, Gauss packages for LHCb. JINR WLCG infrastructure

V. Ivanov (LIT, JINR) dCache in JINR Internet / GRID lxfs07 lxpub01 lxfs07 wn… lxfs07 lxfs71 RAID 500 TB Admin Doors local WNs JINR Backbone Protocols gFTP, SRM, XROOT DCAPPNFS lxfs07 rda02 Pools

V. Ivanov (LIT, JINR) Frames for Grid cooperation of JINR in 2009 Worldwide LHC Computing Grid (WLCG); Enabling Grids for E-sciencE (EGEE); RDIG Development (Project of FASI) CERN-RFBR project “GRID Monitoring from VO perspective” BMBF grant “Development of the GRID-infrastructure and tools to provide joint investigations performed with participation of JINR and German research centers” “Development of Grid segment for the LHC experiments” was supported in frames of JINR-South Africa cooperation agreement; NATO project "DREAMS-ASIA“ (Development of gRid EnAbling technology in Medicine&Science for Central ASIA); JINR-Romania cooperation Hulubei-Meshcheryakov programme Project "SKIF-GRID" (Program of Belarussian-Russian Union State). Project GridNNN (National Nanotechnological Net) We work in close cooperation and provide support to our partners in Armenia, Belarus, Bulgaria, Czech Republic, Georgia, Germany, Poland, Romania, South Africa, Ukraine, Uzbekistan.

V. Ivanov (LIT, JINR) Development of the JINR Grid-environment – Network level: links between Moscow and Dubna on the basis of state-of-the-art technologies DWDM and 10Gb Ethernet. JINR Local area network : JINR High-speed backbone construction – 10Gbps Resource level: requirements of the LHC experiments stimulate the development of a global Grid-infrastructure, together with the resource centers of all the cooperating organizations. First of all, this is of primary concern for such large research centers as the JINR. To reach effective processing and analysis of the experimental data, further increase in the JINR CICC performance and disk space is needed.

Now the RDIG infrastructure comprises 15 Resource Centers with > 7000 kSI2K CPU and > 1850 TB of disc storage. The RDIG infrastructure RDIG Resource Centres: – ITEP – JINR-LCG2 – Kharkov-KIPT – RRC-KI – RU-Moscow-KIAM – RU-Phys-SPbSU – RU-Protvino-IHEP – RU-SPbSU – Ru-Troitsk-INR – ru-IMPB-LCG2 – ru-Moscow-FIAN – ru-Moscow-GCRAS – ru-Moscow-MEPHI – ru-PNPI-LCG2 – ru-Moscow-SINP

V. Ivanov (LIT, JINR) RDIG monitoring&accounting Monitored values CPUs - total /working / down/ free / busy Jobs - running / waiting Storage space - used / available Network - Available bandwidth Accounting values Number of submitted jobs Used CPU time Totally sum in seconds Normalized (with WNs productivity) Average time per job Waiting time Totally sum in seconds Average ratio waiting/used CPU time per job Physical memory Average per job JINR CICC Monitoring – allows to keep an eye on parameters of Grid sites' operation in real time Accounting - resources utilization on Grid sites by virtual organizations and single users

V. Ivanov (LIT, JINR) Production Normalised CPU time per EGEE Regions (June-August 2009)

V. Ivanov (LIT, JINR) RDIG Normalised CPU time per Site and per VO (June-September 2009)

V. Ivanov (LIT, JINR) 24 Production Normalised CPU time per EGEE site for VO LHC (June – August 2009) GRID-site CPU time Num CPU time/N FZK-LCG 6,039, INFN-T1 3,493, CERN-PROD 3,309, CYFRONET-LCG 3,198, PIC 3,037, JINR 2,938, IN2P3-CC 2,668, RAL-LCG2 2,629, UKI-GLASGOW 2,610, UKI-LT2-IC-HEP 2,533,

V. Ivanov (LIT, JINR) JINR-LCG2 in RDIG JINR - 40% of total RDIG CPU time Normalised CPU time per VO at JINR site

V. Ivanov (LIT, JINR) Further participation in the WLCG project Support and development of the JINR WLCG-segment in frames of global WLCG infrastructure; Support and development of the JINR WLCG-segment in frames of global WLCG infrastructure; Participation in Service/Data Challenges in coordination with LHC experiments; Participation in Service/Data Challenges in coordination with LHC experiments; WLCG middleware testing/evaluation; WLCG middleware testing/evaluation; Grid-monitoring the LCG-infrastructure at JINR and other sites of the Russian Tier2 cluster; Grid-monitoring the LCG-infrastructure at JINR and other sites of the Russian Tier2 cluster; Evaluation of new Grid technologies in context of their usage in WLCG; Evaluation of new Grid technologies in context of their usage in WLCG; MCDB development (data base structure; basic modules; MCDB development (data base structure; basic modules; Participation in ARDA activities in coordination with experiments; Participation in ARDA activities in coordination with experiments; JINR WLCG web-portal improvement; JINR WLCG web-portal improvement; User’s support to stimulate their active usage of WLCG resources (courses, lectures, trainings, publication of user guides in Russian); User’s support to stimulate their active usage of WLCG resources (courses, lectures, trainings, publication of user guides in Russian); Support of JINR member states in WLCG activities. Support of JINR member states in WLCG activities.

V. Ivanov (LIT, JINR) Applied level developments of the JINR Grid-environment (I) 1.The applied level covers the user applications working in a virtual organization (VO) environment which comprises both users and owners of computing resources. 2.In the existing Grid-systems, a VO defines a collaboration of specialists in some area, who combine their efforts to achieve a common aim. 3.The virtual organization is a flexible structure that can be formed dynamically and may have a limited life-time. VOs working within the WLCG project are the VOs on the LHC experiments - ATLAS, CMS, Alice, LHCb, the first three being carried out with the noticeable and direct participation of the JINR. Nowadays, as a Grid-segment of the EGEE/RDIG, the JINR CICC supports computations of the virtual organizations registered in RDIG ( LHC experiments, BioMed, PHOTON, eEarth, Fusion, HONE, Panda. VOs working within the WLCG project are the VOs on the LHC experiments - ATLAS, CMS, Alice, LHCb, the first three being carried out with the noticeable and direct participation of the JINR. Nowadays, as a Grid-segment of the EGEE/RDIG, the JINR CICC supports computations of the virtual organizations registered in RDIG ( LHC experiments, BioMed, PHOTON, eEarth, Fusion, HONE, Panda. In the future, as the interest arises at a large-scale level, VOs can be organized at JINR in the fields of nuclear physics and condensed matter physics and, most probably, in the new promising direction related to the research of the nanostructure properties. In the future, as the interest arises at a large-scale level, VOs can be organized at JINR in the fields of nuclear physics and condensed matter physics and, most probably, in the new promising direction related to the research of the nanostructure properties.

V. Ivanov (LIT, JINR) The creation of new VOs gets possible and necessary under maturation of the algorithmic approaches to the problem solution, the development of corresponding mathematical methods and tools: methods and tools for simulation of physical processes and analysis of experimental data software and computer complexes for experimental data processing; methods and tools for simulation of physical processes and analysis of experimental data software and computer complexes for experimental data processing; numerical methods, algorithms and software for modeling complex physical systems; numerical methods, algorithms and software for modeling complex physical systems; methods, algorithms and software of computer algebra; methods, algorithms and software of computer algebra; new computing paradigms; new computing paradigms; adaptation of specialized software for solving problems within the Grid- environment. adaptation of specialized software for solving problems within the Grid- environment. Applied level developments of the JINR Grid-environment (II)

V. Ivanov (LIT, JINR) Formation of the Grid environment of the JINR-Member States Creation of the Grid environment of the JINR-participating countries will allow to bring out the research underway at JINR in the field of high energy physics, nuclear physics, solid state and condensed matter physics, biophysics, biosensor (nano)technologies, etc., on a qualitatively new level. Basic directions: Creation of a working group of the representatives of the JINR- participating countries for the development of the main directions of the activity toward forming a joint Grid environment. Creation of a working group of the representatives of the JINR- participating countries for the development of the main directions of the activity toward forming a joint Grid environment. Definition, together with the JINR Member States, of the basic directions of activity toward forming the joint Grid environment. Definition, together with the JINR Member States, of the basic directions of activity toward forming the joint Grid environment. Preparation of corresponding multi-side agreements. Preparation of corresponding multi-side agreements. Development of the network level of the Grid environment of the JINR Member States. Development of the network level of the Grid environment of the JINR Member States. Development of the resource level of the Grid environment of the JINR Member States. Development of the resource level of the Grid environment of the JINR Member States. Formation of the applied level of the Grid environment of the JINR Member States. Formation of the applied level of the Grid environment of the JINR Member States. Preparation and implementation of joint projects. Preparation and implementation of joint projects.

V.Ivanov (LIT, JINR) Thank you for your attention !