J OINT I NSTITUTE FOR N UCLEAR R ESEARCH OFF-LINE DATA PROCESSING GRID-SYSTEM MODELLING FOR NICA 1 Nechaevskiy A. Dubna, 2012.

Slides:



Advertisements
Similar presentations
Building Portals to access Grid Middleware National Technical University of Athens Konstantinos Dolkas, On behalf of Andreas Menychtas.
Advertisements

Database System Concepts and Architecture
Grid simulation (AliEn) Eugen Mudnić Technical university Split -FESB.
1 Databases in ALICE L.Betev LCG Database Deployment and Persistency Workshop Geneva, October 17, 2005.
Grids: Why and How (you might use them) J. Templon, NIKHEF VLV T Workshop NIKHEF 06 October 2003.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
23/04/2008VLVnT08, Toulon, FR, April 2008, M. Stavrianakou, NESTOR-NOA 1 First thoughts for KM3Net on-shore data storage and distribution Facilities VLV.
Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.
Sergey Belov, Tatiana Goloskokova, Vladimir Korenkov, Nikolay Kutovskiy, Danila Oleynik, Artem Petrosyan, Roman Semenov, Alexander Uzhinskiy LIT JINR The.
Grid Information Systems. Two grid information problems Two problems  Monitoring  Discovery We can use similar techniques for both.
Ian Fisk and Maria Girone Improvements in the CMS Computing System from Run2 CHEP 2015 Ian Fisk and Maria Girone For CMS Collaboration.
CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.
Chapter 3: Operating-System Structures System Components Operating System Services System Calls System Programs System Structure Virtual Machines System.
ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, , JINR, Dubna.
Test Of Distributed Data Quality Monitoring Of CMS Tracker Dataset H->ZZ->2e2mu with PileUp - 10,000 events ( ~ 50,000 hits for events) The monitoring.
Polish Infrastructure for Supporting Computational Science in the European Research Space QoS provisioning for data-oriented applications in PL-Grid D.
Alexandre A. P. Suaide VI DOSAR workshop, São Paulo, 2005 STAR grid activities and São Paulo experience.
LOGO Scheduling system for distributed MPD data processing Gertsenberger K. V. Joint Institute for Nuclear Research, Dubna.
Robert Fourer, Jun Ma, Kipp Martin Copyright 2006 An Enterprise Computational System Built on the Optimization Services (OS) Framework and Standards Jun.
WP9 Resource Management Current status and plans for future Juliusz Pukacki Krzysztof Kurowski Poznan Supercomputing.
+ discussion in Software WG: Monte Carlo production on the Grid + discussion in TDAQ WG: Dedicated server for online services + experts meeting (Thusday.
CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services Job Monitoring for the LHC experiments Irina Sidorova (CERN, JINR) on.
Finnish DataGrid meeting, CSC, Otaniemi, V. Karimäki (HIP) DataGrid meeting, CSC V. Karimäki (HIP) V. Karimäki (HIP) Otaniemi, 28 August, 2000.
Silberschatz, Galvin and Gagne  Operating System Concepts Chapter 3: Operating-System Structures System Components Operating System Services.
ALICE Upgrade for Run3: Computing HL-LHC Trigger, Online and Offline Computing Working Group Topical Workshop Sep 5 th 2014.
Instrumentation of the SAM-Grid Gabriele Garzoglio CSC 426 Research Proposal.
6/26/01High Throughput Linux Clustering at Fermilab--S. Timm 1 High Throughput Linux Clustering at Fermilab Steven C. Timm--Fermilab.
Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, October 2003.
November SC06 Tampa F.Fanzago CRAB a user-friendly tool for CMS distributed analysis Federica Fanzago INFN-PADOVA for CRAB team.
LOGO PROOF system for parallel MPD event processing Gertsenberger K. V. Joint Institute for Nuclear Research, Dubna.
Distributed monitoring system. Why Monitor? Solve them! Identify Problems Ensure conduct Requirements Manage many computers Spot trends in the system.
And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR
Module 3 Planning and Deploying Mailbox Services.
EGEE is a project funded by the European Union under contract IST HEP Use Cases for Grid Computing J. A. Templon Undecided (NIKHEF) Grid Tutorial,
Development of the distributed monitoring system for the NICA cluster Ivan Slepov (LHEP, JINR) Mathematical Modeling and Computational Physics Dubna, Russia,
LOGO Development of the distributed computing system for the MPD at the NICA collider, analytical estimations Mathematical Modeling and Computational Physics.
Nanco: a large HPC cluster for RBNI (Russell Berrie Nanotechnology Institute) Anne Weill – Zrahia Technion,Computer Center October 2008.
FRANEC and BaSTI grid integration Massimo Sponza INAF - Osservatorio Astronomico di Trieste.
HIGUCHI Takeo Department of Physics, Faulty of Science, University of Tokyo Representing dBASF Development Team BELLE/CHEP20001 Distributed BELLE Analysis.
HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.
Copyright 2007, Information Builders. Slide 1 Machine Sizing and Scalability Mark Nesson, Vashti Ragoonath June 2008.
Status report of the KLOE offline G. Venanzoni – LNF LNF Scientific Committee Frascati, 9 November 2004.
Predrag Buncic Future IT challenges for ALICE Technical Workshop November 6, 2015.
Tier3 monitoring. Initial issues. Danila Oleynik. Artem Petrosyan. JINR.
Predrag Buncic CERN ALICE Status Report LHCC Referee Meeting 01/12/2015.
Ian Bird WLCG Networking workshop CERN, 10 th February February 2014
HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino.
03/09/2007http://pcalimonitor.cern.ch/1 Monitoring in ALICE Costin Grigoras 03/09/2007 WLCG Meeting, CHEP.
Gennaro Tortone, Sergio Fantinel – Bologna, LCG-EDT Monitoring Service DataTAG WP4 Monitoring Group DataTAG WP4 meeting Bologna –
Markus Frank (CERN) & Albert Puig (UB).  An opportunity (Motivation)  Adopted approach  Implementation specifics  Status  Conclusions 2.
D.Spiga, L.Servoli, L.Faina INFN & University of Perugia CRAB WorkFlow : CRAB: CMS Remote Analysis Builder A CMS specific tool written in python and developed.
Mobile Analyzer A Distributed Computing Platform Juho Karppinen Helsinki Institute of Physics Technology Program May 23th, 2002 Mobile.
Joint Institute for Nuclear Research Synthesis of the simulation and monitoring processes for the data storage and big data processing development in physical.
Meeting with University of Malta| CERN, May 18, 2015 | Predrag Buncic ALICE Computing in Run 2+ P. Buncic 1.
A web based tool for estimation of Gage R&R and Measurement Uncertainty Siva Venkatachalam & Dr. Jay Raja Center for Precision Metrology The University.
M. Caprini IFIN-HH Bucharest DAQ Control and Monitoring - A Software Component Model.
ALICE Physics Data Challenge ’05 and LCG Service Challenge 3 Latchezar Betev / ALICE Geneva, 6 April 2005 LCG Storage Management Workshop.
G. Russo, D. Del Prete, S. Pardi Kick Off Meeting - Isola d'Elba, 2011 May 29th–June 01th A proposal for distributed computing monitoring for SuperB G.
Cofax Scalability Document Version Scaling Cofax in General The scalability of Cofax is directly related to the system software, hardware and network.
Grid technologies for large-scale projects N. S. Astakhov, A. S. Baginyan, S. D. Belov, A. G. Dolbilov, A. O. Golunov, I. N. Gorbunov, N. I. Gromova, I.
The Tier-1 center for CMS experiment at LIT JINR N. S. Astakhov, A. S. Baginyan, S. D. Belov, A. G. Dolbilov, A. O. Golunov, I. N. Gorbunov, N. I. Gromova,
PROOF system for parallel NICA event processing
Accounting at the T1/T2 Sites of the Italian Grid
Grid Computing.
Simulation use cases for T2 in ALICE
ALICE Computing Upgrade Predrag Buncic
University of Technology
ExaO: Software Defined Data Distribution for Exascale Sciences
Chapter 2: Operating-System Structures
Chapter 2: Operating-System Structures
Presentation transcript:

J OINT I NSTITUTE FOR N UCLEAR R ESEARCH OFF-LINE DATA PROCESSING GRID-SYSTEM MODELLING FOR NICA 1 Nechaevskiy A. Dubna, 2012

A GENDA NICA off-line data processing parameters Tasks for simulation Simulation platform choice Model efficiency estimation First results Conclusion 2

D ATA P ROCESSING S CHEMA F OR NICA MPD 3 NICA’s data flow parameters: high speed of the events generation (to 6 KHZ), in the central collision of Au-Au about 1000 particles are formed, the size of the file with modelled information from detectors for events occupies about 5 TB. №ParameterValue 1Speed of data collection from all detector’s components 4.7 GB/s 2Duration of the set of statistics period within a year 120 days 3Frequency of the event emergence on an installation exit 6 KHz 4Dead time after event emergence1 cicle (50%) 5Average of tracks in an event500 6Average of particles collisions20 7Average of bytes on each collision45 8Average time of event's reconstruction on the processor in capacity 1КSI2K 2 s. MPD parameters

S OURCE D ATA 4 №RequirementsValue 1Quantity of events to processing in a year1.87 е10 2Total data volume to storage in a year8,4 PB 3Total Disk space in case storage is RAID6 (+25%) in a year10 PB 4Total CPUs in grid structure, minimum necessary for data recovery with the speed equal to a set of events, proceeding from 7000 thousand astronomical clock of work a year Numbers of grid sites20 6Minimum of Data transfer speed from JINR to Sites2,5 Gb/s The specification of requirements to NICA experiment off-line data processing The expected number of data processed events is about 19 billions. If data transfer speed from sensors is 4.7 GB/s, the total amount of source data can be estimated as 30 PB annually, or 8.4 PB after processing.

G RID FOR EXPERIMENTS 5 Hierarchical grid infrastructure with some computing centers Tier 0/1/2 already used in ALICE experiment and others. PANDA experiment wants to use it also. Questions For Simulation Grid Infrastructure Architecture? Number Resource centers? Amount of the Resources? Capacity of the network? Resource distribution between users groups? etc. Urgency Recommendation and specification for NICA grid infrastructure creation

S IMULATION T ASKS 6 Task 1. Task 2.

G RID S IM S IMULATION P ACKAGE 7 Allows to simulate various classes of heterogeneous resources, users, applications and brokers There are no restrictions on jobs number which can be sent on a resource; Capacity of a network between resources can be set; System supports simulation of statistical and dynamic schedulers; Statistics of all or the chosen operations can be registered Implemented in Java Configuration files are used to set simulation’s parameters Source code is available A lot of examples of the GridSim using Multilevel architecture allows to add new components easily GridSim Architecture

M ODEL EFFICIENCY ESTIMATION 8 Parameters of the model efficiency: a) Average network loading by days [%] b) Numbers of the running /waiting jobs c) Number of uses CPUs d)Total Data transfers in hours [GB] e)Total Storage uses [%] f) Cluster uses [%] j) Refused CPUs [%]

M ODEL C OMPONENTS 1. User Interface (edit/add model) 2. MySQL database to save simulation parameters 3. Simulation System 4. Results Visualization Tools 9

10 T EST SIMULATION Clusters: 1 Machine 2 CPUs Users: 1 Jobs: 10

E XAMPLE OF G RAPHIC R EPRESENTATION O F T HE S IMULATION R ESULTS Waiting and Running Jobs 2. Average Clusters Usage

D ONE ! The web interface of the model editing with one test scenario of the grid work is created key parameters of the model estimate are allocated; Results visualization tools are created; Simulation passed debugging and verification phase. 12

C ONCLUSION 13 to estimate some architectures (parameters) of the data processing system by changing entrance data only; library of scenarios (Data processing, architectures, other) will allow to compare various technical solutions and to choose optimum. The model will allow : ― the user interface development; ― debugging the model in client-server architecture ― development of a scenarios sets of grid systems work ― user’s editing and adding grid model parameters Plans:

14 Questions?