1 L.Didenko Joint ALICE/STAR meeting HPSS and Production Management 9 April, 2000.

Slides:

Advertisements

Similar presentations

31/03/00 CMS(UK)Glenn Patrick What is the CMS(UK) Data Model? Assume that CMS software is available at every UK institute connected by some infrastructure.

Advertisements

SWIM WEB PORTAL by Dipti Aswath SWIM Meeting ORNL Oct 15-17, 2007.

Production Planning Eric van Herwijnen Thursday, 20 june 2002.

Multiple Tiers in Action

K.Harrison CERN, 23rd October 2002 HOW TO COMMISSION A NEW CENTRE FOR LHCb PRODUCTION - Overview of LHCb distributed production system - Configuration.

Magda – Manager for grid-based data Wensheng Deng Physics Applications Software group Brookhaven National Laboratory.

Experience of a low-maintenance distributed data management system W.Takase 1, Y.Matsumoto 1, A.Hasan 2, F.Di Lodovico 3, Y.Watase 1, T.Sasaki 1 1. High.

The ATLAS Production System. The Architecture ATLAS Production Database Eowyn Lexor Lexor-CondorG Oracle SQL queries Dulcinea NorduGrid Panda OSGLCG The.

Test Of Distributed Data Quality Monitoring Of CMS Tracker Dataset H->ZZ->2e2mu with PileUp - 10,000 events ( ~ 50,000 hits for events) The monitoring.

Alexandre A. P. Suaide VI DOSAR workshop, São Paulo, 2005 STAR grid activities and São Paulo experience.

GLAST LAT ProjectDOE/NASA Baseline-Preliminary Design Review, January 8, 2002 K.Young 1 LAT Data Processing Facility Automatically process Level 0 data.

D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,

Technical Review - PHP Jang Jaeseok. Overview Introduction  What is PHP?  History of PHP  What can do with PHP?  Advantages of PHP PHP syntax.

Marianne BargiottiBK Workshop – CERN - 6/12/ Bookkeeping Meta Data catalogue: present status Marianne Bargiotti CERN.

Bookkeeping Tutorial. Bookkeeping & Monitoring Tutorial2 Bookkeeping content  Contains records of all “jobs” and all “files” that are created by production.

The huge amount of resources available in the Grids, and the necessity to have the most up-to-date experimental software deployed in all the sites within.

Management System of Event Processing and Data Files Based on XML Software Tools at Belle Ichiro Adachi, Nobu Katayama, Masahiko Yokoyama IPNS, KEK, Tsukuba,

Status of the LHCb MC production system Andrei Tsaregorodtsev, CPPM, Marseille DataGRID France workshop, Marseille, 24 September 2002.

The ALICE short-term use case DataGrid WP6 Meeting Milano, 11 Dec 2000Piergiorgio Cerello 1 Physics Performance Report (PPR) production starting in Feb2001.

CERN IT Department CH-1211 Genève 23 Switzerland t Monitoring: Tracking your tasks with Task Monitoring PAT eLearning – Module 11 Edward.

Andrey Meeting 7 October 2003 General scheme: jobs are planned to go where data are and to less loaded clusters SUNY.

Claudio Grandi INFN Bologna CHEP'03 Conference, San Diego March 27th 2003 BOSS: a tool for batch job monitoring and book-keeping Claudio Grandi (INFN Bologna)

BNL DDM Status Report Hironori Ito Brookhaven National Laboratory.

PHENIX and the data grid >400 collaborators Active on 3 continents + Brazil 100’s of TB of data per year Complex data with multiple disparate physics goals.

Offline report – 7TeV data taking period (Mar.30 – Apr.6) ALICE SRC April 6, 2010.

Archive Access Tool Review of SSS Readiness for EVLA Shared Risk Observing, June 5, 2009 John Benson Scientist.

1 GCA Application in STAR GCA Collaboration Grand Challenge Architecture and its Interface to STAR Sasha Vaniachine presenting for the Grand Challenge.

The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg,

DDM Monitoring David Cameron Pedro Salgado Ricardo Rocha.

PPDG update l We want to join PPDG l They want PHENIX to join NSF also wants this l Issue is to identify our goals/projects Ingredients: What we need/want.

DØ Data Handling & Access The DØ Meta-Data Browser Pushpa Bhat Fermilab June 4, 2001.

Unified scripts ● Currently they are composed of a main shell script and a few auxiliary ones that handle mostly the local differences. ● Local scripts.

February 28, 2003Eric Hjort PDSF Status and Overview Eric Hjort, LBNL STAR Collaboration Meeting February 28, 2003.

STAR C OMPUTING STAR Analysis Operations and Issues Torre Wenaus BNL STAR PWG Videoconference BNL August 13, 1999.

Youngil Kim Awalin Sopan Sonia Ng Zeng.  Introduction  Concept of the Project  System architecture  Implementation – HDFS  Implementation – System.

PC clusters in KEK A.Manabe KEK(Japan). 22 May '01LSCC WS '012 PC clusters in KEK s Belle (in KEKB) PC clusters s Neutron Shielding Simulation cluster.

D. Heynderickx DH Consultancy, Leuven, Belgium 22 April 2010EuroPlanet, London, UK.

Alaa El-Deen shaban El-Meshneb Assistant Lecturer Physics Department Sohag University Grid Computing Group 3 D modeling in Grid (by the example of Blender)

Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,

1 fileCatalog, tagDB and GCA A. Vaniachine Grand Challenge STAR fileCatalog, tagDB and Grand Challenge Architecture A. Vaniachine presenting for the Grand.

Bookkeeping Tutorial. 2 Bookkeeping content  Contains records of all “jobs” and all “files” that are produced by production jobs  Job:  In fact technically.

1 A lightweight Monitoring and Accounting system for LHCb DC'04 production V. Garonne R. Graciani Díaz J. J. Saborido Silva M. Sánchez García R. Vizcaya.

Analyzing ever growing datasets in PHENIX Chris Pinkenburg for the PHENIX collaboration.

Large scale data flow in local and GRID environment Viktor Kolosov (ITEP Moscow) Ivan Korolko (ITEP Moscow)

The Gateway Computational Web Portal Marlon Pierce Indiana University March 15, 2002.

FTS monitoring work WLCG service reliability workshop November 2007 Alexander Uzhinskiy Andrey Nechaevskiy.

Pavel Nevski DDM Workshop BNL, September 27, 2006 JOB DEFINITION as a part of Production.

RefDB: The Reference Database for CMS Monte Carlo Production Véronique Lefébure CERN & HIP CHEP San Diego, California 25 th of March 2003.

LHCb GRID Meeting 11/12 Sept Sept LHCb-GRID T. Bowcock 2 AGENDA 9:30 LHCb MC Production –Points SICB Processing Req. Data Storage Data Transfer.

Youngil Kim Awalin Sopan Sonia Ng Zeng.  Introduction  System architecture  Implementation – HDFS  Implementation – System Analysis ◦ System Information.

D.Spiga, L.Servoli, L.Faina INFN & University of Perugia CRAB WorkFlow : CRAB: CMS Remote Analysis Builder A CMS specific tool written in python and developed.

CERN, 7 November 2011 Anton Pytel Slovak Technical University TRIP FROM GENERATORS TO GEOMETRIES.

Using the Grid for the ILC Mokka and Marlin on the Grid ILC Software Meeting, Cambridge 2006.

September 26, 2003K User's Meeting1 CCJ Usage for Belle Monte Carlo production and analysis –CPU time: 170K hours (Aug.1, 02 ~ Aug.22, 03)

Internet/Web Databases

Data Formats and Impact on Federated Access

BESIII data processing

Real Time Fake Analysis at PIC

U.S. ATLAS Grid Production Experience

Quick Look on dCache Monitoring at FNAL

CMS High Level Trigger Configuration Management

ALICE Monitoring

Akiya Miyamoto KEK 1 June 2016

Work report Xianghu Zhao Nov 11, 2014.

To Do: 1.Math Boxes Math Boxes Math Boxes Math Boxes Math Boxes Math Boxes Any Unit 1 MB or pages not Not Done.

The LHCb Software and Computing NSS/IEEE workshop Ph. Charpentier, CERN B00le.

LHCb Computing Model and Data Handling Angelo Carbone 5° workshop italiano sulla fisica p-p ad LHC 31st January 2008.

Job workflow Pre production operations:

Status of CVS repository Production databases Production tools

ATLAS DC2 & Continuous production

Presentation transcript:

1 L.Didenko Joint ALICE/STAR meeting HPSS and Production Management 9 April, 2000

April 9, 2000L.Didenko, ALICE-STAR meeting 2 Volume of data processing (last month experience) Production mySql DataBase Tools for production operation Web page retrieving of production summary and operation control plots

April 9, 2000L.Didenko, ALICE-STAR meeting 3 Volume of data processing Reconstruction of GEANT simulated data: total number of events with number of tracks in average 5478 has been produced for last month total size of DST branch 800 GB, geant branch 2.7 TB, total production output for last month 3.5 TB has been transferred to HPSS daily operation : 120 jobs in average processing, 600 output files ( 5 branches for each job) produced one event production info ( No. of tracks 5624) : CPU = 267 sec, Real Time = 341 sec, DST size = 5.9 MB

April 9, 2000L.Didenko, ALICE-STAR meeting 4 Production MySQL DataBase Table FileCatalog Table JobStatus Script created jobfile and JobID Table ProdOptions GEANT input HPSS Jobfiles to autosubmiton JobID jobfilename Input files LibVersion codes opt DST files from HPSS Script combined DST files with JobID No.Evts done Log files info: job status No Evt Done CPU, RealTime Memory sz Avg No.Trks Avg No.Vert Table JobRelations JobID, input file

April 9, 2000L.Didenko, ALICE-STAR meeting 5 Input/output relations JobID input output input output Postproduction operation

April 9, 2000L.Didenko, ALICE-STAR meeting 6 Input/output relations JobID output input t Concatenation of DST files

April 9, 2000L.Didenko, ALICE-STAR meeting 7 Web page retrieving production summary Table FileCatalog Raw data files size No. Events DST files size No.Events done Web page summary Table FC browser dataset1 FC browser dataset2 FC browser dataset3 FC browser dataset4

April 9, 2000L.Didenko, ALICE-STAR meeting 8 Production operation control plots Table JobStatus Query interface box CGI script retrieving data from JobStatus Table and creates GIF plots Mem size No.TrksCPU Real Time No.Vert

April 9, 2000L.Didenko, ALICE-STAR meeting 9 Tools for production operation (more) Tom Throwe’s scripts for jobs submission, status control and transfering input/output files from/to HPSS automated with info when job done, crashed, aborted or killed script for automated jobs submission script which reading info and resubmit jobfile in case job crashed, aborted or killed for technical reasons script which which scanning production DB and give information about failed jobs (not yet Web presentation) People contributed to design and development of production DB and tools Y.Fisyak, T.Wenaus, Wensheng Deng, L.Barnby,S.Vaniashian