Dzero MC production on LCG How to live in two worlds (SAM and LCG)

Slides:

Advertisements

Similar presentations

DataTAG WP4 Meeting CNAF Jan 14, 2003 Interfacing AliEn and EDG 1/13 Stefano Bagnasco, INFN Torino Interfacing AliEn to EDG Stefano Bagnasco, INFN Torino.

Advertisements

GridPP July 2003Stefan StonjekSlide 1 SAM middleware components Stefan Stonjek University of Oxford 7 th GridPP Meeting 02 nd July 2003 Oxford.

Physics with SAM-Grid Stefan Stonjek University of Oxford 6 th GridPP Meeting 30 th January 2003 Coseners House.

EU 2nd Year Review – Jan – Title – n° 1 WP1 Speaker name (Speaker function and WP ) Presentation address e.g.

Workload management Owen Maroney, Imperial College London (with a little help from David Colling)

NIKHEF Testbed 1 Plans for the coming three months.

Rod Walker IC 13th March 2002 SAM-Grid Middleware  SAM.  JIM.  RunJob.  Conclusions. - Rod Walker,ICL.

INFSO-RI Enabling Grids for E-sciencE EGEE Middleware The Resource Broker EGEE project members.

The DataGrid Project NIKHEF, Wetenschappelijke Jaarvergadering, 19 December 2002

Basic Grid Job Submission Alessandra Forti 28 March 2006.

The Sam-Grid project Gabriele Garzoglio ODS, Computing Division, Fermilab PPDG, DOE SciDAC ACAT 2002, Moscow, Russia June 26, 2002.

A tool to enable CMS Distributed Analysis

JIM Deployment for the CDF Experiment M. Burgon-Lyon 1, A. Baranowski 2, V. Bartsch 3,S. Belforte 4, G. Garzoglio 2, R. Herber 2, R. Illingworth 2, R.

Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug

Jun 29, 20101/25 Storage Evaluation on FG, FC, and GPCF Jun 29, 2010 Gabriele Garzoglio Computing Division, Fermilab Overview Introduction Lustre Evaluation:

The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.

SAMGrid – A fully functional computing grid based on standard technologies Igor Terekhov for the JIM team FNAL/CD/CCF.

Zhiling Chen (IPP-ETHZ) Doktorandenseminar June, 4 th, 2009.

The SAMGrid Data Handling System Outline:  What Is SAMGrid?  Use Cases for SAMGrid in Run II Experiments  Current Operational Load  Stress Testing.

Grid Job and Information Management (JIM) for D0 and CDF Gabriele Garzoglio for the JIM Team.

November 7, 2001Dutch Datagrid SARA 1 DØ Monte Carlo Challenge A HEP Application.

Building a distributed software environment for CDF within the ESLEA framework V. Bartsch, M. Lancaster University College London.

CDF Grid Status Stefan Stonjek 05-Jul th GridPP meeting / Durham.

Deploying and Operating the SAM-Grid: lesson learned Gabriele Garzoglio for the SAM-Grid Team Sep 28, 2004.

3rd June 2004 CDF Grid SAM:Metadata and Middleware Components Mòrag Burgon-Lyon University of Glasgow.

CHEP 2003Stefan Stonjek1 Physics with SAM-Grid Stefan Stonjek University of Oxford CHEP th March 2003 San Diego.

CHEP'07 September D0 data reprocessing on OSG Authors Andrew Baranovski (Fermilab) for B. Abbot, M. Diesburg, G. Garzoglio, T. Kurca, P. Mhashilkar.

Grid Technologies  Slide text. What is Grid?  The World Wide Web provides seamless access to information that is stored in many millions of different.

- Iain Bertram R-GMA and DØ Iain Bertram RAL 13 May 2004 Thanks to Jeff Templon at Nikhef.

1 st December 2003 JIM for CDF 1 JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow.

Interactive Job Monitor: CafMon kill CafMon tail CafMon dir CafMon log CafMon top CafMon ps LcgCAF: CDF submission portal to LCG resources Francesco Delli.

INFSO-RI Enabling Grids for E-sciencE Workload Management System Mike Mineter

Belle MC Production on Grid 2 nd Open Meeting of the SuperKEKB Collaboration Soft/Comp session 17 March, 2009 Hideyuki Nakazawa National Central University.

SAMGrid as a Stakeholder of FermiGrid Valeria Bartsch Computing Division Fermilab.

GridPP18 Glasgow Mar 07 DØ – SAMGrid Where’ve we come from, and where are we going? Evolution of a ‘long’ established plan Gavin Davies Imperial College.

DIRAC Review (13 th December 2005)Stuart K. Paterson1 DIRAC Review Exposing DIRAC Functionality.

Status of the LHCb MC production system Andrei Tsaregorodtsev, CPPM, Marseille DataGRID France workshop, Marseille, 24 September 2002.

November SC06 Tampa F.Fanzago CRAB a user-friendly tool for CMS distributed analysis Federica Fanzago INFN-PADOVA for CRAB team.

The SAM-Grid and the use of Condor-G as a grid job management middleware Gabriele Garzoglio for the SAM-Grid Team Fermilab, Computing Division.

22 nd September 2003 JIM for CDF 1 JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow.

4 March 2004GridPP 9th Collaboration Meeting SAMGrid:JIM and CDF Development CDF Accepts the Need for the Grid –Requirements How to Meet the Need –Status.

The SAM-Grid / LCG Interoperability Test Bed Gabriele Garzoglio ( ) Speaker: Pierre Girard (

What is SAM-Grid? Job Handling Data Handling Monitoring and Information.

Data reprocessing for DZero on the SAM-Grid Gabriele Garzoglio for the SAM-Grid Team Fermilab, Computing Division.

GridPP11 Liverpool Sept04 SAMGrid GridPP11 Liverpool Sept 2004 Gavin Davies Imperial College London.

Jan 10, 2007, Clermont-Ferrand Tibor Kurca, Tutorial Grille1 DØ Computing Introduction - Fermilab & Tevatron & DØ Experiment DØ Computing Model 1. data.

AliEn AliEn at OSC The ALICE distributed computing environment by Bjørn S. Nilsen The Ohio State University.

Testing and integrating the WLCG/EGEE middleware in the LHC computing Simone Campana, Alessandro Di Girolamo, Elisa Lanciotti, Nicolò Magini, Patricia.

Eileen Berman. Condor in the Fermilab Grid FacilitiesApril 30, 2008  Fermi National Accelerator Laboratory is a high energy physics laboratory outside.

December 07, 2006Parag Mhashilkar, Fermilab1 Samgrid – OSG Interoperability Parag Mhashilkar, Fermi National Accelerator Laboratory.

April 25, 2006Parag Mhashilkar, Fermilab1 Resource Selection in OSG & SAM-On-The-Fly Parag Mhashilkar Fermi National Accelerator Laboratory Condor Week.

The DataGrid Project NIKHEF, Wetenschappelijke Jaarvergadering, 19 December 2002

Distributed Analysis Tutorial Dietrich Liko. Overview  Three grid flavors in ATLAS EGEE OSG Nordugrid  Distributed Analysis Activities GANGA/LCG PANDA/OSG.

+ Support multiple virtual environment for Grid computing Dr. Lizhe Wang.

DZero Monte Carlo Production Ideas for CMS Greg Graham Fermilab CD/CMS 1/16/01 CMS Production Meeting.

D.Spiga, L.Servoli, L.Faina INFN & University of Perugia CRAB WorkFlow : CRAB: CMS Remote Analysis Builder A CMS specific tool written in python and developed.

July 26, 2007Parag Mhashilkar, Fermilab1 DZero On OSG: Site And Application Validation Parag Mhashilkar, Fermi National Accelerator Laboratory.

Sept Wyatt Merritt Run II Computing Review1 Status of SAMGrid / Future Plans for SAMGrid  Brief introduction to SAMGrid  Status and deployments.

Enabling Grids for E-sciencE Work Load Management & Simple Job Submission Practical Shu-Ting Liao APROC, ASGC EGEE Tutorial.

5/12/06T.Kurca - D0 Meeting FNAL1 p20 Reprocessing Introduction Computing Resources Architecture Operational Model Technical Issues Operational Issues.

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations.

Grid Computing: Running your Jobs around the World

Monte Carlo Production and Reprocessing at DZero

Production Resources & Issues p20.09 MC-data Regeneration

Job workflow Pre production operations:

5. Job Submission Grid Computing.

Tibor Kurca, Tutorial CPPM

YongPyong-High Jan We appreciate that you give an opportunity to have this talk. Our Belle II computing group would like to report on.

DØ MC and Data Processing on the Grid

Status and plans for bookkeeping system and production tools

Presentation transcript:

Dzero MC production on LCG How to live in two worlds (SAM and LCG)

What is SAM? SAM: Sequential Access by Metadata –file store –sam stations –replica catalog –metadata database –optimizes file delivery to consumers batch interface 2Grid meeting Jan 28, 2008

What is SAMGrid? SAM: Sequential Access by Metadata JIM: Job Information and Monitoring Fermigrid OSG LCG … 3Grid meeting Jan 28, 2008

resource broker user interface SAM station dorsvlegel.nikhef.nl 4Grid meeting Jan 28, 2008 Until January 2007

What is SAMGrid? SAM: Sequential Access by Metadata JIM: Job Information and Monitoring Fermigrid OSG LCG … 5Grid meeting Jan 28, 2008

submission site execution site jim client SAM station cluster (Nikhef) FNAL Wuppertal Lyon 6Grid meeting Jan 28, 2008 From January 2007

JIM client Submits job to submission site –input.jdl specifies: request number cluster 7Grid meeting Jan 28, 2008

input.jdl lcg_requirement_string = tbn20.nikhef.nl:2119/jobmanager-pbs-qlong job_type = dzero_monte_carlo runjob_requestid = d0_release_version = p jobfiles_dataset = sg_p v3_d0r _cfv minbias_dataset = zerob_p _RunIIbMC_REALsample_2 runjob_numevts = events_per_file = 250 station_name = ccin2p3-grid1 8Grid meeting Jan 28, 2008

Submission site Submits cluster job to execution site –Collects non physical output (logs, …) Collects monitoring data Monitoring web pages 9Grid meeting Jan 28, 2008

Execution site User interface Submits individual jobs to cluster (edg-job-submit) Resource broker 10Grid meeting Jan 28, 2008

SAM station Delivers exec files delivers input files stores metadata and data in SAM 11Grid meeting Jan 28, 2008

Output from WN To SAM database: –metadata of intermediate files and MC output To SAM store: –reconstructed MC output To submission site –log files 12Grid meeting Jan 28, 2008

Pro’s Always in line with Fermi lab (no home grown scripts) less labor intensive better monitoring 13Grid meeting Jan 28, 2008

Cons Not scalable SAM in2p3 Resource wuppertal User wuppertal less flexible (no home grown scripts) 14Grid meeting Jan 28, 2008

Scalability SAM in2p3 ic.ac.uk Resource wuppertal –Moved to dorsvlegel.nikhef.nl User wuppertal 15Grid meeting Jan 28, 2008

Until January 2007 –UI + SAM station Until January 2008 –UI + JIM station Now –UI + JIM station + RB (dzero only) 16Grid meeting Jan 28, 2008

Monitoring Go to Click submission sites Click samgrid.fnal.gov 17Grid meeting Jan 28, 2008