Metadata requirements for HEP Paul Millar. Slide 2 12 September 2007 Metadata requirements for HEP Some of the players in this game... WLCG – Umbrella.

Slides:

Advertisements

Similar presentations

1 ALICE Grid Status David Evans The University of Birmingham GridPP 14 th Collaboration Meeting Birmingham 6-7 Sept 2005.

Advertisements

CMS Applications – Status and Near Future Plans

Review of WLCG Tier-2 Workshop Duncan Rand Royal Holloway, University of London Brunel University.

GridPP9 – 5 February 2004 – Data Management DataGrid is a project funded by the European Union GridPP is funded by PPARC GridPP2: Data and Storage Management.

Metadata Progress GridPP18 20 March 2007 Mike Kenyon.

31/03/00 CMS(UK)Glenn Patrick What is the CMS(UK) Data Model? Assume that CMS software is available at every UK institute connected by some infrastructure.

Plateforme de Calcul pour les Sciences du Vivant SRB & gLite V. Breton.

1 Grid services based architectures Growing consensus that Grid services is the right concept for building the computing grids; Recent ARDA work has provoked.

Regional Computing Centre for Particle Physics Institute of Physics AS CR (FZU) TIER2 of LCG (LHC Computing Grid) 1M. Lokajicek Dell Presentation.

AMI S.A. Datasets… Solveig Albrand. AMI S.A. A set is… A number of things grouped together according to a system of classification, or conceived as forming.

CERN/IT/DB Multi-PB Distributed Databases Jamie Shiers IT Division, DB Group, CERN, Geneva, Switzerland February 2001.

GridPP Steve Lloyd, Chair of the GridPP Collaboration Board.

December 17th 2008RAL PPD Computing Christmas Lectures 11 ATLAS Distributed Computing Stephen Burke RAL.

GridPP: the UK's contribution to the international collaboration building a worldwide Grid, the LHC Computing Grid GridPP – is the system usable? Tony.

Grid Computing Oxana Smirnova NDGF- Lund University R-ECFA meeting in Sweden Uppsala, May 9, 2008.

5 November 2001F Harris GridPP Edinburgh 1 WP8 status for validating Testbed1 and middleware F Harris(LHCb/Oxford)

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America The AMGA metadata catalog with use cases.

3 Sept 2001F HARRIS CHEP, Beijing 1 Moving the LHCb Monte Carlo production system to the GRID D.Galli,U.Marconi,V.Vagnoni INFN Bologna N Brook Bristol.

Nurcan Ozturk University of Texas at Arlington Grid User Training for Local Community TUBITAK ULAKBIM, Ankara, Turkey April 5 - 9, 2010 Overview of ATLAS.

Nick Brook Current status Future Collaboration Plans Future UK plans.

LHC: ATLAS Experiment meeting “Conditions” data challenge Elizabeth Gallas - Oxford - August 29, 2009 XLDB3.

3 June 2004GridPP10Slide 1 GridPP Dissemination Sarah Pearce Dissemination Officer

DOSAR Workshop, Sao Paulo, Brazil, September 16-17, 2005 LCG Tier 2 and DOSAR Pat Skubic OU.

4th February 2004GRIDPP91 LHCb Development Glenn Patrick Rutherford Appleton Laboratory.

ATLAS and GridPP GridPP Collaboration Meeting, Edinburgh, 5 th November 2001 RWL Jones, Lancaster University.

Bookkeeping Tutorial. Bookkeeping & Monitoring Tutorial2 Bookkeeping content  Contains records of all “jobs” and all “files” that are created by production.

ATLAS Database Operations Invited talk at the XXI International Symposium on Nuclear Electronics & Computing Varna, Bulgaria, September 2007 Alexandre.

Databases E. Leonardi, P. Valente. Conditions DB Conditions=Dynamic parameters non-event time-varying Conditions database (CondDB) General definition:

F. Fassi, S. Cabrera, R. Vives, S. González de la Hoz, Á. Fernández, J. Sánchez, L. March, J. Salt, A. Lamas IFIC-CSIC-UV, Valencia, Spain Third EELA conference,

А.Минаенко Совещание по физике и компьютингу, 03 февраля 2010 г. НИИЯФ МГУ, Москва Текущее состояние и ближайшие перспективы компьютинга для АТЛАСа в России.

Caitriana Nicholson, CHEP 2006, Mumbai Caitriana Nicholson University of Glasgow Grid Data Management: Simulations of LCG 2008.

…building the next IT revolution From Web to Grid…

Metadata Mòrag Burgon-Lyon University of Glasgow.

ATLAS is a general-purpose particle physics experiment which will study topics including the origin of mass, the processes that allowed an excess of matter.

LCG ARDA status Massimo Lamanna 1 ARDA in a nutshell ARDA is an LCG project whose main activity is to enable LHC analysis on the grid ARDA is coherently.

Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010.

WLCG and the India-CERN Collaboration David Collados CERN - Information technology 27 February 2014.

Post-DC2/Rome Production Kaushik De, Mark Sosebee University of Texas at Arlington U.S. Grid Phone Meeting July 13, 2005.

Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,

Performance of The NorduGrid ARC And The Dulcinea Executor in ATLAS Data Challenge 2 Oxana Smirnova (Lund University/CERN) for the NorduGrid collaboration.

Bookkeeping Tutorial. 2 Bookkeeping content  Contains records of all “jobs” and all “files” that are produced by production jobs  Job:  In fact technically.

Predrag Buncic Future IT challenges for ALICE Technical Workshop November 6, 2015.

Testing and integrating the WLCG/EGEE middleware in the LHC computing Simone Campana, Alessandro Di Girolamo, Elisa Lanciotti, Nicolò Magini, Patricia.

M. Oldenburg GridPP Metadata Workshop — July 4–7 2006, Oxford University 1 Markus Oldenburg GridPP Metadata Workshop July 4–7 2006, Oxford University ALICE.

TAGS in the Analysis Model Jack Cranshaw, Argonne National Lab September 10, 2009.

tons, 150 million sensors generating data 40 millions times per second producing 1 petabyte per second The ATLAS experiment.

U.S. ATLAS Facility Planning U.S. ATLAS Tier-2 & Tier-3 Meeting at SLAC 30 November 2007.

1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.

ATLAS Distributed Analysis Dietrich Liko IT/GD. Overview  Some problems trying to analyze Rome data on the grid Basics Metadata Data  Activities AMI.

The ATLAS Computing & Analysis Model Roger Jones Lancaster University ATLAS UK 06 IPPP, 20/9/2006.

Distributed Analysis Tutorial Dietrich Liko. Overview  Three grid flavors in ATLAS EGEE OSG Nordugrid  Distributed Analysis Activities GANGA/LCG PANDA/OSG.

Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010.

Finding Data in ATLAS. May 22, 2009Jack Cranshaw (ANL)2 Starting Point Questions What is the latest reprocessing of cosmics? Are there are any AOD produced.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Response of the ATLAS Spanish Tier2 for.

ATLAS Distributed Analysis DISTRIBUTED ANALYSIS JOBS WITH THE ATLAS PRODUCTION SYSTEM S. González D. Liko

WLCG November Plan for shutdown and 2009 data-taking Kors Bos.

ATLAS Physics Analysis Framework James R. Catmore Lancaster University.

SAM architecture EGEE 07 Service Availability Monitor for the LHC experiments Simone Campana, Alessandro Di Girolamo, Nicolò Magini, Patricia Mendez Lorenzo,

Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.

1 S. JEZEQUEL- First chinese-french workshop 13 December 2006 Grid: An LHC user point of vue S. Jézéquel (LAPP-CNRS/Université de Savoie)

CERN IT Department CH-1211 Genève 23 Switzerland t EGEE09 Barcelona ATLAS Distributed Data Management Fernando H. Barreiro Megino on behalf.

University of Texas At Arlington Louisiana Tech University

Database Replication and Monitoring

Gavin McCance University of Glasgow GridPP2 Workshop, UCL

Moving the LHCb Monte Carlo production system to the GRID

ALICE analysis preservation

Readiness of ATLAS Computing - A personal view

Zhongliang Ren 12 June 2006 WLCG Tier2 Workshop at CERN

YongPyong-High Jan We appreciate that you give an opportunity to have this talk. Our Belle II computing group would like to report on.

ATLAS DC2 & Continuous production

Presentation transcript:

Metadata requirements for HEP Paul Millar

Slide 2 12 September 2007 Metadata requirements for HEP Some of the players in this game... WLCG – Umbrella organisation for three grids: Nordugrid, EGEE (mostly Europe) and OSG. EGEE – Provides grid software, deployed mostly in Europe. Atlas/LHCb/CMS/Alice – the experiments GridPP – UK's contribution to HEP Grid. Metadata group – initiated within GridPP, members from HEP experiments.

Slide 3 12 September 2007 Metadata requirements for HEP Overview of LHC/HEP

Slide 4 12 September 2007 Metadata requirements for HEP View of ATLAS

Slide 5 12 September 2007 Metadata requirements for HEP How we get data...

Slide 6 12 September 2007 Metadata requirements for HEP Simulated 3D collision

Slide 7 12 September 2007 Metadata requirements for HEP Work flows Production: –Raw --> ESD (500kB) --> AOD (100kB) + Tag (1kB) ‏ –Raw --> Digi --> DST --> RDST -->... Monte Carlo –Physical model --> Raw --> [ ESD -->... ] Analysis –dataset selection (querying dataset metadata) ‏ –dataset creation (AOD) ‏ –AOD --> DPD -->...

Slide 8 12 September 2007 Metadata requirements for HEP Conditions database Measurements of physical conditions of the detector –In fact, the conditions of the sub-detectors. Measures “physical” conditions: –Magnetic field, temperature, electrical,... Large! Stored in database, replicated to some sites. Is this metadata?

Slide 9 12 September 2007 Metadata requirements for HEP Different metadata Job metadata –production –analysis Sites “datasets” –location of files of a dataset –provenance system –Physics metadata common to dataset.

Slide September 2007 Metadata requirements for HEP Existing solutions Atlas: AMI, DQ2, prod.sys Amga (LHCb logging and bookkeeping) ‏ Alice (Alien) ‏ CMS dataset selector. Metadata group: –Case study: “Unlucky for some: the 13 core use- cases in HEP” –Review of metadata Schema. –Help in creating a Metadata Query Language