Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010

April 2010Elizabeth Gallas - COMA2 Outline History of Conditions Metadata / “Current” COMA Schema Evolution in context to overall TAG DB project Overview of Oracle Schemas for TAGs 1.TAG DB (Event-wise Metadata) 2.CATALOG (Dataset Catalogue Metadata) 3.COMA (COnditions MetadatA) Upload procedure/Status Uses of Conditions Metadata ELSSI usage runBrowser – new interface runBrowserReport – new interface Documentation and Links Features of the interfaces Functionality in our Future Summary Sorry to miss the meeting in person

April 2010Elizabeth Gallas - COMA3 Early history of this Conditions Metadata project 2007: first Conditions Metadata tables were filled for MC simulation tests/prototype development Streaming Test, Full Dress Rehearsal (FDR) exercises Run/LB-wise conditions were collected from MC log files and other sources into relational DB tables (COMA) And INSERTED into new folders in Conditions database (COOL) Which formed the prototype for the ATLAS Luminosity calculation Other Trigger/DAQ COOL folders defined at same time (RichardHawkings, Trigger/TDAQ) 2008: COMA tables used by ELSSI prototype (still just MC) Conditions, particularly trigger configuration … things not practical to store event-wise 2009: start extracting Run/LB-wise information from COOL into COMA tables to facilitate efficient access to Conditions Metadata by ELSSI for Online Runs

April 2010Elizabeth Gallas - COMA4 History of this Conditions Metadata project 2009 … (continued) … runBrowser developed runBrowser: web interface for finding Runs sharing common conditions Initially development tool: check data integrity/relationships of COMA tables We (TAG developers) realized this runBrowser would be more generally useful … separate Run-browsing (runBrowser) from Event- browsing (ELSSI) … make runBrowser a stand-alone tool May2009: TAG group developed first DTD for GoodRunList XML This xml was how runBrowser would communicate to ELSSI the Run/LB selection and selections criteria. This DTD has since been taken up/over by the ATLAS experiment to communicate good Run/LB ranges between subsystems. NOTE: runBrowser is NOT a replacement for runQuery (the web based browser to all online Runs in the Conditions DB) runBrowserReport development started Which will help ELSSI and runBrowser describe underlying COMA info and provides more links to other ATLAS systems COMA tables expanded to include more conditions in anticipation of use cases / expand selection criteria

April 2010Elizabeth Gallas - COMA5 Current COMA Tables Don’t send me into the volcano … users don’t need to know these tables … Users use: ELSSI, runBrowser, runBrowserReport

April 2010Elizabeth Gallas - COMA6 COMA runBrowser: Interface Status As data has been added to COMA tables runBrowser has evolved accordingly Current selection criteria includes: Data Source (online,simulation), Data Type, Run Number, Duration, Number of LBs, temporal, DAQ configuration, Filename tag (Project Name), Trigger Super Master Key, Trigger LVL1 and/or HLT Prescale Key. Useful for initial purpose and query development But initial implementation was not generally well constructed … October 2009: Ryan Buckingham joined project Oxford graduate student – 1 st year. Move to CERN Aug 1, 2010 Working on runBrowser Improved internal communication object oriented structure introducing further classes - allow criteria to be added in modules Will facilitate better syntax alignment with runQuery and inclusion of criteria to the GRL xml to pass to ELSSI Improve look/feel of interface: using php and jQuery widgets

April 2010Elizabeth Gallas - COMA7 Evolution of overall TAG DB project (last year) TAG DB / ELSSI: evolved to a distributed architecture Realization: Not possible to upload all TAGs at any one Oracle site Advantageous to have some TAGs at multiple sites … So ELSSI: needs to know which TAGs are uploaded at which voluntary sites Add new relational schema TAGS CATALOG (Elisabeth Vinek)  Contains processing/upload information needed to deploy distributed TAG services on the grid TAG DB / CATALOG / COMA Schemas work together Common threads include: RunNumber, Stream, ProjectName … COMA tables also use the CATALOG Upload only Run/LB metadata for Runs in CATALOG This reduces handling of conditions anomalies  allowing us to focus on Runs of ‘event analysis interest’ Steps in Database loading – ideally within hours of reconstruction 1.TAGs uploaded to Oracle 2.CATALOG tables updated 3.COMA tables updated Now changing to use Tier-0 DB to advance the COMA loading phase

April 2010Elizabeth Gallas - COMA8 TAG DB Event-wise metadata tables Stores Event-wise attributes: electron (Et, eta, phi …) muon (Et, eta, phi …) … and references to RAW, ESD, AOD files Official data processing chain: RAW  ESD  AOD  TAG files Data Catalogue tables Stores information on file and dataset processing and location Project name AMI tag (what processing occurred)… Sources: AMI (ATLAS Metadata Catalogue, Tier0 … ‘COMA’ (COnditions MetadatA) tables Conditions of data taking Beam conditions Trigger and DAQ conditions Magnetic field … Various sources: Conditions DB, Log files, xml files, … Oracle Database: TAG DB and associated metadata tables RUNS 3. COMA tables …EventLBRun 1. TAG DB: Event-wise metadata ORACLE DB 2. DATA Catalogue Tables

April 2010Elizabeth Gallas - COMA9 Some Current Activity COMA: Loading programs continue to improve/refine Some areas better resolved than others and still some information needs to be loaded (magnet, fill …) New: Define Run-wise Aggregate prescales, passthrough Calculates Run-wise summary: PS, PT, Rerun per triggger Useful to ELSSI to display active/passive/disabled triggers per Run … So that users do not select triggers in a Run which would yield 0 events because that trigger was not active Useful for runBrowser to find the active chains for each Run Work with Trigger group for validation Data Quality upload Loads final LBSUMM DQ assessments when tagged/locked Includes Detector and Physics DQ assessments Can envision ways to include virtual flags with this information Schema and Loading – groundwork laid, interfaces underway Work with DQ group for verification/refinement

April 2010Elizabeth Gallas - COMA10 Documentation and Links COMA Documentation COMA Schema COMA Tables COMA developer documentation (not user doc): COMA user documentation Will be integrated into the interfaces below COMA User Interface Links (test versions) NOTE: The interfaces require a grid certificate in you browser NOTE: The interfaces are not in production server locations runBrowser (2) Link: /runBrowser2/runBrowser.php runBrowserReport Link: erReport.php?q=find+run https://lxvm0341.cern.ch/tagservices/dev/gallas/TAGS/runBrowser/runBrows erReport.php?q=find+run  for Run (replace the Run Number in the URL with your Run of interest) Let’s try them

April 2010Elizabeth Gallas - COMA11 Functionality in our Future Criteria for selecting Runs (combine COMA/CATALOG info) Select Online or MC Runs The TAGs for Online/MC have the same attributes We track the COMA & Catalog Metadata for both in same way Select Runs uploaded to TAG DB These Runs ARE the Runs of analysis interest  The Runs ATLAS chooses to reconstruct and reprocess  These Runs follow the CREM deletion policy Select Runs with a particular AMI tag (tracked in the Catalog) Select Runs with many of the criteria now available in runQuery (but NOT a replacement for runQuery) Project Name, Magnetic field, Beam conditions, DAQ config, Master Key (Trigger, LVL1/HLT PS Keys) … Select Runs with one/more particular trigger chains Moreover those which are active during the Run Select Runs in particular Streams (with event counts available)

April 2010Elizabeth Gallas - COMA12 Conclusions This talk reflects an evolving system Current information in the system is growing  Adding Conditions Metadata to a relational database  Making that criteria available in a dynamic useable interface We want to insure the Metadata is  complete enough to satisfy use cases while  reflecting accurately its limitations Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS i.e. runQuery, GoodRunList xml … This facilitates cross checks with other systems Continuous process: talking with various experts to ensure data integrity and completeness

April 2010Elizabeth Gallas - COMA13 Backup