1. Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010

2. April 2010Elizabeth Gallas - COMA2 Outline History of Conditions Metadata Evolution Overview of Oracle Schemas for TAGs 1.TAG DB (Event-wise Metadata) 2.CATALOG (Dataset Catalogue Metadata) 3.COMA (COnditions MetadatA) “Current” COMA Schema Uses of Conditions Metadata ELSSI usage Decoding bit packed event-wise TAG attributes (Trigger) runBrowserReport runBrowser Upload procedure and COMA Loading Status COMA Browser - Interface Status Documentation and Links Plans Summary

3. April 2010Elizabeth Gallas - COMA3 History of this Conditions Metadata project 2007: first COMA tables were filled for simulation tests/prototype development Streaming Test and (FDR) Full Dress Rehearsal exercises (MC) Run/LB-wise conditions were collected from log files and other sources And INSERTED into new folders in Conditions database (COOL) Which formed the prototype for the ATLAS Luminosity calculation Other Trigger/DAQ COOL folders defined at same time 2008: COMA tables used by ELSSI prototype (still just MC) Conditions, particularly trigger configuration, not practical to store event-wise 2009: start extracting Run/LB-wise information from COOL into COMA to facilitate efficient access to Conditions Metadata for ELSSI for Online Runs runBrowser developed Initially private tool: to check data integrity/relationships between COMA tables We realized this runBrowser would be more generally useful … separate Run- browsing (runBrowser) from Event-browsing (ELSSI) … make runBrowser a stand-alone tool runBrowserReport emerged Which will help ELSSI and runBrowser describe underlying COMA info and provides more links to other ATLAS systems COMA tables expanded to include more conditions in anticipation of use cases

4. April 2010Elizabeth Gallas - COMA4 Evolution in the last year of overall project TAG DB / ELSSI evolved to a distributed architecture Not possible to upload all TAGs at any one Oracle site ELSSI – also needs to know which TAGs are uploaded at which voluntary sites An additional relational schema has been added: ATLAS TAGS CATALOG (Elisabeth Vinek)  Contains processing/upload information needed to deploy distributed TAG services on the grid The COMA tables use the TAG CATALOG Upload only Run/LB metadata for Runs in CATALOG This reduces handling of conditions anomalies, allowing us to focus on Runs of ‘event analysis interest’ Steps in Database loading – ideally within hours of reconstruction TAGs uploaded to Oracle CATALOG tables updated COMA tables updated tho plans now to use Tier-0 DB to advance the COMA loading phase

5. April 2010Elizabeth Gallas - COMA5 TAG DB Event-wise metadata tables Stores Event-wise attributes: electron (Et, eta, phi …) muon (Et, eta, phi …) … and references to RAW, ESD, AOD files Official data processing chain: RAW  ESD  AOD  TAG files Data Catalogue tables Stores information on file and dataset processing and location Project name AMI tag (what processing occurred)… Sources: AMI (ATLAS Metadata Catalogue, Tier0 … ‘COMA’ (COnditions MetadatA) tables Conditions of data taking Beam conditions Trigger and DAQ conditions Magnetic field … Various sources: Conditions DB, Log files, xml files, email… Oracle Database: TAG DB and associated metadata tables RUNS 3. COMA tables …EventLBRun 1. TAG DB: Event-wise metadata ORACLE DB 2. DATA Catalogue Tables

6. April 2010Elizabeth Gallas - COMA6 Current Activity COMA: Loading programs continue to improve/refine Some areas better resolved than others and still some information needs to be loaded (magnet, fill …) New: Define Run-wise Aggregate prescales, passthrough Calculates Run-wise summary: PS, PT, Rerun per triggger Useful to ELSSI to display active/passive/disabled triggers per Run … So that users do not select triggers in a Run which would yield 0 events Useful for runBrowser to find the active chains for each Run Data Quality upload Loads final LBSUMM DQ assessments when tagged/locked Includes Detector and Physics DQ assessments Can envision ways to include virtual flags with this information Schema and Loading – groundwork laid, interfaces underway Work with DQ group for verification/refinement

7. April 2010Elizabeth Gallas - COMA7 COMA runBrowser: History and Evolution A web-based browser was developed for COMA tables to facilitate checking data integrity and relationships in the relational tables March2009: Features of this initial browser were presented … we realized that this kind of browser could be a Run/LB wise component of ELSSI … but also could be a stand-alone browser for Conditions data http://indico.cern.ch/materialDisplay.py?contribId=1&materialId=slides&confId=54409 The name of this browser: runBrowser (or COMABroswer) Not yet released to the collaboration Not a replacement for runQuery (the web based browser to all online Runs in the Conditions DB) May2009: TAG group developed first DTD for the GoodRunList XML(GRL): This xml will be how runBrowser communicates to ELSSI the Run/LB selection. This DTD has since been taken up by the ATLAS experiment to communicate good Run/LB ranges between subsystems.

8. April 2010Elizabeth Gallas - COMA8 COMA runBrowser: Interface Status As data has been added to COMA tables runBrowser has evolved accordingly Current selection criteria includes: Data Source (online,simulation), Data Type, Run Number, Duration, Number of LBs, temporal, DAQ configuration, Filename tag (Project Name), Trigger Super Master Key, Trigger LVL1 and/or HLT Prescale Key. Useful for initial purpose and query development But initial implementation was not generally well constructed … October 2009: Ryan Buckingham joined project Oxford graduate student – 1 st year. Move to CERN Aug 1, 2010 Working on runBrowser Improved internal communication object oriented structure introducing further classes - allow criteria to be added in modules Will facilitate better syntax alignment with runQuery and inclusion of criteria to the GRL xml to pass to ELSSI Improve look/feel of interface: using php and jQuery widgets

9. April 2010Elizabeth Gallas - COMA9 Documentation and Links COMA Schema COMA Tables https://gallas.web.cern.ch/gallas/COMA_Tables.html COMA developer documentation: https://gallas.web.cern.ch/gallas/Conditions_to_TAGs.html Interface Links NOTE: The interfaces require a grid certificate in you browser NOTE: The interfaces are not in production server locations runBrowser (2) Link: runBrowserReport Link:

10. April 2010Elizabeth Gallas - COMA10 Functionality in our Future Criteria for selecting Runs (some criteria combining COMA/CATALOG Metadata) Select Online or MC Runs The TAGs for Online/MC have the same attributes Select Runs uploaded to TAG DB These Runs ARE the Runs of analysis interest (the Runs ATLAS chose to reconstruct and reprocess) These Runs follow the CREM deletion policy Select Runs with a particular AMI tag Select Runs with many of the criteria now available in runQuery (but NOT a replacement for runQuery) Project Name, Magnetic field, Beam conditions, DAQ config, Trigger Master Key (and LVL1/HLT PS Keys) … Select Runs with one/more particular trigger chains Moreover those which are active during the Run

11. April 2010Elizabeth Gallas - COMA11 Conclusions This talk reflects an evolving system Current information in the system is growing Adding Conditions Metadata to a relational database Making that criteria available in a useable interface We want to insure the Metadata is complete enough to satisfy use cases while Reflecting accurately its limitations Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS i.e. runQuery, GoodRunList xml … This facilitates cross checks with other systems Continuous process: talking with various experts on insuring data integrity and completeness