1. Metadata and COMA Elizabeth Gallas Oxford. ATLAS Weekly / Open EB Meeting October 18, 2011

2. Outline  Overview of ATLAS Metadata Interfaces  What is COMA ?  Principles  Tour  VERY quick tour today…hope you will try yourself!  Summary and Plans  Conclusions  Backup Slides: Links Thanks to many experts, developers and also many of you who post questions / use cases to e-groups !!! June 2011Elizabeth Gallas - COMA2

3.  Subsystem specific metadata applications... driven by use cases/workflow  Trigger: TriggerTool … Web: trigconf and trigger timelines  Geometry DB: Detector Description Browser  Conditions DB:  runQuery (Run information from Conditions DB)  ATLAS WEB DQ  COOL Tag Browser  Data Summary Reports (Luminosity, Beam)  Beam Spot Summary  GANGA and PAthena  Panda / monitor  DQ2 Client  Tag Collector – software releases ...  Dedicated Metadata Catalogs  TAGs – event level (file and relational formats)  iELSSI and Suite of TAG Services  TAG Catalog  AMI – Dataset, processing … other metadata  And the AMI Suite of services  COMA – Conditions and configuration ATLAS Metadata Application Overview June 2011Elizabeth Gallas - COMA3 Connected in many ways (and with links to many above applications) … Some mentioned today in the course of “COMA” topic discussed today

4. Medicine: deep sleep … a state of extreme unresponsiveness ? coma ? Many definitions: June 2011Elizabeth Gallas - COMA4 Optics: aberration inherent to certain optical designs or due to imperfection in the lens or other components… Astronomy: luminous cloud surrounding the frozen solid nucleus in the head of a comet … ATLAS: COMA is an ~ acronym for a metadata catalog and its related services: Conditions / Configuration Metadata for ATLAS

5. June 2011Elizabeth Gallas - COMA5 What is COMA ?  COMA: “Conditions/Configuration Metadata for ATLAS”  Originally: built to support dynamic queries of TAG DB  Evolved: into a standalone system with interfaces  Now part of general effort to consolidate/relate ATLAS Metadata  COMA Components:  Relational Database  Contains extracted, refined, reduced, and derived information from system specific data sources (e.g. Conditions DB, …) plus information from non-database sources.  Interfaces  A set of Browsers and Reports  Main Documentation https://twiki.cern.ch/twiki/bin/viewauth/Atlas/ConditionsMetadata  COMA Portal (grid certificate in browser is required) https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html

6. June 2011Elizabeth Gallas - COMA6 COMA Interfaces: General Principles  Report or Browse the COMA catalogue:  Originally developed to check COMA loading and do cross checks  Evolved to current form: on request from experts and users  Logical division of information into separate but cross linked reports  Include links to more details (COMA links and External links)  runQuery, Luminosity, AMI, Trigger, DQ …  Provide reports [distinctive from, complimentary to] other systems  (Avoid duplication of effort)  Display COMA ‘derived’ information  Not available in other systems  Note: All the COMA Reports use GET methods:  Inputs are not complex, user can modify URL to get report they want, put links into TWiki or other documentation https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=161562

7. June 2011Elizabeth Gallas - COMA7 COMA Interfaces Portal https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html Tour stops today DB mtg Weds

8. June 2011Elizabeth Gallas - COMA8 Introduction: ATLAS Data Periods  A Data Period is a set of ATLAS Runs grouped for a purpose  Defined by Data Preparation Coordinators  Used in ATLAS data processing, assessment, and selection …  Each Period uniquely defined with a combination of  Project name (i.e. ‘data10_7TeV’)  Period name (i.e. ‘C1’, ‘C2’, ‘C’, ‘AllYear’ …)  Before 2011, Data Periods were  Described on TWiki page  https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods  Stored in a file based system  Edited by hand by Data Prep Coordination (experts)  Structure evolved over last year with experience  This experience  valuable to decide/define long term solution  Early in 2011: Data Periods stored in the COMA DB  Thanks: Beate (DataPrep Coordinator), AMI team, me.

9. June 2011Elizabeth Gallas - COMA9 https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php Menu Purpose: Show DataPrep defined Periods giving links to reports of general info about their Runs. Choose the Period of interest:  By Year  E.G. all ‘2010’  By Project  e.g. ‘data10_7TeV’  By specific Period or Group  Click on the project and then Period of interest Wildcards can be entered in this optional section, then click on Submit button “highlight” expanding sections

10. June 2011Elizabeth Gallas - COMA10 Period Description Report Example for data10_7TeV -/+ highlighted links: These sections expand to show period members Members of data11_7TeV.M are M1, M2, M3, M4 Links: to COMA & runQuery multi-Run Reports for chosen Period (COMA link: next slide) Header: Input criteria; Links: Contact, Documentation Period “Status”: M1 is ‘frozen’, M2-M4 are ‘locked’ … M ‘unlocked’ (M5 is on-going)

11. June 2011Elizabeth Gallas - COMA11 COMA Multi-Run Report: Purpose: show basic information for all runs matching the input criteria Run links: COMA, runQuery, and LumiDataSummary Single Run report (COMA link: next slide) SMK links:  COMA Trigger Super Master Key Report Links in Table column headers:  Short description of column Hover over any link: Indicates what will happen https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?fnt=data10_7TeV&pn=VdM

12. June 2011Elizabeth Gallas - COMA12 COMA Single Run 180776 Overview https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=180776 https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=180776 General Run info:  Project, LB range, Date …  Events, HLT Release …  Links to COMA reports  COMA Master Key report (no time to show today)  Links to external reports  HLT release, AMI, LumiDS, runQuery, Trigger, PSEvol Yellow sections expand:  Data Periods  Active Streams  AMI tags (TAG production) Shortcut find runs w/AMI tag  Prescale Evolution PS Key changes  Trigger section All HLT signatures All LVL1 items

13. June 2011Elizabeth Gallas - COMA13 COMA Single Run -/+AMITag Section Recent data11_7TeV M4 Period Run 190933 Shows:  All physics streams  Including debug !  “First Pass” – first processing  New columns appear for reprocessing passes  AMI tag used to process data  Link: COMA multi-Run Report  All runs processed w/this tag  “?”: link to AMI  Description of AMI tag  Launch iELSSI !  For this Run/Stream/AMItag  Take a quick look at this run with the ELSSI TAG browser !!!  See if events pass your trigger in the debug stream …

14. June 2011Elizabeth Gallas - COMA14 COMA Single Run -/+Trigger Section Start to see here some COMA “derived” info Open this section in next slide… COMA divides HLT trigger signatures:  “Physics”: complete EF-L2-L1 chains  “Commissioning”: has missing level(s) COMA looks at each LB range: 1. Decides if each chain is “active” based on prescales, passthrough, rerun in every LB range of Prescale evolution 2. For “active” chains via prescale, calculates an “Aggregate prescale” (EF*L2*L1 prescale) in each LB range 3. Determines Min, Max of AggPS over the Run for each chain 4. Determines Run and LB-wise PS,PT,RR Flags

15. June 2011Elizabeth Gallas - COMA15 COMA single-Run Report: “Physics” Trigger Section Choose (using jQueryColumnFilters) Chains: EF*JPSI and Stream: egamma COMA Chain Report: All Runs w/ this chain/item COMA Prescale Report: LB-wise PS info for this Run/Signature (next slide) Run-wise Aggregate prescale Range and Flags(PT,PS,RR)

16. June 2011Elizabeth Gallas - COMA16 COMA Prescale Report (Run 180776,EF_e9_tight_e5_tight_Jpsi) LB Range Summary table:  LB-wise Prescale Evolution  Link:Trigger DB Prescale Reports (useful cross check !)  Prescales:  L1,L2,EF and Aggregate PS  Chain Inactive from  LB 1-99 and 114-134 (LB ranges in COMA always inclusive) Next slide: COMA Chain Report for EF_e9_tight_e5_tight_Jpsi Run-wise Summary:  Includes links to COMA reports:  Run, Master Key, Chain and Prescale Reports (each level)  Associated L2, L1 names  Aggregate PS Range: 4 to 5  Flags:  Partially active via PS  Never passthrough (L2 or EF)  Never Rerun

17. June 2011Elizabeth Gallas - COMA17 RBR Chain Report (EF_e9_tight_e5_tight_Jpsi) Activation Summary: Shows the Runs in which this element is “active” separated by:  “Physics” EF-L2-L1 signatures  Via prescale  Via pass through  Via rerun  “Commissioning” signatures  Via prescale  Via pass through  Via rerun Configuration Summary: Shows where this element is configured:  Super Master Key(s),  Runs (Summary) Period Evolution: Shows chain/item bit, version evolution for “Physics” chains during Period Runs Open these sections in next slides…

18. June 2011Elizabeth Gallas - COMA18 COMA Chain Report (EF_e9_tight_e5_tight_Jpsi) Expand Run-wise Activation …  “Physics” EF-L2-L1 signatures  Active via Prescale  Runs in Data Periods Table Shows (Run Count):  Periods, Link: Run, SMK Reports  Level bit assignments  Link to: Chain/Item Reports (3)  Range of Aggregate Prescale while chain is active via prescale in Run  Links: COMA Prescale Report (3) 70 Period Runs where this chain is “active”

19. June 2011Elizabeth Gallas - COMA19 COMA Chain Report (EF_e9_tight_e5_tight_Jpsi) Period Evolution Section …  For “Physics” chains in Period Runs  Separate table for each EF-L2-L1 signature at each beam energy Each Table Shows:  Row-wise:Distinct set of bit and chain/item versions Columns:  Bit assignments  Chain version (links to Trig diff)  Chain Report Links  Range of AggPS, SMK, Run, Period, Date, HLTRelease Thanks to Tomasz, Joerg for many useful discussions

20. June 2011Elizabeth Gallas - COMA20 COMA Chain Wildcard Report (input: “EF_g10%”) Purpose: See all the names matching a pattern or Find exact name from part of the name Report: Displays chain/item names matching the input string … text size proportional to occurrence in SMK  In Period Runs and in All Runs

21. June 2011Elizabeth Gallas - COMA21 Summary and Plans  COMA – an integral part of ATLAS Metadata infrastructure  Essential to ATLAS event-level metadata decoding  Ideally placed to provide links and interface to other metadata  Special relationship to AMI (and TAG catalog)  Launch iELSSI to take a quick look at any Run  Primary source for “ATLAS Data Periods”  Periods in Lum, DQ, Run Summary, AMI reports comes from COMA  Reports feature “derived” information not available elsewhere  Trigger experts recommend COMA Trigger/Prescale  Report usage: from ~200 to over 5000 pages viewed/month.  Peaked in July as users did final preparations for summer conferences  Current efforts:  COMA Database content growing  Watch use cases to identify new areas to focus growth  COMA Report and Browser development  keep pace with content, improve functionality and usability  Beyond COMA: Interface development  Connecting COMA to other parts of the infrastructure

22. June 2011Elizabeth Gallas - COMA22 COMA Conclusions  This is an evolving system … information in the system is growing based on information available and use cases  Adding more dimensions to the Conditions data  With suitable relationships to facilitate queries  Making that criteria available in dynamic useable interfaces  We want to insure the Metadata is  complete enough to satisfy use cases while  reflecting accurately its limitations  Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS  This facilitates cross checks with other systems  Continuous process: talking with various experts to ensure  data integrity, completeness, compatibility w/other systems … Very positive feedback so far … more always welcome … hn-atlas-physicsMetadata@cern.ch

23. June 2011Elizabeth Gallas - COMA23 Backup

24. Link Summary Main TWiki Documentation: https://twiki.cern.ch/twiki/bin/viewauth/Atlas/ConditionsMetadata COMA Report Documentation: https://atlas-tagservices.cern.ch/RBR/rBR_Documentation.html COMA Portal (grid certificate in browser is required) https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html Trigger General Meeting (June 8, 2011): “User Guide to Trigger/Prescale information in COMA Reports” https://indico.cern.ch/materialDisplay.py?contribId=0&materialId=slides&con fId=125723  Data sources, Business rules and Caveats; Form input; Example links; details of Data Period & derived quantities; Data Periods TWiki (revised): https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods June 2011Elizabeth Gallas - COMA24

25. June 2011Elizabeth Gallas - COMA25 COMA Report(s) Example Links COMA single-Run Report (Run 161562): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php? runs=161562  General Run information, AMI tags, DQ, Prescale Evolution, Triggers  Trigger tables show derived Run-integrated prescales, passthrough COMA Trigger Super Master Key Report (Key 877): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php? smk=877  Runs using this SMK, physics/commissioning chains, Level 1 items COMA Chain Report (Chain EF_g11_etcut): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php? cn=EF_g11_etcut  Summary: SMK, Runs, Signature appearances of this chain  Runs with this chain ‘active’ COMA Chain Wildcard Report (chain name matching string EF_g1%): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?cn=EF_g1%  Summary: SMK, Runs with this chain: Configured, Active COMA Prescale Report (Run 162882 Prescales for chain EF_g11_etcut): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=162882&cn=EF_g11_etcut  Derived chain activity over the Run and LB-wise prescales

26. June 2011Elizabeth Gallas - COMA26 Data Periods: Links to Reports and Services Revised TWiki page: https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods  Interactive USERS  COMA Data Period Documentation Interface  https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php  Comments: hn-atlas-physicsMetadata@cern.ch.hn-atlas-physicsMetadata@cern.ch  Programmatic USERS For systems needing period info: runQuery, beamspot, Data Quality, …, “Data Period Services” provided via pyAMI:  http://ami.in2p3.fr/opencms/opencms/AMI/www/Client/DataPeriods_pyAMI.pdf http://ami.in2p3.fr/opencms/opencms/AMI/www/Client/DataPeriods_pyAMI.pdf  Comments: AMI / Tag_Collector Team.AMI / Tag_Collector Team  Data Preparation EXPERTS: Entry Interface:  https://ami.in2p3.fr/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?linkId=1479 https://ami.in2p3.fr/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?linkId=1479  Comments: AMI / Tag_Collector Team.AMI / Tag_Collector Team

27. June 2011Elizabeth Gallas - COMA27 COMA runBrowserReport Index page  INDEX: https://atlas-tagservices.cern.ch/RBR/runBrowserReport.htmlhttps://atlas-tagservices.cern.ch/RBR/runBrowserReport.html  GUIDE: https://atlas-tagservices.cern.ch/RBR/rBR_Documentation.htmlhttps://atlas-tagservices.cern.ch/RBR/rBR_Documentation.html “Index” page:  Enter one/more input quantity,  IE: a Run number  IE: a Chain Name  Click on submit (or return key) Links at bottom of every page: Reports use “GET” methods: Modify the URL to change input Reports contain links No need to use index page for every query Links lead you to other COMA reports other external reports AMI, runQuery, TriggerDB…