1. Data Periods in COMA Elizabeth Gallas - Oxford February 14, 2011

2. Feb 2010Elizabeth Gallas - COMA / RunBrowser2 Outline Purpose TODAY: talk about ATLAS Data Periods (entry, storage, reports and services)  What is a “Data Period” ? What is “COMA” ?  Current implementation  Future implementation  Schema proposal  What has been done so far ?  What is needed ? (interfaces: Entry, Report, Services)  Timescale  Other unresolved issues:  Loading issues  Production location  Replication plan

3. Feb 2010Elizabeth Gallas - COMA / RunBrowser3 What is a Data Period ?  Data Period is a set of ATLAS Runs grouped for a purpose.  Used in ATLAS data processing, assessment, and selection …  Each Period is designated a name … concatenation of  Project name (i.e. ‘data09_900GeV’)  Period name (i.e. ‘C1’, ‘C2’, ‘C’, ‘AllYear’ …)  But proceeded sometimes with ‘period’ (i.e. ‘periodC1’)   propose dropping this in DB implementation  Current implementation:  Resides in files on afs at CERN (next slide)  Edited by hand by Data Prep Coordination (experts)  Expert human entry is unavoidable … many factors  Structure evolved over last year with experience  Many systems read these AFS files including COMA loading … other systems use them from COMA.  Plan described here:  Use COMA as DB Storage behind current/future Data Periods  Define entry and report services to provide Period info

4. Feb 2010Elizabeth Gallas - COMA / RunBrowser4 Data Period AFS files to date data10_7TeV.periodAllYear.runs.list data10_7TeV.periodA1.runs.list data10_7TeV.periodA.runs.list data10_7TeV.periodB[1-2].runs.list data10_7TeV.periodB.runs.list data10_7TeV.periodC[1-2].runs.list data10_7TeV.periodC.runs.list data10_7TeV.periodD[1-6].runs.list data10_7TeV.periodD.runs.list data10_7TeV.periodE[1-7].runs.list data10_7TeV.periodE.runs.list data10_7TeV.periodF1.runs.list data10_7TeV.periodF2.runs.list data10_7TeV.periodF.runs.list data10_7TeV.periodG[1-6].runs.list data10_7TeV.periodG.runs.list data10_7TeV.periodH[1-2].runs.list data10_7TeV.periodH.runs.list data10_7TeV.periodI[1-2].runs.list data10_7TeV.periodI.runs.list data09_2TeV.AllYear.runs.list data09_900GeV.AllYear.runs.list data10_7TeV.periodVdM[1-3].runs.list data10_7TeV.periodVdM.runs.list data10_900GeV.periodB.runs.list data10_hi.periodJ[1-7].runs.list data10_hi.periodJ.runs.list /afs/cern.ch/atlas/ www/GROUPS/ DATAPREPARATION/ DataPeriods  Each file contains simply a list of Run numbers  [n-m] == range of files  Note: period C contains all Runs of period C1 and C2;

5. Feb 2010Elizabeth Gallas - COMA / RunBrowser5 What is COMA ? COMA: short name for “COnditions MetadatA”:  part of general effort to consolidate/relate ATLAS Metadata. 3 Components of the COMA Project:  COMA Database Tables  A collection of selected/derived Run/LB-wise Conditions  Source: mostly ATLAS Conditions DB and Trigger DB  Originally, built to support dynamic queries of TAG DB  Now, also a standalone system with interfaces (below)  RunBrowser Package:  runBrowserReport – A report interface for COMA tables  But AMI now has access … can also report …  runBrowser – interface for finding Run/LB of interest  runBrowser is another related interface in development NOTE: in diagrams, all table and view names start w/’COMA_’ (i.e. RUNS  table name is COMA_RUNS)

6. Feb 2010Elizabeth Gallas - COMA / RunBrowser6 COMA Schema Overview PERIOD_OBS LBN_TIMES EVENTS RUN_TYPES SOURCES MASKS COOL_CONNECTIONS FILLS* RUNS LBN_LUMS TRANSITIONS LOAD STATUS SHORTNAMES EXPLAINS RUN_FILLS* STREAM_EVENTS RUN_MAGNETS* LUM_DETS STREAM_LBN_EVENTS PERIODS RUN_PERIODS RUN_LUMS DETSTATUS DETECTORS CODES GROUPS VIRTUAL_FLAGS VFLAG_DETS HLT_PSLVL1_PS MK_Triggers PS_Sets Master_Keys Prescale_Keys LVL1_PSKHLT_PSK Run_Agg_PS Trig_Agg_PS MKT_Physics MKT_Others STREAM Trig_Stream Period related

7. Feb 2010Elizabeth Gallas - COMA / RunBrowser7 Current Periods in COMA Current COMA Tables related to Periods: 1.RUNS: table of all COMA Run Numbers 2.PERIODS: table of period names to date 3.PERIOD_RUNS: Runs in each Period 4.PERIOD_OBS: attempt to archive old records when periods changed (or catch human error) New implementation goals:  Make {entry,storage,report} more robust.  Some periods: are a group of periods  Want new tables to reflect this hierarchy  Want each piece of information in the schema only once and more robustly  Eliminate: PP_RUN_STRING, PERIOD_RANK  Need better system for obsolete entries  Allow flexibility to data entry  Improve indexing; better track insert/update dates;  AMI needs to know when entries change 1. RUNS RUN_INDEX 4. PERIOD_OBS P – CPO_INDEX PROJECT_PERIOD PERIOD_NAME FILENAME_TAG PP_FILE_TIME 2. PERIODS P – PROJECT_PERIOD PERIOD_NAME FILENAME_TAG PP_FILE_TIME PP_START_TIME PP_END_TIME PP_RUN_STRING PP_DESC PERIOD_RANK 3. PERIOD_RUNS P – PERIOD_NAME P – RUN_INDEX 2010 implementation

8. Feb 2010Elizabeth Gallas - COMA / RunBrowser8 Current  Proposed: Data Periods in COMA (tables) PERIOD_DEFS P – P_INDEX F – PDOM_NAME P_LEVEL ( 1, 2, 3 ) P_PROJECT_PERIOD P_PROJECT P_PERIOD P_DESC P_STATUS P_CREATE_USER P_CREATE_DATE P_MODIFY_USER P_MODIFY_DATE PERIOD_PHI_TO_PLOS PF – PHI_INDEX PF – PLO_INDEX PERIOD_DOMAINS P – PDOM_NAME PDOM_DESC PERIOD_DOMAIN_ROLES PDR_INDEX PDOM_NAME PDR_USER PDR_ROLE PERIOD_P1_TO_RUNS PF – P_INDEX PF – RUN_INDEX RUNS P -- RUN_INDEX PERIOD_GRLS PF – P_INDEX P – GRL_NAME PGRL_STATUS 1. RUNS RUN_INDEX 2. PERIODS P – PROJECT_PERIOD PERIOD_NAME FILENAME_TAG PP_FILE_TIME PP_START_TIME PP_END_TIME PP_RUN_STRING PP_DESC PERIOD_RANK 3. PERIOD_RUNS P – PERIOD_NAME P – RUN_INDEX (same) (P_LEVEL = 1) (P_LEVEL = 2,3) (add reference to GRLs) So 3 levels of periods (can be used like RANK): P_LEVEL = 1 is a Period, 2 is Period Group, 3 is Parent Group. (add ‘Domain’ dimension and people and roles for Period Entry)

9. Feb 2010Elizabeth Gallas - COMA / RunBrowser9 Definitions PERIOD_DEFS table  Current Periods are from DataPrep domain  PDOM_NAME = “DP”  Contains Period definitions (P_LEVEL=1,2,3)  P_LEVEL = 1 (a set of Run Numbers)  e.g: ‘D1’, ‘D2’,... ‘D6’ …  Names: alpha followed by a number  P_LEVEL = 2  A Period Group is a set of Level 1 periods  e.g.: ‘D’ contains ‘D1’-’D6’  Names: alpha with no number  P_LEVEL = 3  Parent Period Group: set of Level 2 periods  e.g.: ‘AllYear’ contains ‘A’, ‘B’, … ‘I’  P_DESC: written description  P_STATUS: ‘locked’ or ‘unlocked’  ‘locked’ == no more members can be added PERIOD_DEFS P – P_INDEX F – PDOM_NAME P_LEVEL ( 1, 2, 3 ) P_PROJECT_PERIOD P_PROJECT P_PERIOD P_DESC P_STATUS P_CREATE_USER P_CREATE_DATE P_MODIFY_USER P_MODIFY_DATE

10. Feb 2010Elizabeth Gallas - COMA / RunBrowser10 New implementation status  New COMA Period Tables, Views (next slide) have been defined  In ATLAS_TAGS_METADATA_SG@INT8R  Table/Column descriptions:  https://gallas.web.cern.ch/gallas/TableDoc_COMA.html https://gallas.web.cern.ch/gallas/TableDoc_COMA.html but some minor modifications not checked in yet … sorry  Legacy data has been loaded: Old  New tables  Some new runBrowserReports show results (my php dev area):  rBR: Data Period Report input menu  https://voatlas69.cern.ch/tagservices/dev/users/gallas/RunBrowser/trunk/ runBrowserReport/rBR_Period_Report.php https://voatlas69.cern.ch/tagservices/dev/users/gallas/RunBrowser/trunk/ runBrowserReport/rBR_Period_Report.php  rBR: Multi-Run Report for Period J (project ‘data10_hi’):  https://voatlas69.cern.ch/tagservices/dev/users/gallas/RunBrowser/trunk/ runBrowserReport/runBrowserReport.php?pn=J%&fnt=data10_hi https://voatlas69.cern.ch/tagservices/dev/users/gallas/RunBrowser/trunk/ runBrowserReport/runBrowserReport.php?pn=J%&fnt=data10_hi  rBR: Front page for General runBrowserReports:  https://voatlas69.cern.ch/tagservices/dev/users/gallas/RunBrowser/trunk/ runBrowserReport/runBrowserReport.html https://voatlas69.cern.ch/tagservices/dev/users/gallas/RunBrowser/trunk/ runBrowserReport/runBrowserReport.html  rBR Documentation:  https://twiki.cern.ch/twiki/bin/viewauth/Atlas/ConditionsMetadata https://twiki.cern.ch/twiki/bin/viewauth/Atlas/ConditionsMetadata  https://voatlas69.cern.ch/tagservices/dev/users/gallas/RunBrowser/trunk/ runBrowserReport/rBR_Documentation.html https://voatlas69.cern.ch/tagservices/dev/users/gallas/RunBrowser/trunk/ runBrowserReport/rBR_Documentation.html  not updated yet wrt new period implementation

11. Feb 2010Elizabeth Gallas - COMA / RunBrowser11 Proposed: Data Periods and Period Groups (tables and views) PERIOD_DEFS P – P_INDEX F – PDOM_NAME P_LEVEL ( 1, 2, 3 ) P_PROJECT_PERIOD P_PROJECT P_PERIOD P_DESC P_STATUS P_CREATE_USER P_CREATE_DATE P_MODIFY_USER P_MODIFY_DATE PERIOD_PHI_TO_PLOS PF – PHI_INDEX PF – PLO_INDEX PERIOD_DOMAINS P – PDOM_NAME PDOM_DESC PERIOD_DOMAIN_ROLES PDR_INDEX PDOM_NAME PDR_USER PDR_ROLE PERIOD_P1_TO_RUNS PF – P_INDEX PF – RUN_INDEX RUNS P -- RUN_INDEX PERIOD_GRLS PF – P_INDEX P – GRL_NAME PGRL_STATUS V_PHI_PLOS PDOM_NAME PHI_LEVEL PHI_INDEX PHI_PROJECT PHI_PERIOD PLO_INDEX PLO_PROJECT PLO_PERIOD V_P1_RUNS PDOM_NAME P1_INDEX P1_PROJECT P1_PERIOD RUN_INDEX V_P2_RUNS PDOM_NAME P2_INDEX P2_PROJECT P2_PERIOD P1_INDEX P1_PROJECT P1_PERIOD RUN_INDEX V_P3_RUNS PDOM_NAME P3_INDEX P3_PROJECT P3_PERIOD P2_INDEX P2_PROJECT P2_PERIOD P1_INDEX P1_PROJECT P1_PERIOD RUN_INDEX V_PERIOD_RUNS PDOM_NAME P_LEVEL P_INDEX P_PROJECT P_PERIOD RUN_INDEX

12. Feb 2010Elizabeth Gallas - COMA / RunBrowser12 Open Issues  Schema includes: Opportunity for other Period Domains  Right now, all Periods in PDOM_NAME = DP (“Data Preparation”)  In all interfaces, this domain is the default (when unspecified)  Do other groups want to define sets of runs  e.g. alignment?  create Domain = AL (“Alignment Group”)  PERIOD_GRLs (table)  Current TWiki includes Period to GRLs, so I added this table  This could evolve to include more tables for DQ/GRLs  PERIOD_REQUIREMENTS (column ?)  A new column or table in the schema (not existing now)  Each P_LEVEL=1 period will in future have a set of Conditions requirements (i.e. stable beams, measured luminosity) … from which candidate Runs for that Period can be generated.  I am unsure what form it will take … postpone until more information is in place … just basic entry is needed for now w/o this feature  3 tables (%_TO_% and %_GRLS): primary keys w/ multi-column index  AMI require these to have (additional) single-column index?  Any other suggestions for schema ?

13. Feb 2010Elizabeth Gallas - COMA / RunBrowser13 Period Information Service Interfaces

14. Feb 2010Elizabeth Gallas - COMA / RunBrowser14 Period Info Service: needed by other systems Various systems use Data Period information:  From COMA: TAG services using COMA can continue to use COMA  From AFS files: Need to provide the same information to those systems (put the AFS files behind us)  Create pyAMI (or other ?) service … Steps:  Agree on the schema  Contact developers: runQuery, DataQuality, DataPrep … (more ?)  Agree on basic services: input, output and format 1.“Period to Run service”:  Input Project, Period Name  Output: list of Run Numbers 2.“Run to Period service”:  Input: Run Number (and possibly P_LEVEL)  Output: list of Period names containing that Run  Develop these services with existing data  Give instructions to system how to use new service . Timescale: ASAP ! Give other systems time to adjust before mid-march 2011 collisions

15. Feb 2010Elizabeth Gallas - COMA / RunBrowser15 Entry Interfaces

16. Feb 2010Elizabeth Gallas - COMA / RunBrowser16 Data Period Entry Interfaces 1.Insert a new Data Period definition (PERIOD_DEFS table)  User authorization determines Domain (PDOM_NAME)  User enters:  P_Level, Project, Period name, description  (P_PROJECT_PERIOD = P_PROJECT + ‘.’ + P_PERIOD)  Initial status is ‘unlocked’ 2.Add members to an existing Period 1.Level 1: Add Runs to Periods ( PERIOD_P1_TO_RUNS )  Runs must be in COMA … provide users with a list of available Runs 2.Level 2: Add Periods to Groups ( PERIOD_PHI_TO_PLOS )  Provide user with Level 1 Periods candidates 3.Level 3: Add Period Groups to Parent Periods ( PERIOD_PHI_TO_PLOS )  Provide user with Level 2 Period candidates 3.Update a Data Period definition (PERIOD_DEFS)  User can change P_DESC (description) at any time  User can set P_STATUS from ‘unlocked’ to ‘locked’ 4.Associate a Period with a GRL (PERIOD_GRLS)  Would be good to integrate with GRL system information for integrity PERIOD_DEFS P – P_INDEX F – PDOM_NAME P_LEVEL ( 1, 2, 3 ) P_PROJECT_PERIOD P_PROJECT P_PERIOD P_DESC P_STATUS P_CREATE_USER P_CREATE_DATE P_MODIFY_USER P_MODIFY_DATE Timescale: Before mid-March First 2011 collisions

17. Feb 2010Elizabeth Gallas - COMA / RunBrowser17 Entry Interface Issues  Data Period Entry users must be experts  Experts must be aware of naming restrictions  If those restrictions can be coded, specify them  Certification options (2): 1.Need grid certificate on browser … in ATLAS VO 2.AMI login can identify users with particular roles  Special writer account ?  For Data Period related tables in COMA

18. Feb 2010Elizabeth Gallas - COMA / RunBrowser18 User Report Interfaces

19. Feb 2010Elizabeth Gallas - COMA / RunBrowser19 Reporting Services for Users  runQuery  switch from AFS file usage to new DataPeriod service utility  runBrowser and runBrowserReport  Will continue to include Period information (eg next slide)  Adjust to new schema  A preliminary runBrowserReport contains information as in the TWiki DataPeriods (next next slide)  I developed this to see the data entered  Assist with schema development, facilitate communication  Could be maintained … better optimized, organized … more information like luminosity, events added  The other option is to have this in AMI pages

20. Feb 2010Elizabeth Gallas - COMA / RunBrowser20 RBR Multi-Run Report: Click on a specific Run link:  RBR Single Run report Contains: basic information for the runs matching the input criteria Input criteria: printed in the header Click on SMK link:  RBR Trigger Super Master Key Report Links in Table column headers:  Short description of column Hover over link: Indicates what will happen Click on duck:  RBR documentation

21. Feb 2010Elizabeth Gallas - COMA / RunBrowser21  https://voatlas69.cern.ch/tagservices/dev/users/gallas/Run Browser/trunk/runBrowserReport/rBR_Period_Report.php https://voatlas69.cern.ch/tagservices/dev/users/gallas/Run Browser/trunk/runBrowserReport/rBR_Period_Report.php Period Documentation Front Page Purpose: Generate Period documentation for chosen input criteria Click on -/+ links: Expands to show The Periods Period Groups, Parent Periods

22. Feb 2010Elizabeth Gallas - COMA / RunBrowser22  https://voatlas69.cern.ch/tagservices/dev/users/gallas/Run Browser/trunk/runBrowserReport/rBR_Period_Report.php? pd=DP&fnt=data10_hi https://voatlas69.cern.ch/tagservices/dev/users/gallas/Run Browser/trunk/runBrowserReport/rBR_Period_Report.php? pd=DP&fnt=data10_hi Example Period Document (like TWiki) Prototype ! Format can be improved…

23. Feb 2010Elizabeth Gallas - COMA / RunBrowser23 Other unresolved issues  COMA Run loading -- now more time critical  Data Periods must be directly loaded in the COMA master  COMA must contain a Run to be included in a Data Period !  COMA loading cron executes daily on ATM@INT8R  INT8R is an integration database at CERN  Loading program: latest Runs in the TAG Catalog Schema  Usually within a day of Tier-0 processing, soon enough?  Modify Run loading to use Tier-0 DB rather than Catalog ?  Exports COMA tables to ATLARC then TAG remote sites  Production location issue:  TAG Catalog recently moved to ATLR  COMA should move there too  critical service should not be on the archive database ATLARC.  Revise replication plan ?  Data Prep should be mindful that Period replication not automatic  I don’t think we can use Oracle streams … schema is evolving

24. Feb 2010Elizabeth Gallas - COMA / RunBrowser24 Thanks and Apologies  Thanks to Beate and Solveig for useful discussions.  Apologies for  Things I forgot  I have some notes not cross checked to write slides …  After the meeting, I propose to update slides  And let you know I have done so  Belated meeting announcement  Belated sending of links  Was hoping to send  more finalized schema  Links in production locations  More complete documentation  BUT this schema is in development  Good point to get feedback now !

25. Feb 2010Elizabeth Gallas - COMA / RunBrowser25 Backup

26. Feb 2010Elizabeth Gallas - COMA / RunBrowser26 runBrowserReport: General Principles runBrowserReport  Purpose: Report interface for COMA  Display what COMA knows about Runs  Useful for experts/users (also for COMA loading and cross checks)  Provide links to more detailed information (in other systems)  runQuery, AMI, Trigger, Data Quality …reports using COOLCherryPy  Provide reports distinctive from other systems  (Avoid duplication of effort)  Display COMA ‘derived’ information  Not available in other systems  runBrowserReport php uses GET methods:  Inputs are not complex, user can modify URL to get report they want, put links into TWiki or other documentation https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=161562  Currently: ELSSI and runBrowser generate links to runBrowserReports for selections in those systems … links in AMI ???

27. Feb 2010Elizabeth Gallas - COMA / RunBrowser27 runBrowserReport: Reports available (example links) RBR Run Report (Run 161562): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=161562  General Run information, AMI tags, DQ, Prescale Evolution, Triggers  Trigger tables show derived Run-integrated prescale, passthrough RBR Master Key Report (Key 877): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?smk=877  Runs using this SMK, physics/commissioning chains, Level 1 items RBR Chain Report (Chain EF_g11_etcut): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?cn=EF_g11_etcut  Summary: SMK, Runs, Signature appearances of this chain  Runs with this chain ‘active’ RBR Chain Wildcard Report (chain name matching string EF_g1%): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?cn=EF_g1%  Summary: SMK,Runs with this chain, activation in Runs RBR Prescale Report (Run 162882 Prescales for chain EF_g11_etcut): https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=162882&cn=EF_g11_etcut  Derived chain activity over the Run and LB-wise prescales

28. Feb 2010Elizabeth Gallas - COMA / RunBrowser28 COMA runBrowserReport Index page  INDEX: https://atlas-tagservices.cern.ch/RBR/runBrowserReport.htmlhttps://atlas-tagservices.cern.ch/RBR/runBrowserReport.html  GUIDE: https://atlas-tagservices.cern.ch/RBR/rBR_Documentation.htmlhttps://atlas-tagservices.cern.ch/RBR/rBR_Documentation.html “Index” page:  Enter any input quantity,  IE: a Run number  IE: a Chain Name  Click on submit (or return key) Links at bottom of every page: RBR reports use “GET” methods: Modify the URL to change input RBR reports contain links No need to use index page for every query Links lead you to other COMA reports other external reports AMI, runQuery, TriggerDB… Bottom of each page

29. Feb 2010Elizabeth Gallas - COMA / RunBrowser29 RBR: Single Run Overview General Run info  Period, LB range, Date …  Links to COMA reports  Links to external reports Yellow sections expand (5):  AMI tags for this Run  In the TAG Catalogue  DQ LBSUMM assessments  COOL tagged/locked  Active Streams  Prescale Evolution  PS Key changes  Trigger section  HLT summary  HLT subsections  Physics  Commissioning  Level 1 Items Open this section in next slide…