Metadata in ATLAS Elizabeth Gallas – Oxford 2nd ATLAS-South Caucasus Software/Computing Workshop & Tutorial October 26, 2012

 Metadata  What is it ? Challenges in ATLAS  Survey some user oriented systems using Metadata  Describe a few areas: metadata in evolution  Dataset Nomenclature, PhysicsShort, AMI tags  Delicate balance between constraints and usability  Focus on Datasets containing events in this talk  Transforms and metadata  AMI Hierarchical Search  New interface … aim to help metadata issues in MC  COMA Content evolution and aggregation  Runs, Periods, and Conditions DB management metadata  Summary and Conclusions Metadata … Outline Oct 2012E Gallas / Metadata2

 What is metadata ?  Concisely:  “data about data”  More precisely:  “data used to describe the context, content or structure of data”  Structural or Descriptive  Metadata: used extensively in ATLAS … in fact, no process in ATLAS doesn’t use metadata … examples range from:  Upstream: data taking with the correct calibrations …  Downstream: physicist finding Events of interest …  Metadata challenges:  Size/Scope of ATLAS data … Volume/Diversity of metadata  Data/metadata: has grown organically as the experiment evolved  Try to offer a coherent / integrated view to physicists while devising strategic placement for processing and analysis  Impossible to inventory all ATLAS Metadata in reasonable time  Today: highlight a few areas of user metadata which play important roles in ATLAS which are currently evolving  Note: unable to cover many areas Metadata ? Oct 2012E Gallas / Metadata3

 Subsystem specific metadata: driven by subsystem specific needs  Trigger: TriggerTool … Web: trigconf and trigger timelines  Geometry DB: Detector Description Browser  Conditions DB:  runQuery (Run information from Conditions DB)  ATLAS WEB DQ  COOL Tag Browser  Data Summary Reports (Luminosity, Beam)  Beam Spot Summary  GANGA and PAthena  Panda / monitor  DQ2 Client  Tag Collector – software releases ... (not a complete list !)  Dedicated Metadata Catalogs  TAGs (and TAG Catalog) – event level metadata  iELSSI and Suite of TAG Services  AMI – Datasets, processing … other metadata  And the AMI Suite of services  COMA – Run/LB level Conditions and configuration  Plus Conditions DB management metadata  Important metadata facilitator: ATLAS Job Transforms ATLAS Metadata User Application Overview Oct 2012E Gallas / Metadata4

 Dataset names used extensively:  Storage and operating systems, DDM, ProdSys, Metadata repositories  But needs to be mnemonic from user point of view  Dataset naming rules:  Carefully defined by experts, evolved somewhat, has served us well  But was last updated in 2010 … needs of ATLAS have grown  2012 Task Force formed to try to amend the rules to address these needs   Overall length < 231 characters (base directory name): Hard limit !  If each field at field limit, overall limit is exceeded !  Many pressures on component lengths … highest areas of concern:  “physicsShort” – for MC datasets  AMI tag – for both data and MC  Importance of Name: coherence must be understood at all ATLAS levels  From Management to Users … and sometimes limits are good  Keep a rational balance !!! Dataset Names Oct 2012E Gallas / Metadata5 Project.datasetNumber.physicsShort.productionStep.dataType.AMITag[/] Project.runNumber.streamType.productionStep.dataType.AMItag[/] dataNN_* or mcNN_* ESD, AOD, … Concatenation of configurations

 Example of one proposed “Physics Short”: MadgraphPythia8_NNPDF21NLOME_AU2NNPDF21LOMPI_SingleTopTChanWelenu_LeptonFilter  Rules: physicsShort field must not exceed 40 characters.”  This one: 78 characters (and is it really user friendly ?)  This kind of ‘growth’ is oblivious to the rules, shows addiction of experts/users to depending entirely on the Dataset Name to identify/find their data  General frustration  finding MC needed, Twiki pages, understanding the MC they use, and identifying additional MC samples they need or what exists …  Jamie Boyd: “General feeling is this level of info should be encoded in AMI rather than the filename – need to follow up with generators group on this”  Progress in 2012:  “Simulation Metadata Workshop” – held in April 2012  Commendable effort by MC Coordination: add more metadata to AMI  Metadata systems need to provide better tools which  Better explains relays the metadata behind the dataset AND  Better allows browsing of the datasets and the metadata “PhysicsShort” for MC Oct 2012E Gallas / Metadata6 Project.datasetNumber.physicsShort.productionStep.dataType.AMITag[/]

 AMI: ‘ATLAS Metadata Interface’  AMI Portal Page:  Means of finding datasets to use in analysis using physics metadata predicates  Not just dataset names, but also the underlying metadata  AMI contains a LOT more than just a list of datasets  Dataset provenance  Files  Lost Lumi blocks  Links to other applications  Nomenclature reference tables  New AMI interface: Dataset ‘browsing’ (hierarchical search)  Now available to users (first version !)  Good feedback from users … important for evolution  AMI team working on refining this tool based on feedback AMI Introduction Oct 2012E Gallas / Metadata7

 User is guided to the AMI catalog specific to the project of interest  Information varies according to project  Allows users progressive selection to iteratively narrow result set AMI Dataset Browsing Oct 2012E Gallas / Metadata8

 The AMI Tag:  Definition:  Is composed of concatenated strings encoding processing steps  Example: r2713_p705 … encodes information about  which ATLAS releases ( )  which database releases ( )  which transforms (reco_trf.py), job configurations, …  Why is it called the “AMI tag” ?  AMI provides interfaces for its interpretation  Rules for AMI tags also listed in Nomenclature doc  Original specification now also needs revision  Max length sometimes exceeds limit (22) – multiple factors driving this …  Highlight some issues to be addressed:  Running out of lower case letters  Numeric parts … require more characters (99 … 999 … 9999 ….?)  More processing/merging steps: add more/more fields  Must find a way to consolidate steps in a managed way  These issues also being discussed in the Task Force  Solveig Albrand: evolving document describing issues/possibilities  AMI (“Config”) Tags Oct 2012E Gallas / Metadata9

Transforms and Metadata  “ATLAS Transforms”: a wrapper to Athena & python job options  Thanks to the Transform Group !  Graeme Stewart, Stephen Beal, Thomas Gadfort, Harvey Maddocks, Bjorn Sarrazin  See Graeme’s talk during Software week (last week)  Required, for example, by the ATLAS production system  Provides uniform, coherent mechanisms for specifying, executing tasks  Even multi-step transforms  New Transforms:  General merging capabilities  Also need for the merging of file based metadata  Provide important computations  Such as Event counts  Bridges the gap in metadata communication  uniform information transfer to other systems and metadata repositories Oct 2012E Gallas / Metadata10

E Gallas / Metadata What is COMA ?  COMA: “Conditions/Configuration Metadata for ATLAS”  Originally: built to support dynamic queries of TAG DB  Evolved: into a standalone system with interfaces  Now part of general effort to consolidate/relate ATLAS Metadata  COMA Components:  Relational Database  Contains extracted, refined, reduced, and derived information from system specific data sources (e.g. Conditions DB, …) plus information from non-database sources.  Interfaces  A set of unique Browsers and Reports  Special relationship to AMI  Enhances COMA in many ways (e.g. Data Periods);  pyAMI methods make COMA information available to many systems.  Main Documentation  COMA Portal (grid certificate in browser is required) Oct

 A Data Period is a set of ATLAS Runs grouped for a purpose  Defined by Data Preparation Coordinators  Used in ATLAS data processing, assessment, and selection …  Each Period uniquely defined with a combination of  Project name (i.e. ‘data10_7TeV’)  Period name (i.e. ‘C1’, ‘C2’, ‘C’, ‘AllYear’ …)  Before 2011, Data Periods were  Described on TWiki page   Stored in a file based system  Edited by hand by Data Prep Coordination (experts)  Structure evolved over 2010 with experience  This experience  valuable to decide/define long term solution  In 2011: Data Periods stored in the COMA DB  Coordination/Effort: Data Prep, AMI, COMA experts  Since then, COMA has enhanced content in many areas  Allows for more details reports and information to other systems  Enables aggregation of this information by Run, … Period. COMA: ATLAS Data Periods … + aggregating new content Oct 2012E Gallas / Metadata12

Shows “at a glance”: the latest Period Runs with Magnet states, ‘ready fraction’, link to Stable Beam fill(s), beam information … Oct 2012E Gallas / Metadata13 COMA multi-Run report: Latest 8 runs

New aggregated information Oct 2012E Gallas / Metadata14 COMA Period Documentation Report: enhanced content

Conditions DB Management Metadata ATLAS Conditions Database: Non-event wise conditions and configuration information  Designed for primary purpose, extensive documentation  Processing of event-wise data  IOV (interval of validity) based structure is ideal  To non-experts  Mysterious, cryptic, specialized  What’s a COOL Global Tag ???  DB and COOL Tag Coordination need overview  Storage, tools based on individual schemas: make Overview difficult  Extra part of COMA: COOL Management Metadata (for Folders & Tags)  July 2011 SW week - Discussion with Lasha Sharmazanashvili  Then with Misha Borodin, Andrea Formica, Paul Laycock … other experts  NOT intended to replace existing Storage or tools  Makes possible:  Quicker access to overview information from external systems  Web based reports  A different view and can offer links to other metadata tools  Since then  In production operation (daily synchronization)  Other applications start development to incorporate it (Such as COOL Tag Browser)  Now working on:  Improving content and interfaces  Speeding up synchronization  Possible incorporation of “Current”, “Next” … COOL Tags into COMA (move from AFS files) Oct 2012E Gallas / Metadata15

E Gallas / Metadata COMA Conditions DB Folder Browser Menu  Enter criteria into textbox at left:  Type manually or  Click on options at right or re-generates Menu applying selection  Choose for the Global Report  Choose for the Folder Report Oct

E Gallas / Metadata COMA (Multi) Global Tag Report  COMA_CBAMI_GTAGS table  AMI Team has populated with counts showing Global Tag usage in dataset processing Oct

 A lot of progress in many areas using metadata:  Transforms, Data Processing, Dataset related metadata  Dedicated Metadata Catalogs: AMI, COMA, (TAGs)  Metadata in ATLAS continues to evolve  Naming conventions/rules  Important to form coherent view over datasets, runs, periods, …  Increased cooperation between systems  Upstream and downstream  Use cases continue to expand  Improvements in metadata  Storage  Consistency  Delivery  Usage  Challenges ahead  Offer coherence at Management and User levels  To keep pace with  system evolution (such as DDM  Rucio, ProdSys, … upgrades)  Analysis pattern evolution and use cases Summary and Conclusions E Gallas / MetadataOct

Oct 2012E Gallas / Metadata19 Backup

 Motivated by: COOL Tag Coordination, Upcoming reprocessing, LS1  Significant new “Expert level” selection criteria:  allows viewing of Folders, Folder Tags, and Global tags by, for example, creation dates (or ranges) by themselves or in combination with any of the other criteria.  New selection criteria: Open Expert Section of the Folder Browser Menu  Example links: Global Tags created since March 2012: CBAction=GlobalTagReport&cbgtdate= Global Tags created in the last 30 days: CBAction=GlobalTagReport&cbgtdate=latest30 Global Tags for MC: CBAction=GlobalTagReport&cbgt=OFLCOND* Global Tags for real data: CBAction=GlobalTagReport&cbgt=COMCOND* Oct 2012E Gallas / Metadata20 COMA COOL Folder&Tag Metadata: Report improvements