Summary of Metadata Workshop Peter Hristov 28 February 2005 Alice Computing Day

Metadata: Definition Traditionally: Metadata has been understood as “data about data” Example(s): A library catalogue contains information (metadata) about publications (data) A file system maintains permissions (metadata) about files (data) More definitions: try Googletry Google

General Applications of Metadata (Web) Cataloguing (item and collections) Resource discovery Electronic commerce Intelligent software agents Digital signatures Content rating Intellectual property rights Privacy preferences & policies

Statement of the Problem A user wants to process “some events” He/she needs to know where they are Which file Where in the file Sounds simple!

Tools at Hand In Grid file catalogues files may have metadata that identify a file However we are not sure what the grid catalogue will look like in this moment Given a ROOT file, a TRef allows us to “jump” directly to a given object

General Idea on the Grid MD MD MD keys TAG Database Event catalogue MD MD MD LFN SE File catalogue MD MD MD PFN Local catalogue

Main Questions What happens if we are not on the grid The system should not be too different What do we put in each catalogue How much do we depend on the file catalogue being exposed to us Which information do we put in the TAG database Which are its dimensions and its efficiency?

Event Model RAW data; Written once, read (not too) many times. Size: 1 – 50 MB per event, exist only one per event. ESD; Written (not too) many times, read many times. Size: ~ 1/10 of raw per event, exist only one per event. AOD; Written many times, read many times. Size: ~ 1/10 of ESD per event, exist many (~10) per event. … Tag; Written (not too) many times, read many times. Size: 100 B – 1 kb per event, exist many per event. This is done for fast event data selection.  It’s not directly for analysis, histogram production etc.  Even (by chance) if the information is there you may do it. For discussion.  Global experiment tags.  Physics working group tags.  User defined tags.

Terminology Define common terms, first ● Metadata: Key-Value pairs Any data necessary to work on the grid not living in the files ● Entry: Entities to which metadata is attached Denoted by a string, format like file-path in Unix, wild-cards allowed ● Collection: Set of entries Collections are themselves entries, think of Directories ● Attribute: Name or key of piece of metadata Alphanumerical string with starting letter ● Value: Value of an entry's attribute Printable ASCII string ● Schema: Set of attributes of an entry Classifies types of entries: Collection's schema inherited by its entries ● Storage Type: How back end stores a value Back end may have different (SQL) datatypes than application ● Transfer Type: How values are transferred Values are transported as printeable ASCII

Event Metadata Tag: event building and physics related information Catalog information => how to retrieve the event What else => links to another sources of information?

Tag Database Karel Safarik

TAG Structure Event building information. Allows to find all the information about the event.  Event ESD and all the AODs.  Maybe also RAW data (hopefully will not be used often).  … (This is not my job). Physics information. Query-able (that’s on what you select data). Information about trigger, quality etc. Usually same global physics variable. But also one may have there which may not too much physical sense but is good for selection.

TAG Size Has to be reasonable to be able to query in reasonable time Somewhere around disk size --- O(100GB) Typical yearly number of events 10 7 for heavy ion 10 9 for pp However TAG size (in principle) is independent on multiplicity But it is collision-system dependent, trigger dependent… For heavy-ion: few kb gives few 10 GB For pp: 100 B gives 100 GB STAR: 500 physics tag fields in 0.5 kb (in average 1 par B)

TAG Content (Only physics information) Technical part – the same for every TAG database Run number, event number, bunch crossing number, time stamp Trigger flags (an event may be trigger by more than one trigger class), information from trigger detectors Quality information: which detectors were actually on, what was their configuration, quality of reconstruction Physics part – partly standard, partly trigger/physics/user dependent Charged particle multiplicity Maximum pt Sum of the pt Maximum el-mag energy Sum of el-mag energy Number of kaons …

TAG Construction Basic (experiment wide) TAG database Written during reconstruction – ESD production But it has also to navigate to (all ?) AOD (produced later) ? There is part which is untouchable (nobody is allowed to modify) There is part which maybe modified, as result of further analysis From this one all other TAG databases start The real content of definite instant of TAG database  Trigger dependent  Detector configuration dependent  Physics analysis dependent Define the physics group TAG databases  Derived from experiment wide database Maybe allow for user TAG databases  Derived from physics group database Useful tag fields are then pushed up in this hierarchy

TAG Conclusion We have to define prototype of experiment-wide TAG database Implement this in reconstruction program Physics working group – to define physic group databases Test the mechanism of inheritance from experiment-wide TAG database Decide if the ‘event building’ information has to allow to navigate  To all the AODs  Or just to those created within that working group When ?, who ?

Metadata in CDC Fons Rademakers

AliMDC - ROOT Objectifier ROOT Objectifier reads raw data stream via shared memory from the GDC Objectifier has three output stream: Raw event file via rootd to CASTOR Event catalog (tag DB) File catalog  In raw event file  In MySQL  In alien

RAW Data DB Raw data file contains a tree of AliEvent objects: An AliEventHeader object  16 data members (72 bytes) An TObjArray of AliEquipment objects  An AliEquipmentHeader object 7 data members (28 bytes)  An AliRawData object Char array (variable length) An TObjArray of sub-events (also AliEvents) No compression (need more CPU power) Size of individual raw DB files around 1.5 GB

Event Catalog - Tag DB The tag DB contains tree of AliEventHeader objects: Size, type, run number, event number, event id, trigger pattern, detector pattern, type attributes Basic physics parameters (see Karel’s talk) Compressed Used for fast event selection Using compressed bitmap indices (as used in the STAR grid collector)? Do we store LFN’s for the events or do we look the LFN’s up in the file catalog using run/event number? Might need more than one LFN per event (RAW, RECO, ESD, AOD), or do we use naming conventions?

File Catalog The file catalog contains one AliStats object per raw data file: Filename of raw file, number of events, begin/end run number, begin/end event number, begin/end time, file size, quality histogram Same info also stored in a central MySQL RDBMS In AliEn catalog only: LFN, raw filename, file size  LFN: /alice_md/dc/adc- /  : _ _.Root

File/event Collections Not yet addressed A file collection can be one (or more) sub- directories in alien (with symbolic links to original LFN)? An event collection can be a file collection with an associated event list per file? Or fully dynamic: a collection is just a (stored) query in the event and file catalogs (like grid collector)

ARDA Experiences With Metadata Nuno Santos

Experience with Metadata ● ARDA tested several Metadata solutions from the experiments: ● LHCb Bookkeeping – XML-RPC with Oracle backend ● CMS: RefDB - PHP in front of MySQL, giving back XML tables ● Atlas: AMI - SOAP-Server in Java in front of MySQL ● gLite (Alien Metadata) - Perl in front of MySQL parsing command, streaming back text ● Tested performance, scalability and features => a lot of plots omitted, see the original presentation.

Synthesis ● Generally, the scalability is poor ● Sending big responses in a single packet limits scalability ● Use of SOAP and XML-RPC worsens the problem ● Schema evolution not really supported ● RefDB and Alien don't do schema evolution at all. ● AMI, LHCb-Bookkeeping via admins adjusting tables. ● No common Metadata interface Experience with existing Software Propose Generic Interface & Prototype as Proof-Of-Concept

Design decisions ● Metadata organized as a hierarchy - Collect objects with shared attributes into collections ● Collections stored as a table - allows queries on SQL tables ● Analogy to file system: ● Collection Directory ● Entry File ● Abstract the backend – Allows supporting several backends. ● PostgreSQL, MySQL, Oracle, filesystem… ● Values restricted to ASCII strings ● Backend is unknown ● Scalability: ● Transfer large responses in chunks, streaming from DB ● Decreases server memory requirements, no need to store full response in memory ● Implement also a non-SOAP protocol ● Compare with SOAP. What is the performance price of SOAP?

Conclusions on SOAP Tests show: SOAP performance is generally poor when compared to TCP streaming.  gSOAP is significantly faster than other toolkits. Iterators (with stateful server) helps considerably – results returned in small chunks. SOAP puts an additional load on server. SOAP interoperability very problematic. Writing an interoperable WSDL is hard. Not all toolkits are mature.

Conclusions ● Many problems understood studying metadata implementations of experiments ● Common requirements exist ● ARDA proposes generic interface to metadata on Grid: ● Retrieving/Updating of data ● Hierarchical view ● Schema discovery and management (discussed with gLite, GridPP, GAG, accepted by PTF) ● Prototype with SOAP & streaming front end built ● SOAP can be as fast as streaming in many cases ● SOAP toolkits still immature ●