QC database requirements and tools in Run3 B. von Haller CERN 21.07.2017

QC-specific database(s) vs aggregated data database(s) Outline Scope Run 3 vs Run 2 Database vs client QC-specific database(s) vs aggregated data database(s) Outline Architecture Requirements Possible solutions B. von Haller | WP7 | 21.07.2017

Reminder B. von Haller | WP7 | 21.07.2017

Actually repositories ? QC repository Actually repositories ? Generic client (shifters, experts) Specific clients (experts) Client Interface Interface “ALICE Aggregated data” (QC, logbook, CCDB, …) “Raw QC” (~Histos + metadata*) From sync and async QC tasks “Derived QC” (Trending and correlation) Interface B. von Haller | WP7 | 21.07.2017

Preservation and backups Is there something missing ? Requirements Types of data Amount of data Sources Access Preservation and backups Is there something missing ? Review of each point in next slides B. von Haller | WP7 | 21.07.2017

Requirements Types of data MonitorObjects (MO) : TObject (mostly histos) with metadata (e.g. quality, source...) (already merged) Trending : derived data under the form of histos or graphs or trees (to be clarified) with metadata (e.g. source MO) Correlations : derived data under the form of histos (to be confirmed). Does it need to be actually stored or could it be generated in memory on the fly ? B. von Haller | WP7 | 21.07.2017

Requirements Amount of data 25000 MOs updated every minute (i.e. a new version comes in every minute) From survey: 10000 (but we don’t believe it) 50% to be kept for 1 month, 40% for 1 year max, 10% forever 1 MO between 550b and 50MB, average 250 kB (online) Trending : at least 15 detectors* 10 objects/detector = 150 To be kept forever Correlation : ? inserts/update : > 400Hz, 100MB/s 6 GB per run initially (3GB after 1 month, 0.6 after 1 year) 2016 : 2300 global runs with recording, 2500 standalone runs with recording 2.8TB per year to be kept forever, 3TB for last month, 5TB for last 6 months (Actually a lot less : standalone runs have 1/15 of the data of a global run) B. von Haller | WP7 | 21.07.2017

Requirements Sources Mergers running inside the O2 farm (getting their data from QC tasks and other O2 devices) Processes running outside the O2 farm when offloading synchronous processing and asynchronous final processing B. von Haller | WP7 | 21.07.2017

The results of the QC must be available worldwide Requirements Access The results of the QC must be available worldwide Well defined and stable interface hides the underlying technology Access limited to the members of the Collaboration. A public access shall be granted to a selected set of interesting data and results for Public Relations (PR). Data should be queryable (filters at least) SWAN support (?) B. von Haller | WP7 | 21.07.2017

It should therefore support schema evolution. Requirements Preservation and backups QC data is to be kept forever or for a limited duration, depending on the detectors and the tasks. It should therefore support schema evolution. Backups must ensure that data can be recovered at any time in case of major failure B. von Haller | WP7 | 21.07.2017

Is there something missing ? Review of each point in next slides Solutions File-based database SQL database noSQL database CCDB HDFS Is there something missing ? Review of each point in next slides B. von Haller | WP7 | 21.07.2017

Data in (ROOT) files [and metadata in a DB on top] Solutions File-based Data in (ROOT) files [and metadata in a DB on top] Current scheme for Run 2 FXS, OCDB, offline QA Used in Overwatch prototype Concerns Non atomic operations Scaling (number of files and load on metadata server) Archiving not trivial B. von Haller | WP7 | 21.07.2017

Metadata and data (blob) are stored in an SQL database Solutions SQL database Metadata and data (blob) are stored in an SQL database Currently used in : DQM (MySQL) Prototype for QC exists and benchmarked Concerns : Backing up is not trivial on large DB continuously used B. von Haller | WP7 | 21.07.2017

Store metadata and data Solutions noSQL database Store metadata and data Maybe not for the Raw QC but for the Derived QC Prototype exists for trending (ElasticSearch) Concerns : Querying can be complex Data format conversion needed for storing and retrieving Size on disk (?) B. von Haller | WP7 | 21.07.2017

Do not solve the problem ourselves but rely on the CCDB Solutions CCDB Do not solve the problem ourselves but rely on the CCDB Either the main CCDB or a dedicated QC CCDB Concerns Are our requirements possible with the CCDB as envisaged ? How to use it on a development system (detector expert station) ? B. von Haller | WP7 | 21.07.2017

I don’t know this system but it seems promising. HDFS I don’t know this system but it seems promising. B. von Haller | WP7 | 21.07.2017

Clarification on the various subsystems involved Summary Clarification on the various subsystems involved Requirements for the Raw and Derived Database(s) List of possible solutions Discussion Then : interface definition, more prototyping with possibility to switch between backends B. von Haller | WP7 | 21.07.2017

If you want to participate in these discussions but were not involved until now : send me an email ! Requirements work document : https://docs.google.com/document/d/1npTUMwosuwHnvdD3Mb9hZRTBwG5TWAgsGUlUvp22jAk B. von Haller | WP7 | 21.07.2017