Computing and Software – Calibration Flow Overview Kay Graf - ECAP Timecalibration Workshop Nikhef, May 2017

Calibration and Data Processing

Calibration and Data Processing Data processing and analysis Calibration application Calibration determination Interplay of calibration and analysis: Determination of calibration Application of calibration (to data) (Offline) Data processing: simulation, reconstruction, analysis Example from time calibration: analysis drives calibration Timecalibration WS - Comp&Soft — K. Graf — May 2017

Data Processing: Simulation Chain Simulation Data Format Event Generation Event Simulation and Cherenkov Light Generation Online Data Format Background Light Generation Detector Response Data Filtering Offline Data Format Event Classification Event Reconstruction Summary Data Fromat Analyses antcc-format (.evt) Geometry, Configuration, Monitoring and Calibration Data (File, DataBase, ascii, .detx) Jpp format (.root) Jpp root format/aanet (.root) aanet offline format (.root) Processing chains for simulation and data should be as equal as possible, i.e. use of the same algorithms/programs as much as possible Comp&Soft WG — K. Graf — KM3NeT Collaboration Meeting — February 2015

Data Processing: Data Chain detector offshore CLB – recording (Electronics WG) shore station DataQueue – receiving, sorting / DataFilter – filtering/DataWriter – writing online monitoring quasi-online Postprocessing/data quality –add info quasi-online processing – quick reco, high-level monitoring offline processing calibration / data quality reconstruction analysis at institutes (user) develop analysis software package UDP – CLB data format Jpp root format/controlhost Geometry, Configuration, Monitoring and Calibration Data (File, DataBase, ascii, .detx) Jpp root format offline root format (aanet) Processing chains for simulation and data should be as equal as possible, i.e. use of the same algorithms/programs as much as possible Comp&Soft WG — K. Graf — KM3NeT Collaboration Meeting — February 2015

Interface Calibration⬄ Simulation/Analysis Some thoughts: To allow for consistent calibration information parsing and efficient resource management (developing, testing and processing) calibration tasks/processing should have clear interace definitions i.e.: the determination of calibration can be done in analysis software but in a separate programs/algorithms That does not mean that development/improvement/testing of calibration data in analysis is discouraged, on the contrary – we calibration with drive analysis and vice-versa for the next years. Necessities: Clear interfaces of calibration data are needed → database Clear definitions for calibration streams/handling in the analysis software “intelligence” in the handling of calibration data to improve on calibration Timecalibration WS - Comp&Soft — K. Graf — May 2017

Combined fit ( line shape) → future Other calibrations General Scheme Detector Acoustic system Orientation system Database Position Calibration Positions Orientation Combined fit ( line shape) → future Other calibrations Offline Analysis/Reco (could extent to quasi-online) Timecalibration WS - Comp&Soft — K. Graf — May 2017

What we currently use in offline calibration Position/orientation in detx – derived from standard detector geometry hard coded values no orientation at all T0 in detx – derived from in situ measurements + dark room (see time calibration procedure) Detx format: Human readable Large overhead (4MB for a buidling block)– redundant information on DOM geometry) only contains position, orientation and T0 Other calibration parameters (especially from K40 analysis) As ASCII files Timecalibration WS - Comp&Soft — K. Graf — May 2017

global_det_id format_version\n UTC_validity_from UTC_validity_to \n Dataformats See http://www.wiki.km3net.de/index.php/Dataformats Detx (v2) global_det_id format_version\n UTC_validity_from UTC_validity_to \n UTM_ref_grid UTM_ref_easting UTM_ref_northing UTM_ref_z \n ndoms \n dom_id line_id floor_id npmts \n pmt_id_global x y z dx dy dz t0 \n pmt_id_global x y z dx dy dz t0 \n ... pmt_id_global x y z dx dy dz t0 \n #repeat for each dom Timecalibration WS - Comp&Soft — K. Graf — May 2017

What we need Possibility to stream all calibration data from the database to analysis codes Stream should be efficient (small, only necessary information) Unique identification of the “best” available calibration on an event basis NB: ”best” could change in time possibility to use an intermediate data format for caching For now: position/orientation data, however: scheme should be extendable also to other calibration data, e.g. from K40 analysis Timecalibration WS - Comp&Soft — K. Graf — May 2017

Calibration Data for Offline Analysis Discussion between comp&soft, calibration and DAQ group Improvement necessary in data bookkeeping and data transfer Many parameters handled via ASCII files (MonitorK40 output, PMT parameters, etc). Better quality control needed Challenge: find the appropriate, best calibration for each individual event; pull the necessary information from the database Strategy defined: the detx file contains all general detector information for a given run – as is the case now; Additional (light-weight) calibration stream to provide data changing during run Position/orientation, (PMT information) Hosted at separate tables in the DB that mark the interface from calibration to analyses Giorgio to finalise the proposal Comp&Soft WG — K. Graf — PSC Meeting, May 2017

Proposal Store data in dedicated DB table(s) Time info (UTC time) Detector ID Calibration tag (define e.g. online, offline v1, v2, v3 calibration sets) (RUN) DOM ID Position (x,y,z) Orientation (yaw, pitch, roll) ... Repeat 1-7 for all DOMs in detector or: repeat 5-7 for all DOMs in detector? Detx provides the general setting (detector setup; DOM ID to position in detector; DOM geometry; T0s) Position/orientation is streamed as ntuple ⇒ result: detector object in the individual analysis/reco code Timecalibration WS - Comp&Soft — K. Graf — May 2017

How to retrieve from and store to the database calibration-related information Timecalibration WS - Comp&Soft — K. Graf — May 2017

Database Interaction Via dedicated pages, see http://www.wiki.km3net.de/index.php/Database/Sitemap E.g. detx: https://km3netdbweb.in2p3.fr/detx, see http://www.wiki.km3net.de/index.php/Software/Detector_Descriptions for all available detx-files Stream Data Service: ascii file streams, see http://www.wiki.km3net.de/index.php/Database/Stream_Data_Service XML and JSON Data Service: xml, json data file up- and download, see http://www.wiki.km3net.de/index.php/Database/XML_Data_Service E.g. tests/calibration upload: http://www.wiki.km3net.de/index.php/Database/Product_Tests_and_Para meters Special file parsers (for Excel files): Registration and Integration data upload Timecalibration WS - Comp&Soft — K. Graf — May 2017

How to define tests/calibrations Define parameters: Define test type: Link test type with parameters: Upload tests via XML service Timecalibration WS - Comp&Soft — K. Graf — May 2017

Handle Calibrations via Tags http://www.wiki.km3net.de/index.php/Database/Calibration_Handling we can handle calibration sets – by combining calibrations ⇒ add to run generation → easy to generate (xml file upload possible) Timecalibration WS - Comp&Soft — K. Graf — May 2017