The DAQ and IT infrastructures of KM3NeT Robert Lahmann (slides by Kay Graf) Earth and Sea Science with KM3NeT NCSR Demokritos, Athens, Dec. 2017

KM3NeT Data Management Plan (DMP) KM3NeT established a DMP for astrophysical data in KM3NeT-INFRADEV WP4 (Open Access Data) Basis: FAIR (findable, accessible, interoperable and reusable) data usage as defined by the EU Finding the best solutions and establishing pilot projects are aims of the KM3NeT-INFRADEV WP4 project (not: the full implementation and hosting of the services) Something similar should also be established for the ESS data (addition to DMP?) Key topics to address: data characteristics (types, formats, origins and sizes/bandwidth) user groups (communities, sizes, frequency of access) discoverability, identifiability, restrictions (authentication and authorisation) necessary tools to produce, make public, access and analyse the data data quality and persistency Community input is essential! ESS with KM3NeT - Dec. 2017 - KM3NeT DAQ & IT

KM3NeT Computing Model (for astrophysical data) Data abstraction (low level to high level) ESS with KM3NeT - Dec. 2017 - KM3NeT DAQ& IT

Data Processing Steps Data products Data formats: raw data (including metadata) from the detectors themselves – low-level monitoring data – including environmental data (from ESS instruments) ⇒ most interesting here calibrated data: adding detector status and calibration to the raw data; reconstructed data – high-level; simulation data; analysis results. Data formats: Root, HDF5, ascii for astrophysical data ESS with KM3NeT - Dec. 2017 - KM3NeT DAQ& IT

KM3NeT Data Storage and Resources KM3NeT internal data (see also next slides) administrated by the KM3NeT computing and software working group IT Services (documentation, information exchange) Database (persons, institutes, detector information, calibration data, data quality information) Raw data + processed data at large computing infrastructures Mainly: CC-Lyon (data access via iRODS), CNAF (data access via gridftp) Open access data: TBD → in consultation with WP4, WP8, Comp&Soft WG (for high-level astrophysical data usage of dedicated services or the Virtual Observatories is under consideration) Data services have to be tailored to the needs (data management plan); there is not “the solution” ⇒ we need to define those together ESS with KM3NeT - Dec. 2017 - KM3NeT DAQ& IT

IT Services Comp&Soft — K. Graf — Bootcamp Valencia, November 2017 6

IT Services ⇒ one service per task for links and information, see this wiki page db.km3net.de/km3netdbweb.in2p3.fr – db, user profile, data wiki.km3net.de – wiki indico.km3net.de – Indico meetings elog.km3net.de – electronic log books svn.km3net.de/trac.km3net.de – SVN and trac for software development also git.km3net.de → push to SVN jenkins.km3net.de – continuous software integration server drive.km3net.de – Google Drive for documents chat.km3net.de – quick chats (individual and channels) mailing lists see this wiki page support always available from admins@km3net.de Comp&Soft — K. Graf — Bootcamp Valencia, November 2017

Database Web Interface https://db.km3net.de ⇒ browsable data Comp&Soft — K. Graf — Bootcamp Valencia, November 2017

Database Access Database allows also data streams from detector (download data directly) km3pipe: python programme to access these data, example (humidity in DOM): Web access also possible (HTTP POST/GET commands) HTTP access! >>> humid = db.datalog("humid", run=4780, det_id="D_ARCA003") Database lookup took 3.931s (CPU 0.192s). >>> type(humid) <class 'pandas.core.frame.DataFrame'> >>> humid.head(3) RUN UNIXTIME SOURCE_NAME PARAMETER_NAME DATA_VALUE \ 0 4780 1478735722766 3.4.3.2/V2-2-1/2.138 humid 3694 1 4780 1478735732768 3.4.3.2/V2-2-1/2.138 humid 3694 2 4780 1478735742766 3.4.3.2/V2-2-1/2.138 humid 3694 ESS with KM3NeT - Dec. 2017 - KM3NeT DAQ& IT

Raw Data Access Example: root access All kinds of access, processing and conversion possible ESS with KM3NeT - Dec. 2017 - KM3NeT DAQ& IT