1. The DAQ and IT infrastructures of KM3NeT
Robert Lahmann (slides by Kay Graf)
Earth and Sea Science with KM3NeT NCSR Demokritos, Athens, Dec. 2017

2. KM3NeT Data Management Plan (DMP)
KM3NeT established a DMP for astrophysical data in KM3NeT-INFRADEV WP4 (Open Access Data)
Basis: FAIR (findable, accessible, interoperable and reusable) data usage as defined by the EU
Finding the best solutions and establishing pilot projects are aims of the KM3NeT-INFRADEV WP4 project (not: the full implementation and hosting of the services)
Something similar should also be established for the ESS data (addition to DMP?)
Key topics to address:
data characteristics (types, formats, origins and sizes/bandwidth)
user groups (communities, sizes, frequency of access)
discoverability, identifiability, restrictions (authentication and authorisation)
necessary tools to produce, make public, access and analyse the data
data quality and persistency
Community input is essential!
ESS with KM3NeT - Dec KM3NeT DAQ & IT

3. KM3NeT Computing Model (for astrophysical data)
Data abstraction (low level to high level)
ESS with KM3NeT - Dec KM3NeT DAQ& IT

4. Data Processing Steps Data products Data formats:
raw data (including metadata) from the detectors themselves – low-level
monitoring data – including environmental data (from ESS instruments) ⇒ most interesting here
calibrated data: adding detector status and calibration to the raw data;
reconstructed data – high-level;
simulation data;
analysis results.
Data formats:
Root, HDF5, ascii for astrophysical data
ESS with KM3NeT - Dec KM3NeT DAQ& IT

5. KM3NeT Data Storage and Resources
KM3NeT internal data (see also next slides)
administrated by the KM3NeT computing and software working group
IT Services (documentation, information exchange)
Database (persons, institutes, detector information, calibration data, data quality information)
Raw data + processed data at large computing infrastructures
Mainly: CC-Lyon (data access via iRODS), CNAF (data access via gridftp)
Open access data: TBD → in consultation with WP4, WP8, Comp&Soft WG (for high-level astrophysical data usage of dedicated services or the Virtual Observatories is under consideration)
Data services have to be tailored to the needs (data management plan); there is not “the solution” ⇒ we need to define those together
ESS with KM3NeT - Dec KM3NeT DAQ& IT

6. IT Services
Comp&Soft — K. Graf — Bootcamp Valencia, November 2017 6

7. IT Services ⇒ one service per task
for links and information, see this wiki page
db.km3net.de/km3netdbweb.in2p3.fr – db, user profile, data
wiki.km3net.de – wiki
indico.km3net.de – Indico meetings
elog.km3net.de – electronic log books
svn.km3net.de/trac.km3net.de – SVN and trac for software development
also git.km3net.de → push to SVN
jenkins.km3net.de – continuous software integration server
drive.km3net.de – Google Drive for documents
chat.km3net.de – quick chats (individual and channels)
mailing lists see this wiki page
support always available from
Comp&Soft — K. Graf — Bootcamp Valencia, November 2017

8. Database Web Interface
⇒ browsable data
Comp&Soft — K. Graf — Bootcamp Valencia, November 2017

9. Database Access
Database allows also data streams from detector (download data directly)
km3pipe: python programme to access these data, example (humidity in DOM):
Web access also possible (HTTP POST/GET commands)
HTTP access!
>>> humid = db.datalog("humid", run=4780, det_id="D_ARCA003") Database lookup took 3.931s (CPU 0.192s). >>> type(humid) <class 'pandas.core.frame.DataFrame'>
>>> humid.head(3)
RUN UNIXTIME SOURCE_NAME PARAMETER_NAME DATA_VALUE \
/V2-2-1/2.138 humid 3694
/V2-2-1/2.138 humid 3694
/V2-2-1/2.138 humid 3694
ESS with KM3NeT - Dec KM3NeT DAQ& IT

10. Raw Data Access Example: root access
All kinds of access, processing and conversion possible
ESS with KM3NeT - Dec KM3NeT DAQ& IT