HDF5 format at European XFEL Djelloul Boukhelef Data processing workshop Eu-XFEL, 03-04 Feb. 2016

The Big Picture The HDF5 format to be used at European XFEL

Data repository structure Reflects metadata models for raw and calibration data Data retention policy, different life time of data depends on data type Time period: facility cycle (campaign) [yyyyNN] year + 00 [yyyy00] for non facility cycle related data Instrument (SPB, SQS, FXE, HED, MID, SCS, TestBed1, …) Data types: raw/intermediate/resultant Raw data repository structure per instrument reflects metadata model BeamTime + Experiment + Run Calibration repository structure per instrument reflects calibration DB model Possible raw data repository structure: Facility BeamTime Experiment Id Run Type TimePeriod XFEL bt0110 exp01 r0023 raw 201601 Instrument FXE

Data categories and Naming convention /detlab_det_lpd/fem/1/femVoltage0 Control data Instrument data Domain Type Instance property FEM Ctrl device header images PC layer device descriptor detector raw Digitizer device processed

File structure RAW Data structure within file facility – description of facility, beamline and instrument user – description of user (PI, operator, …) control – slow data instrument – detectors data User data (example) address affiliation email facilityUserId faxNumber name role telephone Number Instrument data UniqueDataSourceId – type of the data source and its instance /instrument/detlab_det_lpd/2d/lpd/ DetectorDataFormat – generalized 2D detector data format /instrument/detlab_det_lpd/2d/lpd/var_pulse_data/data /instrument/detlab_det_lpd/2d/lpd/var_pulse_data/pulseId /instrument/detlab_det_lpd/2d/lpd/pulse_data/cellId

Physical HDF5 file structure Sequencing Possibility of storing a sequence of data records Unique sequence id at XFEL (trainId, pulseId) Pulse data (Descriptors, image) Train data (header, trailer, det. spec., monitor data) Idx train Train Id Pulse Id Image cellId 234 150 2 843 34 1 250 14 45 55 91 32 Train Id A B C D 234 1003 11.85 44.34 12.8 1 250 1005 12.16 Issue: what to do with removed images?

Internal HDF5 file structure Sequencing Follow relational database model? How far we can go? Train data (header, trailer, det. spec., control data) Train Id A B C D 234 1003 11.85 44.34 12.8 250 1005 12.16 Optimize for size and create indexes to navigate between different entities Pulse data (descriptors) Variable pulse data (Images) Idx Variable pulse data Idx train data Train Id Pulse Id cellId status -1 234 150 2 1 843 34 250 14 45 3 55 4 91 32 Idx pulse data Train Id Pulse Id Image 1 234 2 843 4 250 45 5 55 6 91

Data acquisition and aggregation at PC layer Metadata Catalog Register data files and metadata List of files Run-number PCL0 PCL1 PCL2 PCL3 AGG0 Reader /r0001/r0001.h5 /pcl0 /pcl1 /pcl2 /pcl3 /agg0 0 1 2 3 4 5 … 0 1 … Full images Dispatcher Match train-id to filename using Seq(T0,NC,NTF) Match filename to node/folder using inverted index Using symbolic links Using reverse formula (discovered from data) Using dictionary (filenamelocation) Modules-wise Cal0 Cal1 Cal2 Cal3

Correlation of instrument and control data Sequence number can be calculated assuming statically configured distribution of train data over multiple channels T Train Id T0 First train Id within the run Nc Number of data acquisition channels. N Number of train blocks per file c Channel id. Channels are ordered. i Record id for train based data within a table {i=0,…,N-1} m File sequence number within the channel s File sequence number within the run For the consecutive train ids: 𝑐 𝑇 = 𝑇− 𝑇 0 𝑚𝑜𝑑 𝑁 𝑐 𝑚 𝑇 = 𝑇− 𝑇 0 𝑁 𝑐 ×𝑁 𝑖 𝑇 = 𝑇− 𝑇 0 𝑚𝑜𝑑 𝑁 𝑐 ×𝑁 𝑁 𝑐  𝑠 𝑇 =𝑚 𝑇, 𝑁 𝑐 ,𝑁 × 𝑁 𝑐 + 𝑐 𝑇, 𝑁 𝑐 Can be generalized for periodic train patterns If random trains are delivered use tabular index (not likely) Example: T0 = 0, Nc=3, N=2 Where is train T = 10? c = 1, m = 1, i = 1, s = 4 T0 = 0, Nc=3, N=2 These parameters are stored as group attribute in the main file

Run data files (aka data group) Data files stores datasets Each may contain data from one or multiple data sources or properties Stub file Unique per data group or run Point to actual files that store datasets Indicate formula (e.g. HDF5 attributes to calculate the actual set of files that store a given dataset Dataset in different files groups might be index using different parameters /detlab_det_lpd/2d/lpd/images/ /detlab_det_lpd/fem/fem1/… /detlab_det_alas1/vac/… /detlab_det_alas1/motor/… R0001.h5 Stub file r0001-agg1-s0000.h5 AGG1 r0001-agg1-s0001.h5 r0001-agg1-s0002.h5 r0001-agg1-s0003.h5 r0001-agg0-s0000.h5 AGG0 r0001-agg0-s0001.h5 r0001-lpd-s0000.h5 r0001-lpd-s0001.h5 r0001-lpd-s0002.h5 r0001-lpd-s0003.h5 r0001-lpd-s0004.h5 r0001-lpd-s0005.h5 PCL0 PCL1 PCL2 PCL3

Full path to data Path to data contain four parts: Filesystem path (prefix): this is the same for one data group File location and filename Due to technical reasons, PCLayer nodea or aggregatora stores data from the same run int separate folders (e.g. local folders in HS setup) File location is configuration parameter: e.g. PCL0, PCL1, AGG0, … File names are calculated using Seq(T0,NC,NTF) Data-source id Experiment-wide unique identifier of Karabo data-source Hierarchical: location/data-source/property Data file contain the full data-source id Names of files contain (infix) the location/instance-id Example: r0001-lpd-s0001.h5, r0001-agg0-s0000.h5 Infix: configuration parameter that identifies the group of datasets Stored in the stub file along with the formula