CRISP WP18, High-speed data recording Krzysztof Wrona, European XFEL PSI, 18 March 2013

2 High-speed Data Recording Objectives: 1.“High-speed recording of data to permanent storage and archive” 2.“Optimized and secure access to data using standard protocols” Partners: DESY, ESRF, ESS, European XFEL, GANIL, ILL, UOXF.DB, Univ. Cambridge

Task 1 3 Assembling requirements and use cases for high-speed data recording to storage systems and data archives. –Input collected as part of the IT&DM survey –Presentations by RIs, description of use cases, discussions, identification of common critical issues –Requirement document has been published (MS3) Reviewing available technologies, selecting tools, and investigating their usability for defined use cases –Survey on hardware and software technologies, exchange of experience

Identified synergies Using 10GE network for high throughput data transfers Online data processing models Writing data to storage Data archiving 4 Three categories: Relatively small data rates at ESS, ILL and SPIRAL2 (neutrons and ions oriented physics), High data rates expected at ESRF, EuroFEL and European XFEL (synchrotrons and FELs) Order of magnitude higher data rates at SKA (astrophysics) but expected few years later.

Network Tests Using 10GE network for high throughput data transfers –Results from 10GE tests presented at the October IT&DM topic meeting –Achieved: UDP and TCP protocols at wire speed Minimized UDP packet loss with advanced tuning Tuning summary –Linux kernel and driver parameters –NIC configuration –PCI bus –IRQ affinity –Disabling general system services Application tuning summary –Binding threads to cores 5

Local Buffers Implementation of local buffer at ESRF 6 Details in presentation by B. Rousselle How to saturate 10GE link through a remote filesystem Using http protocol for data transfer Interoperability between windows 32/64 and Linux OS

Online data processing Online data processing model at XFEL –Pipelining and multithreading on multi-core architecture –Multiple data channels to handle output stream from single detector –Receiving, processing, monitoring, formatting, and sending data to storage 7

Data recording Sending single channel formatted data through 10GE interface using TCP Storage performance: cached vs. direct IO Tests with two types of storage systems: –14 x 900GB 10Krpm, SAS, 2 x 6Gbps, RAID6 –12 x 3TB 7.2Krpm, NL SAS, 2 x 6Gbps, RAID6 8 Details in presentation by D. Boukhelef Results –Achieved data rate per channel: 1.1GB/s and 0.97GB/s, resp. –Direct IO improves performance and stability Currently investigated –Concurrent IO operations: write/write, write/read, file merging –Storage manager: file indexing, disk space management, dynamic data switching

Task 2 9 Collecting requirements for data protection and understanding their implications for high-speed data recording and data access. –Input collected as part of the IT&DM survay –Presentation from extended study case at ILL on October IT&DM topic meeting –Requirement document published as MS8

Data access protection 10 Highlights –Analysis of requirements for data protection Open access data policy with initial period when data access is restricted Large collaborations vs. small experimental teams Online data processing, access to data in the local buffers Long term storage of immutable data –Defining users and facility roles required for accessing and curating data –Extended study case at ILL Datasets identified by proposal id Heterogeneity of the environment, legacy protocols and applications Implementation based on filesystem ACLs –List of recommendations

Task 3 11 Defining and selecting use case applications requiring high throughput data access. Evaluating the usability of standard access protocols Defining data-access architecture Status: –Evaluation of several cluster filesystems (Fraunhofer, glustre), dCache implementation of (pNFS) NFS4.1, NetApp using NFS3 and NFS4.1 –Lesson learned so far Interplay between hardware, operating system, networks and applications Standard tools mdtest+iozone+h5perf help testing basic capabilities –Poor performance on tests -> poor performance in real applications –Good performance on tests does not imply good performance in real applications –Architecture document expected in month 24 (milestone)