WP18: High-Speed Data Recording Krzysztof Wrona, European XFEL 07 October 2011 CRISP

Objective Provide solutions for:  high-speed recording of data to permanent storage and archive  optimised and secured access to data using standard protocols 2 K.Wrona CERN,

Motivations The rapid developments and increasing complexity of experimental techniques in many scientific domains and the usage of highly advanced instruments and detectors results in extremely high data rates exceeding tens of GB/s. Cost effective recording of data to storage systems and archives becomes an increasingly complex and challenging task, especially in cases where real time data reduction is not possible and the complete data streams must be recorded on the storage systems  no simple selection criteria, noisy images  experiment lasts for very short time (scale of hours or days)  experiment setup is changed frequently (feedback) K.Wrona CERN,

Initial considerations Aim for cost-effective solution Data originates in multiple sources Integration of commercial devices Parallel data streams for transfer optimization Writing data to storage system with high throughput (~10GB/s per device) Writing data to the archive  Splitting data streams for archiving and offline storage  Concurrent write and read K.Wrona CERN,

Participants Participating institutes/contact persons  XFEL (36pm) – K.Wrona  ESRF (12pm) – A.Goetz  DESY (19pm) – F.Schluenzen  ESS (8pm) – K. Leffman, S.Skelboe  ILL (2pm) – J.F. Perrin  UOXF.DB (36pm) – D.Wallom  GANIL (1pm) – ? 5 K.Wrona CERN,

Task 1 Assembling requirements and use cases for high-speed data recording to storage systems and data archives.  Description of use cases  Identification of common critical issues  Requirement document by month 9 Reviewing available technologies, selecting tools, and investigating their usability for defined use cases  Survey on hardware and software technologies  Exchange of experience  Mapping concrete solutions to identified use cases Coordination activities  Monthly video conference/phone meetings  Face-to-face meeting after 6 months 6 K.Wrona CERN,

Task 2 Collecting requirements for data protection and understanding their implications for high-speed data recording and data access.  Description of use cases  Definition of data protection model  Integration with authentication system (WP16)  Requirement document (month 12) Evaluating existing data-protection schemes including simple and advanced access-control models (i.e. NFSv4 and POSIX ACLs) and mapping them according to the defined requirements Coordination activities  see task 1 7 K.Wrona CERN,

Task 3 Defining and selecting use case applications requiring high throughput data access.  Selection of several flagship applications  Definition of requirements and benchmark criteria Evaluating the usability of standard access protocols  i.e. NFS4.1 (pNFS)  integration issues with storage, network and computing infrastructures Defining data-access architecture  Parallel access, caching strategies, etc  Architecture document (month 24) 8 K.Wrona CERN,

Task 4 Implementing the prototype system for the selected data- protection and data-access models according to the architecture design.  Complete test bed system will be implemented at selected facility(ies)  Other institutes may implement system partially Re-evaluating and refining the system architecture. Improving implementation. Deploying the prototype system and demonstrating its functionality. 9 K.Wrona CERN,

Timeline Overall timeline  Definition of use cases and requirements – start 1m  Evaluation of software and hardware components – start 10m  Designing system architecture – start 10m  Implementation and testing – start 24m  Improvements and optimization – 30m  Report drafting – 33m 10 K.Wrona CERN,

Milestones & Deliverables M1. Requirements for data recording to storage media  31 June 2012  Document approved by all WP members M2. Identification of data protection requirements and storage implications  30 September 2012  Document approved by all WP members M3. High-speed data access architecture design  30 September 2013  Architecture document approved by all WP members D1. Report on prototype system and future work  30 September K.Wrona CERN,