WP18, High-speed data recording CRISP WP18, High-speed data recording Krzysztof Wrona Hamburg, 8th December 2011

K.Wrona, Harmonzation meeting WP18 objective Provide solutions for: high-speed recording of data to permanent storage and archive optimised and secured access to data using standard protocols 08/12/2011 K.Wrona, Harmonzation meeting

K.Wrona, Harmonzation meeting Participants RIs ESRFUP, EuroFEL, European XFEL, ESS, ILL2020, SKA, SPIRAL2 Institutes ESRF, DESY, European XFEL, ESS, ILL,UOXF.DB, GANIL 08/12/2011 K.Wrona, Harmonzation meeting

K.Wrona, Harmonzation meeting Motivations Extremely high data rates Exceeding tens of GB/s per data source Especially when triggers cannot be applied Cost effective recording of data to storage systems and archives becomes an increasingly complex and challenging task experiment setup is changed frequently Aggregation of data from multiple sources Ensuring integrity of data through the full acquisition chain How to store, archive and access data What are the implications from data protection models 08/12/2011 K.Wrona, Harmonzation meeting

K.Wrona, Harmonzation meeting Task 1 Assembling requirements and use cases for high-speed data recording to storage systems and data archives. Description of use cases Identification of common critical issues Requirement document by June 2012 Reviewing available technologies, selecting tools, and investigating their usability for defined use cases Survey on hardware and software technologies Exchange of experience Mapping concrete solutions to identified use cases 08/12/2011 K.Wrona, Harmonzation meeting

K.Wrona, Harmonzation meeting Task 2 Collecting requirements for data protection and understanding their implications for high-speed data recording and data access. Description of use cases Definition of data protection model Integration with authentication system (WP16) Requirement document by Sep 2012 Evaluating existing data-protection schemes including simple and advanced access-control models (i.e. NFSv4 and POSIX ACLs) and mapping them according to the defined requirements 08/12/2011 K.Wrona, Harmonzation meeting

K.Wrona, Harmonzation meeting Task 3 Defining and selecting use case applications requiring high throughput data access. Selection of several flagship applications Definition of requirements and benchmark criteria Evaluating the usability of standard access protocols i.e. NFS4.1 (pNFS) integration issues with storage, network and computing infrastructures Defining data-access architecture Parallel access, caching strategies, etc Architecture document by Sep 2013 08/12/2011 K.Wrona, Harmonzation meeting

K.Wrona, Harmonzation meeting Task 4 Implementing the prototype system for the selected data-protection and data-access models according to the architecture design. Complete test bed system will be implemented at selected facility(ies) Other institutes may implement system partially Re-evaluating and refining the system architecture. Improving implementation. Deploying the prototype system and demonstrating its functionality. 08/12/2011 K.Wrona, Harmonzation meeting

Relations to other projects/WPs Data archiving Compliance with PanData Scientific data policy Data access requires strong authentication service CRISP/WP16, PanDataODI/WP3 proposed AAI needs to provide appropriate level of security Data protection scheme should be consistent with equivalent solution for metadata CRISP/WP17 Data access Is moonshot project helpful here? 08/12/2011 K.Wrona, Harmonzation meeting