Mirjam van Daalen, (Stephan Egli, Derek Feichtinger) :: Paul Scherrer Institut Status Report PSI PaNDaaS2 meeting Grenoble 6 – 7 July 2016

Current projects at PSI Data Analysis Service Data Policy Remote access Metadata catalogue Petabyte archive Remote data transfer PSI, PSI, 20. September 201820. September 2018 20. September 2018

Covering larger parts of the life cycle

Project Overview: Data Analysis Service SUK Project 142-004 Project Manager: Dr. Stephan Egli, Dr. Derek Feichtinger, Paul Scherrer Institut Partner: ETHZ Project Duration: 04.2015 - 04.2017 Financing Support (50% matching funds): CHF 1'618'000 Team members: 16 (including 3 new positions financed by project) Workpackages WP1: Common Tools and Services (4) WP2: Data Analysis Environments for major use cases (3) WP3: Identity Management, DUO, Authentication and Authorization (3) WP4: Integration and development of scientific analysis codes (2) WP5: Procurement, installation, operation of analysis cluster infrastructure (4) WP6: Infrastructure sharing with other institutions (3) WP7: Project Management (2) Numbers in parentheses refer to the number of involved project members

DaaS Project Status Main purpose: provide an integrated solution for all SLS Users to do offline data analysis for data taken at SLS (and later SwissFel) Cluster of moderate size (~900 Cores, 2 PB Storage) Hired 3 persons dedicated to this project . Currently about 50% of the foreseen hardware installed and in operation Now in test phase with internal users Adjusting the system and software according to concrete use cases of these users. So far very good feedback Next phase: add external users Planning for Storage upgrade up to a total of about 3 PB until mid 2017 Option for extending the cluster also with “dedicated” resources (for paying customers), but within the same infrastructure and using centrally provided hardware choices

Data Policy Status Data Policy based on PaNdata framework Draft existing Embargo period 3 years, with easy extension to 5 years Should be adopted by PSI directorate in August 2016 Implementation from the end of August 2016

Remote Access Usecases: online and offline analysis, remote measurements, shift operation,sharing of sessions for support tasks , Sharing of sessions for collaboration Support for 3D Hardware Acceleration Access to the beamlines and to the DaaS Cluster through a common gateway Architecture based on separation of “server” and “node” processes of the Nomachine Software Version 5 Added graphical management tool to define (time based) access to beamline resources and offline compute cluster, with role based delegated management to resource responsible

Data Catalogue Currently comparing two possible approaches: ICAT/TOPCAT combination Approach based on NoSQL document databases (MongoDB), taking advantage of recent developments for middleware (Strongloop/Loopback) and component based graphical user interfaces (Angular2) Compare approaches concerning Ease of data ingestion Potential to integrate into existing IT infrastructures and storage/archive systems Flexibility to cope with multitude and growing requirements coming from different facilities, research groups, beamline instruments and experimental method in the area of data queries, data display and data analysis

Interactive and Batch data Analysis Support for doing interactive (e.g. Matlab) data analysis on the cluster, nodes can be reserved for interactive work. Standard Batch processing based on Slurm

Petabyte Archive PSI must prepare for the archiving of high amounts of data being expected for SLS and SwissFEL over the next decades. Strategic collaboration of PSI with the Swiss National Supercomputing Center (CSCS) in Lugano for building a Petabyte Tape Archive solution at CSCS Project initiated by a PoC within the DaaS project Volume increase driven by detector and instrumentation advances. Planning to leverage IBM Spectrum Scale (GPFS) AFM technology for the asynchronous data transfers between the sites. Data growth for high data volume beamlines in 2016 Estimated projections for yearly data production

Remote Data Transfer Support for rsync/scp and gridftp (Globus Online) Also evaluated Aspera solution from IBM. Could be added, but only if (paying) customer would request for it The integration with the longterm archive will create additional requirements

Possible Collaborations Remote Access Lossy Compression . TOMCAT group at SLS is developing solutions here. Should be driven from scientific community. Umbrella federated identity management.

Wir schaffen Wissen – heute für morgen My thanks go to Stephan Egli Derek Feichtinger Gerd Mann