22/04/2018 SOLEIL PROGRESS IN PROVIDING REMOTE DATA ANALYSIS SERVICES Majid OUNSY: Data Analysis Software Project Leader

SOLEIL PROGRESS IN PROVIDING REMOTE DATA ANALYSIS SERVICES The AutoBIOSAXS project news SOLEIL strategy for upcoming months

BioSaxs DaaS architecture : reminder

AutoBioSaxs workflow automation: reminder I0, l IT Detected : I(Q) 2θ 2D detector Ln ( I ) Q=4π sinq / l Radial average + buffer subtraction (Foxtrot application) SWING SAXS - WAXS - GISAXS (ATSAS suite of programs) Data Acquisition Data Reduction Data Analysis

Technologies used for this project  IspyB (ESRF) Remote Experiment Declaration Sample Tracking Remote Experiment Results Visualization Based on Java Web Development Standards Collaborative Work with ESRF Passerelle EDM (Isencia) Data Reduction and Analysis Automation Engine Based on Java Web Development Standards Easy integration with IspyB through remote Web services Parallel Workflow Execution Multiple Data Source Handling with CDMA

AutoBioSaxs: Project Status 22/04/2018 Analysis workflows completely implemented Both for HPLC and SampleChanger experiments IspyB Webservices completely integrated in workflows Added error handling for IspyB clients in HPLC WebServices Service opened to users Not yet permanently (temporary infrastructure) Standard projects only (Waiting for SOLEIL official rules governing BAG projects)

AutoBioSaxs: Next steps 22/04/2018 Towards a continuous online service Deployment on the new HPC platform (SUMO): still in installation Training of support groups before integrating the service in on call perimeter Offline delivery of the service First: providing current workflows in interactive replay mode Second: Extending the interactive mode to other data analysis workflows

SOLEIL strategy for upcoming months 22/04/2018 Provide an official data analysis service to its users Define a process (see next slide) Make up a dedicated team Find collaborations Enhance efforts on high throughput experiments data handling in the frame of collaborations around two main topics Tomography Ptychography

Process Preparation of a questionnaire by a small team (1 scientist & 2 IT p.) Review and Tests of the questionnaire, by 5 other p. (4 scientists & 2 IT p.)  it allowed to clarify and/or simplify some questions Presentation/explanation of the questionnaire at 2 meetings adding some recommendations : complete the questionnaire collegially, etc. Questionnaire sent to all BL scientists and staff of support groups (detectors, optics, laboratories, …) : beginning of September In a second step, meeting with each BL team / group:  to clarify their answers and verify our good understanding Overall restitution Discussions on this basis with Experiment Division, SOLEIL Direction, … by associating Users’ Representatives  to prepare an action plan Here we are The process is made up of two phases: -Questionnaire elaboration -Questionnaire exploration Elaboration : prepared with a small team Tested by scientists and It representatives inside and outside SOLEIL Presented at 2 meetings for las recommandations Second phase started in September: Beamlines interviews Overall restitution Results analysis to prepare an action plan

Questionnaire content For each type of experiment realized or considered : Some general questions : a short description of the experiment principle, current and future experiment duration and occurrence, … Make a status and identify projects, needs and expectations of scientists: Preprocessing and reduction of raw data : formatting, qualifying and helping to decide what to do next  producing reduced data Processing of raw and/or reduced data : analyzing data  producing publishable results Level of automation for preprocessing and processing workflow Support to optimize analysis codes and software, algorithmic support Collect users’ feedbacks to SOLEIL scientists about data recovery, data curation, data analysis Evaluate the current and future data production, compression, level of confidentiality required, … Questionnaire organized towards the topics General : expriment focus, acquisition principale Reduction phase : which kind of treatments, data volume Analysis phase : which…. Automation : is there any need Code optimazation : Need for help User’s feedback

First results End of November: 29 completed questionnaires, coming from 16 BLs+3 others: describing 31 types of experiments / measurements techniques We already met scientists of 6 BLs to clarify their answers and to verify our good understanding The results that follow are still preliminary : Only half of SOLEIL BLs answered for the moment Sometimes, the answers have to be amended after discussion, due to a misunderstanding of some questions However, that should already be fairly representative as the 4 beta testers + 6 BLs we already met are from different scientific domains. 15 lignes completed answers (Half of beamlines) Spectroscopy, XAS, Diffraction, … 28 answers 4 BLs met Preliminary : some imprecise answers Fairly representative

Conclusion already done for 6 BLs The need for support dedicated to data reduction / analysis is already confirmed by the first results Next steps: Services to provide: which are needed / which are desirable Axes on which to focus efforts of a SOLEIL structure dedicated to "Data Reduction/Analysis" Potential collaborations etc. already done for 6 BLs