1. 22/04/2018
SOLEIL PROGRESS IN PROVIDING REMOTE DATA ANALYSIS SERVICES Majid OUNSY: Data Analysis Software Project Leader

2. SOLEIL PROGRESS IN PROVIDING REMOTE DATA ANALYSIS SERVICES
The AutoBIOSAXS project news
SOLEIL strategy for upcoming months

3. BioSaxs DaaS architecture : reminder

4. AutoBioSaxs workflow automation: reminder
I0, l IT
Detected : I(Q) 2θ
2D detector
Ln ( I )
Q=4π sinq / l
Radial average +
buffer subtraction
(Foxtrot application)
SWING
SAXS - WAXS - GISAXS
(ATSAS suite of programs)
Data Acquisition
Data Reduction
Data Analysis

5. Technologies used for this project
 IspyB (ESRF)
Remote Experiment Declaration
Sample Tracking
Remote Experiment Results Visualization
Based on Java Web Development Standards
Collaborative Work with ESRF
Passerelle EDM (Isencia)
Data Reduction and Analysis Automation Engine
Based on Java Web Development Standards
Easy integration with IspyB through remote Web services
Parallel Workflow Execution
Multiple Data Source Handling with CDMA

6. AutoBioSaxs: Project Status
22/04/2018
Analysis workflows completely implemented
Both for HPLC and SampleChanger experiments
IspyB Webservices completely integrated in workflows
Added error handling for IspyB clients in HPLC WebServices
Service opened to users
Not yet permanently (temporary infrastructure)
Standard projects only (Waiting for SOLEIL official rules governing BAG projects)

7. AutoBioSaxs: Next steps
22/04/2018
Towards a continuous online service
Deployment on the new HPC platform (SUMO): still in installation
Training of support groups before integrating the service in on call perimeter
Offline delivery of the service
First: providing current workflows in interactive replay mode
Second: Extending the interactive mode to other data analysis workflows

8. SOLEIL strategy for upcoming months
22/04/2018
Provide an official data analysis service to its users
Define a process (see next slide)
Make up a dedicated team
Find collaborations
Enhance efforts on high throughput experiments data handling in the frame of collaborations around two main topics
Tomography
Ptychography

9. Process
Preparation of a questionnaire by a small team (1 scientist & 2 IT p.)
Review and Tests of the questionnaire, by 5 other p. (4 scientists & 2 IT p.)  it allowed to clarify and/or simplify some questions
Presentation/explanation of the questionnaire at 2 meetings adding some recommendations : complete the questionnaire collegially, etc.
Questionnaire sent to all BL scientists and staff of support groups (detectors, optics, laboratories, …) : beginning of September
In a second step, meeting with each BL team / group:  to clarify their answers and verify our good understanding
Overall restitution
Discussions on this basis with Experiment Division, SOLEIL Direction, … by associating Users’ Representatives  to prepare an action plan
Here we are
The process is made up of two phases:
-Questionnaire elaboration
-Questionnaire exploration
Elaboration :
prepared with a small team
Tested by scientists and It representatives inside and outside SOLEIL
Presented at 2 meetings for las recommandations
Second phase started in September:
Beamlines interviews
Overall restitution
Results analysis to prepare an action plan

10. Questionnaire content
For each type of experiment realized or considered :
Some general questions : a short description of the experiment principle, current and future experiment duration and occurrence, …
Make a status and identify projects, needs and expectations of scientists:
Preprocessing and reduction of raw data : formatting, qualifying and helping to decide what to do next  producing reduced data
Processing of raw and/or reduced data : analyzing data  producing publishable results
Level of automation for preprocessing and processing workflow
Support to optimize analysis codes and software, algorithmic support
Collect users’ feedbacks to SOLEIL scientists about data recovery, data curation, data analysis
Evaluate the current and future data production, compression, level of confidentiality required, …
Questionnaire organized towards the topics
General : expriment focus, acquisition principale
Reduction phase : which kind of treatments, data volume
Analysis phase : which….
Automation : is there any need
Code optimazation : Need for help
User’s feedback

11. First results End of November:
29 completed questionnaires, coming from 16 BLs+3 others:
describing 31 types of experiments / measurements techniques
We already met scientists of 6 BLs to clarify their answers and to verify our good understanding
The results that follow are still preliminary :
Only half of SOLEIL BLs answered for the moment
Sometimes, the answers have to be amended after discussion, due to a misunderstanding of some questions
However, that should already be fairly representative as the 4 beta testers + 6 BLs we already met are from different scientific domains.
15 lignes completed answers (Half of beamlines)
Spectroscopy, XAS, Diffraction, …
28 answers
4 BLs met
Preliminary : some imprecise answers
Fairly representative

12. Conclusion already done for 6 BLs
The need for support dedicated to data reduction / analysis is already confirmed by the first results
Next steps:
Services to provide: which are needed / which are desirable
Axes on which to focus efforts of a SOLEIL structure dedicated to "Data Reduction/Analysis"
Potential collaborations
etc.
already done for 6 BLs