1. WP5 Report: Virtual Laboratories
Final Review, Brussels, November 24th 2014
Final Review

2. Introduction WP5 Objectives:
Demonstrate PaN-data ODI services and JRA results using 3 different techniques
VL1: Structural ‘joint refinement’ against X-ray and neutron powder diffraction data
VL2: Simultaneous analysis of SAXS and SANS data for large scale structures
VL3: Access to tomography data exemplified through paleontological samples
Became part of the Science3D project, Frank Schlünzens WP2 report.
Final Review

3. Introduction VL1, VL2 tasks
An authenticated user searches data in a catalogue.
Access is authorized and data downloaded from facility archive.
Search relevant software in a database.
Download software and run it locally or at the facility.
Analyze data
Upload results and any relevant reduced data to facility archive.
Issues:
ICAT deployment: only ISIS and DLS have fully operational instances
Populated with restricted access experiment data
NeXus integration: configuration, experiment description
Final Review

4. Revised WP5 DoW D5.3 contents:
Survey of the implementation of PaN-data services
Demonstration and evaluation of ICAT services
File discovery, using VL1 and VL2 data
Data download
Inspection of metadata and data
Involving scientists
Science3D – the tomography database
WP2, Frank Schlünzen
NeXus integration
NeXus library
Configuration procedure
Implementation at other PaN-data facilities
Comparison of data processing frameworks
Dawn, DPDAK, Mantid
Use cases from VL1 and VL2
VL1, VL2 tasks
An authenticated user searches data in a catalogue.
Access is authorized and data downloaded from facility archive.
Search relevant software in a database.
Download software and run it locally or at the facility.
Analyze data
Upload results and any relevant reduced data to facility archive.
Review report 2013: Deliverable D5.2 (…) has been approved on condition of completion of VL1 deployment before the end of the projects duration..
Final Review

5. Data Policy
Data Policy
PaN-Data Type
ALBA
In Progress
Yes
CEA/LBB
DESY No
DLS
ELETTRA
ESRF
HZB
Draft
In Discussion
ILL
ISIS
JCNS
MAX IV
PSI
SOLEIL
PaN-data type preferred (open access, facility acts a custodian, embargo period, optional extension of the embargo period)
DESY: PI has the full responsibility of the data, DESY offers to store the data at the expense of the PIs institute.
Final Review

6. The Common Identity Management System Umbrella
Operational
Test Instance
Planned
ALBA No
CEA/LBB
DESY
Yes
DLS
ELETTRA
ESRF
HZB
In progress
ILL
ISIS
JCNS
MAX IV
PSI
SOLEIL
Soon
So far Umbrella is used for the digital user offices only. ICAT integration is planned at some facilities.
Final Review

7. The Standard Data Format NeXus
In Production
Full Desc.
Mand. MD
Respect ADs
Autom. Ingest
ALBA
Yes No
CEA/LBB
DESY
Yes,In progress
DLS
Yes/In progress
ELETTRA
No, HDF5
ESRF
HZB
ILL
ISIS
JCNS
In progress
MAX IV
PSI-SINQ
PSI-SLS
SOLEIL
Final Review

8. The Metadata Catalogue ICAT4
IDS: Integrated Data Server, enables downloads
In Prod.
Test Inst.
Restr. Data
Authent.
Author.
IDS
ALBA No
Yes
CEA/LBB
DESY
DLS
ELETTRA
ESRF
HZB
ILL
ISIS
JCNS
MAX IV
PSI
SOLEIL
ISIS: 12 M entries, size 40 TB, DOIs, Umbrella before the end of PaN-data
DLS: 500 M entries, size 2 PB, DOIs planned
ESRF: catalogue for ID16A-NI, will be populates as soon as data taking commences
ILL: production state will be reached soon, incl. Umbrella, old data will also be ingested
Final Review

9. Demonstration and Evaluation of ICAT Services
D5.3 contains 3 examples
Powder diffraction (VL1) at DLS
Beamtime in Jan. 2013, 5 days, 347 datasets
Small angle scattering (VL2) at DLS
Beamtime in Dec. 2013, 6 days, 1438 datasets
Small angle neutron scattering at ISIS
Open access data
Final Review

10. DLS ICAT: Investigations
Login to icat.diamond.ac.uk/TOPCATWeb.jsp with fedid
Final Review

11. DLS ICAT Search for an investigation using start date/end date
Final Review

12. DLS ICAT: Investigation Summary
Final Review

13. DLS ICAT: Dataset/Datafile Window
To download data:
Scan number is taken from the log-book
Final Review

14. Demonstration and Evaluation of ICAT services
Summary
ICAT entries are organized in terms of investigations, datasets and files.
TOPCAT offers powerful tools to perform searches.
Metadata are sufficient for files discovery.
The response times for searches are short.
Download speed DLS to DESY: 15 MB/s
Datasets are identified by the scan number, is retrieved from the log-book. Scientists would appreciate, if user-supplied comments were displayed
Downloading entire investigations from SR catalogues problematic
So far no Umbrella support
Final Review

15. The NeXus Integration D5.3
Status of the NeXus library, performance tests
External filters
NeXus integration at DESY
NeXus integration at other PaN-data institutes
Questions
How is the NeXus file production integrated into the data acquisition process?
How is the experiment described?
Which metadata are automatically generated?
NeXus files self-contained?
Applications definitions respected?
Which metadata are used for ingestion?
→ Detailed answers in D5.3
Final Review

16. The NeXus Integration Summary
NeXus configurations based on device selection are integrated into the local experiment control software. Each facility, except 2, has its own control software.
Configuration: GUIs, CLIs, XML file, scripts, dictionaries, static rules
NeXus files can be self-contained or can have external links.
Application definitions are respected by a subset of the experiment stations.
Metadata are gathered from various sources including digital user office databases.
The institutes selected (more or less) the same mandatory metadata, at least for catalogue ingestion.
Mandatory metadata: the subset of metadata created all instruments of a facility.
Final Review

17. Comparison of Data Processing Frameworks
Goal: find an integrated solution for
Visualization of raw and derived data
Online monitoring
Interactive analysis
Scripting, workflows
Batch processing
Candidates from PaN-data institutes:
Dawn,
DLS, ESRF, Eclipse, Java, Passerelle
DPDAK, dpdak.desy.de
DESY, MPG, Python
Mantid,
ISIS, SNS, C++
D5.3 contains 2 test cases, each implemented in Dawn, DPDAK, Mantid
VL1: Online application supporting investigations of disordered or amorphous materials using reduced pair distribution functions (PDF)
VL2: GISAXS analysis, investigation of growth of gold layers
Final Review

18. PDF Online Application
Analysis flow for monitoring the evolution of PDF properties:
The sample was headed at a constant rate.
The raw images contain intensity values as a function of the pixel index.
Using geometry information and the photon wavelength, the images are integrated to a diffractogram, i.e. an intensity distribution as a function of the momentum transfer 𝑞 in reciprocal space. Detector position and orientation are obtained by a powder diffraction calibration procedure using e.g. the Fit2D program or the pyFAI scripts. The integration properly takes into account a mask of unusable image pixels.
A background image containing the scattering pattern of the complete experimental setup excluding the data sample, and the effect of a fluorescence pattern are subtracted and an absorption correction is applied before the intensity distribution is normalised to obtain the coherent scattering intensity 𝐼 𝑒𝑢 𝑐𝑜ℎ (𝑞) in electron units.
Using the atomic composition of the sample and the tabulated atomic scattering factor, the total structure factor 𝑆(𝑞) is derived.
The reduced PDF 𝐺(𝑟) is then obtained by a sine Fourier transformation (using a finite integration range) that converts from reciprocal to real space. The 𝐺(𝑟) distribution oscillates around zero with an amplitude that gets smaller for larger atomic distances 𝑟.
The average distances, aka shells, of the first, second, third, etc. nearest neighbours can be approximated by the centres of gravity of the peaks above zero.
The shell positions are then monitored with time (and thus with temperature) for each incoming picture.
Final Review

19. PDF Online Application with Dawn
Implementation via the Python interface
Part of the Python code could be shared between Dawn, DPDAK and Mantid.
The tool was in operation at P02/PetraIII
Final Review

20. Comparison of Data Processing Frameworks: Conclusion
Dawn and Mantid are fully-featured frameworks supporting a large community, substantially supported by their home institutes
Caveat: the comparison has been done on the basis of two use cases
DAWN
Analyse and visualise arbitrary SR data, many different file formats, generic multi-dimensional browsing, slicing, etc.
Highlight: HDF5 viewer
Based on Eclipse: Perspectives, views, standardized extensions, etc., needs considerable expertise
Jython vs. Python
Final Review

21. Comparison of Data Processing Frameworks: Conclusion
DPDAK
Processes series of images obtained from SR scattering experiments.
A light-weight plug-in system, simple interfaces.
Has an internal database for a detailed inspection or for an export to files, possible scalability problem.
Limited set of data types passed between the plug-ins (scalars, 1D arrays, file paths)
Mantid
For neutron scattering and muon spectroscopy, data processing and visualisation tools, creates 2D or 3D instrument views of the detectors.
Workspaces are the central concept, hold data that are processed by algorithms, applied algorithms are tracked.
Errors can be supplied, enable a proper statistical data analysis.
Highlight: interactive function fitting, allows define position, height and width of peaks using the mouse, the only investigated program excepting constraints.
Final Review