Diamond is all about data…

… some people love huge amounts of data… … new users can be less keen…

… but everyone really wants this. Science! Some variable / Unit Some metric / Unit

OPERATES A BEAMLINE: GDA DOESN’T: DAWN Client server Technology, Communication with EPICS and hardware, Scan mechanism, Requires Beamline. www.opengda.org Java Maths, File Loading, Visualisation, Analysis Tools, Jython and Python. Calibration, Processing, Data Conversion, You Can Take It Home. www.dawnsci.org Acquisition Analysis

Powder Diffraction / Small Angle Scattering

Calibration

Masking, Correction, Reduction, Fitting…

[61, 76, 1679, 1475]

Requirements Integrate tightly with GDA Near infinite number of processing steps Process series images or XY data – linear chain NeXus (n-dimensional datasets – maintain rank) Data provenance Simple.

(Part of Pipeline Runner) Architecture 41k peak fits Highly Modular (Plugins + Services) Pipeline Runner, Processing Steps, File Writing All independent and swappable Data Iterator (Part of Pipeline Runner) Processing Step 1 Processing Step 2 Processing Step 3 File Writer Pipeline Runner

(Part of Pipeline Runner) Runner – Iterates through frames Series, Parallel, Passarelle Workflow, Dynamic (SWMR) Processing steps – Maths + Parameters separate Store id and parameters – Provenance Can be written in DAWN! File Writing – NeXus, when in UI, writes to Screen 41k peak fits Data Iterator (Part of Pipeline Runner) Processing Step 1 Processing Step 2 Processing Step 3 File Writer Pipeline Runner

GDA/Cluster Plot tool for quick processing live (not saved) Headless runner for auto-processing

HDF5-SWMR Read HDF5 files while they are being written! Simultaneously better and looser GDA integration Only need to write SWMR Pipeline Runner Processing steps don’t need to change Raw Processed

Generic Live Processing DAWN processing runs on cluster during data collection “We have our best people working on it...”

Run Processing was pressed > 5000 times in Diamond the last year

Feedback Button “Treat every bug report like a love letter.”

For more information: scientificsoftware@diamond.ac.uk Acknowledgements Beamlines (and scientists therein): I15, I12, I22, I11-1, I15-1, I18, I07 Data Analysis Group @ Diamond Data Acquisition Group @ Diamond For more information: scientificsoftware@diamond.ac.uk www.dawnsci.org www.youtube.com/DAWNScience www.twitter.com/dawnscience J. Synchrotron Rad. (2015). 22, 853-858