1. NSLS II High Data Rate Workshop May 2016
Eigers and Diamond
NSLS II High Data Rate Workshop May 2016

2. Overview Confession: we don’t have an Eiger
Infrastructure at DLS & user expectations
Current DLS beamlines & equipment
Challenges & benefits of moving from Pilatus to Eiger

3. Acknowledgements DIALS / xia2 development teams
Example data from other Eiger users
Diamond IT support teams
Diamond Scientific Software

4. We don’t have an Eiger, why are we here?
We’re buying Eiger 16M
We remember the “teething problems” with the early deployment of Pilatus
We want to be prepared...

5. Worries
Getting the data out of the detector into the file systems
Processing the data well
Processing the data quickly enough
Getting the data to the user
Controlling the detector (AreaDetector)
Reading the images (DIALS)
Non Worries

6. Infrastructure and user expectations

7. Realtime feedback during collection

8. 6 minute data set, 54s processing

9. 3600 image data set - images to density 2 minutes

10. DLS beamlines I02/VMXi - Pilatus2 6M @ 25 Hz
VMXm - TBC
Aggregate: > 300 frames/s > 1800 MB/s sustained rate - sample exchange < 20s
Worth noting that Eiger 16M at 133 frames/s will actually reduce file system load compared with 100 frame/s Pilatus 6M
Also worth noting just because the detectors can go this fast does not mean people do

11. High throughput MX at DLS 13:11:18 < 2m20s 13:09:01

12. Strategy clusters cluster 40x20 core “com12” cluster
3PB parallel file system

13. 4x10Gb/s
upgrade
4x10Gb/s
upgrade

14. processor : 19
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E GHz
stepping : 2
microcode : 54
cpu MHz :
cache size : KB
physical id : 1
siblings : 10
core id : 12
cpu cores : 10
apicid : 56
initial apicid : 56
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu apic sep mtrr pge tla mca cmov wtf pat lol pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall ....
bogomips :
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

15. Example set Eiger 9M @ Soleil 1800 @ 0.1 degrees / 200 Hz (9s)
Transthyretin
Excellent example set because we have the same data in CBF and HDF5 format
High resolution limit
Low resolution limit
Completeness
Multiplicity
I/sigma
Rmerge(I)
Rmeas(I)
Rpim(I)
CC half
Total observations
Total unique
Assuming spacegroup: P
Unit cell:

16. Timing Lustre cbf /hdf5 GPFS RamDisk hdf5 xds Spot Finding 74.5 82.6
70.4
95.8
67.9
79.8
40.2
204.8
Integration
245.3
240.3
262.5
255.4
242.0
235.3
96.6
234.7
xia2
11m50s
10m51s
12m43s
10m02s
10m12s
7m09s
15m04s

17. Conclusions
File systems not our biggest problem - processing from GPFS / Lustre comparable with ramdisk
HDF5 significantly but not substantially slower for spot finding, reverse for integration (using DIALS)
XDS very fast in ramdisk with CBF, much slower with HDF5

18. Challenges New container format:
Fast_dp powered by XDS won’t work natively, works very nicely on CBF images in 66s
User handling for large data sets ill defined - how will they cope with > 20TB per visit?
xia2 / DIALS does work though work to do to get it down to 9s ...
Should be better than 1500 images every 2 minutes per beamline
New detector technology:
Improved readout - effectively no dead time
High speed / low dose / high redundancy collection makes feedback even more important