1. Eiger-Related Development at GM/CA@APS
Mark Hilgart, Sergey Stepanov, Oleg Makarov, Qingping Xu, Nagarajan Venugopalan, Ruslan Sanishvili, Michael Becker, Craig Ogata, Robert F. Fischetti
March 15, 2017
High Data-Rate MX Meeting, Lund, Sweden
Image courtesy Argonne National Laboratory

2. GM/CA@APS Two MX beamlines
We develop our own data acquisition software
Received our Eiger in May 2016
Users started 8 weeks later
Requirements for integrating with JBluIce-EPICS:
Immediate image access
Counter to indicate when files are available on disk
Eiger 16M unpacking May 2016

3. Image Access Requirement
Immediate image access is needed for:
Viewing in JBluIce during collection
Auto-processing initial images to provide quick feedback
Raster, once a grid can be rastered with a single detector arm
Typical data collection is done at 5Hz with 0.2 degrees per frame
Waiting until collection completes is usually not acceptable to users

4. Converter Implementation
Uses FileWriter to write HDF5 files with one image per data file
Server monitors the output directory
eiger2cbf is run on each data file
Provides immediate access for viewing and processing
Converts at 20Hz for real data on the EPU
In use at 23ID-B for ~6 months of user time
JBluIce viewer
Dectris FileWriter
HDF in RAM
CBF on disk
Processing pipeline
HDF on disk

5. Goals for the Next Version
Simplify the server since we have very limited debugging time
Write CBF at 100+ Hz
Write SWMR HDF at 100+ Hz
Hopefully it will be supported by auto-processing software
But if not, it could still be useful for archiving
Maintain compatibility
Retain immediate image access and an on-disk image counter
Improve HDF5 storage
Reduce the master file size
Use a single data file per dataset

6. Streamwriter Overview
CBF Writer
CBF
ZeroMQ
Reader
HDF Writer
Eiger 16M
Reads ZeroMQ and writes CBF and/or HDF from the EPU to BeeGFS storage
CBFs are written to be identical to eiger2cbf-converted images
EPICS is used only to read filename info and write the on-disk image counter
Written in C++

7. SeriesHeader or Image object Multi-part ZeroMQ Message
ZeroMQ Reader
SeriesHeader or Image object
Multi-part ZeroMQ Message
Writer
Eiger DCU
zmq::message_t
zmq::message_t
zmq::message_t …
One reader thread
Queueing is done on the reader side
Images aren’t sent until a thread is available in the writer

8. Decompress bslz4 2) Compress CBF
CBF Writer
200 Threads
SeriesHeader
WriteTask:
Decompress bslz4 2) Compress CBF
3) Write to BeeGFS
CBF
Image

9. CBF Writer Compatibility
Uses CBFlib
eiger2cbf-compatible
StreamWriter output and FileWriter+eiger2cbf output should be identical
We apply the pixel mask the same way
StreamWriter transfers the same header values that eiger2cbf does
countrate_correction_count_cutoff is read from the HTTP interface
All others are available in the stream
Uses the same CBF_BYTE_OFFSET compression

10. Directly write bslz4 data from the stream
SWMR HDF5 Writer
SeriesHeader
Directly write bslz4 data from the stream
SWMR HDF5
Image
Single thread
SWMR requires a single-threaded writer
By not manipulating the image data we hope to achieve high write rates

11. SWMR HDF5 Write Efficiency
Writing HDF5 should be very efficient since:
Image data is not manipulated
HDF5 allows bypassing normal transformations
H5DOwrite_chunk skips:
Hyperslab selection
Datatype conversion
Filters

12. Benchmarks are run on the Dectris-supplied EPU
Development Progress
Test
Result
Blank images to CBF
100+ Hz
Lysozyme collection to CBF
75-100Hz, not yet optimized
Blank images from simulated stream to SWMR HDF
Write implemented but not benchmarked
Benchmarks are run on the Dectris-supplied EPU

13. Development Plan Month(s) Activity March-April Development May Testing
June
Release to users