Eiger-Related Development at GM/CA@APS Mark Hilgart, Sergey Stepanov, Oleg Makarov, Qingping Xu, Nagarajan Venugopalan, Ruslan Sanishvili, Michael Becker, Craig Ogata, Robert F. Fischetti March 15, 2017 High Data-Rate MX Meeting, Lund, Sweden Image courtesy Argonne National Laboratory

GM/CA@APS Two MX beamlines We develop our own data acquisition software Received our Eiger in May 2016 Users started 8 weeks later Requirements for integrating with JBluIce-EPICS: Immediate image access Counter to indicate when files are available on disk Eiger 16M unpacking May 2016

Image Access Requirement Immediate image access is needed for: Viewing in JBluIce during collection Auto-processing initial images to provide quick feedback Raster, once a grid can be rastered with a single detector arm Typical data collection is done at 5Hz with 0.2 degrees per frame Waiting until collection completes is usually not acceptable to users

Converter Implementation Uses FileWriter to write HDF5 files with one image per data file Server monitors the output directory eiger2cbf is run on each data file Provides immediate access for viewing and processing Converts at 20Hz for real data on the EPU In use at 23ID-B for ~6 months of user time JBluIce viewer Dectris FileWriter HDF in RAM CBF on disk Processing pipeline HDF on disk

Goals for the Next Version Simplify the server since we have very limited debugging time Write CBF at 100+ Hz Write SWMR HDF at 100+ Hz Hopefully it will be supported by auto-processing software But if not, it could still be useful for archiving Maintain compatibility Retain immediate image access and an on-disk image counter Improve HDF5 storage Reduce the master file size Use a single data file per dataset

Streamwriter Overview CBF Writer CBF ZeroMQ Reader HDF Writer Eiger 16M Reads ZeroMQ and writes CBF and/or HDF from the EPU to BeeGFS storage CBFs are written to be identical to eiger2cbf-converted images EPICS is used only to read filename info and write the on-disk image counter Written in C++

SeriesHeader or Image object Multi-part ZeroMQ Message ZeroMQ Reader SeriesHeader or Image object Multi-part ZeroMQ Message Writer Eiger DCU zmq::message_t zmq::message_t zmq::message_t … One reader thread Queueing is done on the reader side Images aren’t sent until a thread is available in the writer

Decompress bslz4 2) Compress CBF CBF Writer 200 Threads SeriesHeader WriteTask: Decompress bslz4 2) Compress CBF 3) Write to BeeGFS CBF Image

CBF Writer Compatibility Uses CBFlib eiger2cbf-compatible StreamWriter output and FileWriter+eiger2cbf output should be identical We apply the pixel mask the same way StreamWriter transfers the same header values that eiger2cbf does countrate_correction_count_cutoff is read from the HTTP interface All others are available in the stream Uses the same CBF_BYTE_OFFSET compression

Directly write bslz4 data from the stream SWMR HDF5 Writer SeriesHeader Directly write bslz4 data from the stream SWMR HDF5 Image Single thread SWMR requires a single-threaded writer By not manipulating the image data we hope to achieve high write rates

SWMR HDF5 Write Efficiency Writing HDF5 should be very efficient since: Image data is not manipulated HDF5 allows bypassing normal transformations H5DOwrite_chunk skips: Hyperslab selection Datatype conversion Filters

Benchmarks are run on the Dectris-supplied EPU Development Progress Test Result Blank images to CBF 100+ Hz Lysozyme collection to CBF 75-100Hz, not yet optimized Blank images from simulated stream to SWMR HDF Write implemented but not benchmarked Benchmarks are run on the Dectris-supplied EPU

Development Plan Month(s) Activity March-April Development May Testing June Release to users

www.gmca.anl.gov