25th & 26th August 2009ICAT developer workshop 1

25th & 26th August 2009ICAT developer workshop 2 Overview Introduction Writing templates Real example Main usages Future

25th & 26th August 2009ICAT developer workshop 3 Introduction Why does we need a extracting tool ?  Because the analysis software the users use to process data are not yet able to read NeXus files What were the Requirements  A tool able to extract any kind of data from NeXus files and write it a arbitrary file format (in ASCII or binary mode)  Output description as simpliest as possible  Efficiency 2 possibilities  Extending a scripting language like Python  Developing a specific tool that process template files describing the output Why choosing the second way ?  The templates have to be as short as possible while keeping a good readability: they must be understood even by a scientist !  Fast to execute  Easy to deploy (very few dependencies, apart from NeXus and HDF5)

25th & 26th August 2009ICAT developer workshop 4 Introduction Capabilities  Extracting any kind of data  Writing one or more files (ASCII or binary, it's not a matter) from a single NeXus file  Directly writing 2D data as jpeg or bmp files (png format will be added soon)

25th & 26th August 2009ICAT developer workshop 5 Writing a template In a template you describe the extraction in a “WYSIWYM”-like way (What You See Is What You Mean)  Just write that you want to get  The result is written on the console  Defining a output file No keywords, only symbols ! Actually, it's not a language just a way to describe a output format

25th & 26th August 2009ICAT developer workshop 6 Writing a template Looping over a collection  The NXentry groups Testing values, attributes or dataset presence Defining data block  Not directly written on output Defining and using variables  Recursive evaluation of content

25th & 26th August 2009ICAT developer workshop 7 Writing a template Looping over values  index is in range [0, array_size[ Inserting sub-template  The sub-template inherit variables and data blocks of its parent

25th & 26th August 2009ICAT developer workshop 8 Real example A script used on the SWING beamline used to extract images data from Nexus file and output them as EDF (ESRF Data format) files. Characteristics of the output format  The file is made of 2 parts: a ASCII header and a binary image data  ASCII Header length must be a power of 2  a lot of cabbalistics fields must be filled  binary data must be 2 bytes integer Input NeXus files contains several image, each image in a dedicated NXdata group and named image#i with i in range [0, image_count[ (I known, a image stack would be a better way for storing data in that case)

25th & 26th August 2009ICAT developer workshop 9 Main usages at Soleil During the data acquisition process a extraction is (most of the time) applied automatically for each NXentry group or for the whole NeXus file (on 7 out of 11 beamlines) Our data portal application allows users to download NeXus files as well as on the fly extractions using nxextract on server side  Templates depends of the beamline

25th & 26th August 2009ICAT developer workshop 10 Future Adding PNG format support Defining standard template(s) for exporting data in CIF or imgCif files ? (but no interest for Soleil up to now) Embeding the Python engine ?  Allowing to define data block filled by Python script, like this: