Cleaning up the mess

Scalars Name Dynamic Part Data Time stamp Alarm MetaData Min / Max 0, 450.0 Data Type Double Units Microns

N-D Array Name Data [] /* Dynamic Part */ Time stamp Alarm Min / Max /* value MetaData */ Data Type Units Data Size Number of Dimensions Per Dimension /* Per Axis MetaData */ Size of Dimension Min / Max (offset and extent of this axis) Axis label Axis Representation (interval, Array, (pattern?)) Interval Array of values for each point on this axis or the interval value

N-D Array - Data Portion Data [] /* Dynamic Part - this needs to be a reference too!! */ Time stamp Alarm Min / Max /* value MetaData */ Data Type /* this is the type for all of the samples in the array */ Units Data Size Number of Dimensions

N-D Array – Axis Information Number of Dimensions Per Dimension /* Per Axis MetaData */ Size of Dimension /* how many values in this dimension Min / Max /* what is the origin and maximum */ /* this can be used for tiling */ /* or comparing results from different detectors *. Axis label /.* need to know the units for operations */ Axis Representation (interval, Array, (pattern?)) Interval /* describes the size of this pixel or cell */ Array of values for each point on this axis or the interval value

1D Det., Raster Sample 1-D Detector: 6 10 eV in X The detector is not offset – origin point is 0,0 Sample that is scanned in 2 dimensions: 300 microns x 100 microns 1 step = 10 micron motion in x and y DCM: energy set at 200 eV to 1000 eV ; 2 eV resolution

Scalars X Position Data /* Dynamic Part */ Time stamp Alarm Min / Max 0, 450.0 /* MetaData */ Data Type Double Units Microns Y Position Min / Max 0, 360.0 /* MetaData */ Energy Min / Max 0, 1000.0 /* MetaData */ Units eV

1-D Array – Interval Axis 1D-Detector Data [6] Timestamp Alarm Min / Max 0, 2400 Data Type U16 Units Photons Data Size 6 x Sizeof(UInt) Number of Dimensions 1 Axis information [0] Size of Dimension 6 Min / Max 0, 240 (offset and extent of this axis) Axis label eV Axis representation Interval Interval 10 AxisArray[] NULL

Energy Scan for each sample spot Set Energy to appropriate power ( (for each 50 micron row of the sample) For (x=0, x<=100; x+= 50) (for each 100 micron column) For (y=0; y <= 300; y+= 100)

Instrument Data Y X energy detector frame (pixel) 0 0 200 [6] 0 (4 Y positions) 100 0 200 [6] 1 200 0 200 [6] 2 300 0 200 [6] 3 0 50 200 [6] 4 100 50 200 [6] 5 200 50 200 [6] 6 300 50 200 [6] 7 0 100 200 [6] 8 (3 X positions) 100 100 200 [6] 9 200 100 200 [6] 10 300 100 200 [6] 11

Instrument Data Requests Each Pixel Where Energy Matches Some 1D Pattern Y X energy detector frame 0 50 200 [6] 4 100 50 200 [6] 5 0 100 200 [6] 8 100 100 200 [6] 9

Export Detector Scan as an N-DArray 1D-Detector Data [] Min / Max 0, 2400 Data Type U16 Units Photons Data Size 6 x 12 x sizeof(U16) Number of Dimensions 2 Axis information [0] Size of Dimension 6 Axis Data type U16 (Type of all of the samples) Axis Units Photon Counts Axis representation Interval Interval 1 AxisArray[] NULL Axis information [1] Size of Dimension 12 Min / Max 0, 11 (offset and extent of this axis) Axis Data Type U16 Axis Units Frame AxisArray[] Null

Export Detector Scan – Not 4D but: 1D + Scalar + Scalar (all over time) 1DDetector Data Size 6 x 12 x sizeof(U16 Min / Max 0, 2400 Data Type U16 Units Photons Number of Dimensions 2 Axis information [0] Size of Dimension 6 Min / Max 0, 2400 Axis Data type U16 Axis Units Photon Counts Axis representation Interval Interval 1 Axis information [1] Size of Dimension 12 Min / Max 0, 11 Axis Data Type U16 Axis Units Frame AxisArray[] Null X Position Data Size 12 x sizeof(Double) Min / Max 0, 100.0 Data Type Double Units Microns Number of Dimensions 1 Axis Information [0] Size of Dimension 12 Min/Max 0,11 Axis Dtype U16 Axis Label Frame Axis Represntation Interval Interval 1 AxisArray NULL Y Position Min / Max 0, 300.0

Start cleaning from here……. 2-D Detector: 6x6 pickups 75 micron spacing in X 60 micron spacing in Y 450 micron by 360 micron in size The detector is not offset – origin point is 0,0 Sample that is scanned in 2 dimensions: 300 microns x 100 microns 1 step = 10 micron motion in x and y DCM: energy scans 200 eV to 1000 eV ; 2 eV resolution

Scalars X Position Data /* Dynamic Part */ Time stamp Alarm Min / Max 0, 450.0 /* MetaData */ Data Type Double Units Microns Y Position Min / Max 0, 360.0 /* MetaData */ Energy Min / Max 0, 1000.0 /* MetaData */ Units eV

1-D Array – Interval Axis 1D-Detector Data [6] Timestamp Alarm Min / Max 0, 2400 Data Type U16 Units Photons Data Size 6 x Sizeof(UInt) Number of Dimensions 1 Axis information [0] Size of Dimension 6 Min / Max 0, 240 (offset and extent of this axis) Axis label eV Axis representation Interval Interval 10 AxisArray[] NULL

2-D Array – not an interval 2D Detector Data Size 36 x Sizeof(Data Type) Data Type U16 Units Photons Display Range 0,2400 Number of Dimensions 2 Axis information [0] Size of Dimension 6 Min / Max 0, 450 (offset and extent of this axis) Axis label Microns Axis representation Array Interval .75 AxisArray[] 0.0, .75, 1.50, (gap) 3.50, 4.25, 5.0 Axis information [1] Min / Max 0, 360 Interval .60 AxisArray[] 0.0, .60, 1.20, (gap) 1.50, 2.10, 2.70

Energy Scan for each sample spot (for each 50 micron row of the sample) For (x=0, x<=100; x+= 50) (for each 100 micron column) For (y=0; y <= 300; y+= 100) (energy scan) For (energy = 200; energy <= 1000; energy += 200)

Data Sets Y X energy detector frame 0 0 200 [6x6] 0 (5 energies) 300 0 1000 [6x6] 20 (4 y positions) ………………………….. 300 100 200 [6x6] 56 (3 x positions) 300 100 …. [6x6] …. 300 100 1000 [6x6] 60

Data Set Requests 1 spot of the sample at every energy Y X energy detector frame 0 100 200 [6x6] 6 0 100 400 [6x6] 7 0 100 600 [6x6] 8 0 100 800 [6x6] 9 0 100 1000 [6x6] 10

Data Set Requests All spots at one energy Y X energy detector frame 0 0 200 [6x6] 0 (1 energy) 100 0 200 [6x6] 6 200 0 200 [6x6] 11 300 0 200 [6x6] 16 (4 y positions) 0 50 200 [6x6] 21 ……………………….. 300 100 200 [6x6] 56 (3 x positions)

Export Detector Scan as an N-DArray Data Size [6 x 6 x 60] Data Type U16 Units Photons Display Range 0,2400 Nujmber of Dimensions 3 Axis information [0] Size of Dimension 6 Min / Max 0, 450 (offset and extent of this axis) Axis label Microns Axis representation Array Interval .75 AxisArray[] 0.0, .75, 1.50, (gap) 3.50, 4.25, 5.0 Axis information [1] Min / Max 0, 360 Interval .60 AxisArray[] 0.0, .60, 1.20, (gap) 1.50, 2.10, 2.70 Axis information [2] Size of Dimension 60 Min / Max 0, 11 Axis label Frame Axis representation Interval Interval 1 AxisArray[] Null

Export Detector Scan as an N-DArray Data Size [6 x 6 x 60] Data Type U16 Units Photons Display Range 0,2400 Nujmber of Dimensions 3 Axis information [0] Size of Dimension 6 Min / Max 0, 450 (offset and extent of this axis) Axis label Microns Axis representation Array Interval .75 AxisArray[] 0.0, .75, 1.50, (gap) 3.50, 4.25, 5.0 Axis information [1] Min / Max 0, 360 Interval .60 AxisArray[] 0.0, .60, 1.20, (gap) 1.50, 2.10, 2.70 Axis information [2] Size of Dimension 60 Min / Max 0, 11 Axis label Frame Axis representation Interval Interval 1 AxisArray[] Null

Detector Scan with Energy = 3D+1D Data Size [6 x 6 x 60] Data Type U16 Units Photons Display Range 0,2400 Nujmber of Dimensions 3 Axis information [0] Size of Dimension 6 Min / Max 0, 450 (offset and extent of this axis) Axis label Microns Axis representation Array Interval .75 AxisArray[] 0.0, .75, 1.50, (gap) 3.50, 4.25, 5.0 Axis information [1] Min / Max 0, 360 Interval .60 AxisArray[] 0.0, .60, 1.20, (gap) 1.50, 2.10, 2.70 Axis information [2] Size of Dimension 60 Min / Max 0, 11 Axis label Frame Axis representation Interval Interval 1 AxisArray[] Null Data Size 60 Data Type Double Units Energy Display Range 200,1000 Number of Dimensions 1 Azxis [0] Size of Dimension 60 Axis Label Frame Axis Representation Interval AxisArray Null

Other issues Operations done on data Non-measured Metadata Provenance

Common Operations Flat field subtraction Binning Peak detection ….. Others?

Other Related Variables What about the description of the locations of the sample holder and detector to understand things relative to each other? Should we store them in the archiver as variables with values and metadata? Distance Size Angle What about compiling the positions of the sample into one large array? The 2D data sample gets bigger – do this example.

Tracking Pedigree Processed data sets Any data set can have a number of attributes/processing The processing has source Some action have meta data This is an nearly exhaustive set of libraries Operations – metadata for a data set Read source Flat field subtraction flat field data set Integrate/exposure time time Compression Region of Interest Binning (what if any of this is done at input time?)