ITKv4 Image Interpretation Layer

ITK - IO ImageIOBase VectorImage NumberOfComponents IOComponentType IOPixelType ConvertPixelBuffer Nested Switchs Based on Input Number Of Components Vs Output Number Of Components Buffer Image

Pixel and Component Types ● UNKNOWNPIXELTYPE ● SCALAR ● RGB ● RGBA ● OFFSET ● VECTOR ● POINT ● COVARIANTVECTOR ● SYMMETRICSECONDR ANKTENSOR ● DIFFUSIONTENSOR3D ● COMPLEX ● FIXEDARRAY ● MATRIX ● UNKNOWNCOMPON ENTTYPE ● UCHAR ● CHAR ● USHORT ● SHORT ● UINT ● INT ● ULONG ● LONG ● FLOAT ● DOUBLE

ITK - IO ImageIOBaseImage NumberOfComponents IOComponentType IOPixelType Interpretation Layer User Provided Rules Data Layout Components TimePoint Dimensions Resolutions Buffer

ITK - IO ImageIOBase NumberOfComponents IOComponentType IOPixelType User Provided Rules Data Layout Components TimePoint Dimensions Resolutions Array of Images Interpretation Layer Potentially: Rearrange the pixel data when reading it from IO, and before storing it in the interpretation layer

ITK - IO ImageIOBase NumberOfComponents IOComponentType IOPixelType User Provided Rules Data Layout Components TimePoint Dimensions Resolutions Array of Images Programmable at Run Time Interpretation Layer Default Rules could come from the file format too (when the data layout is specified).

What Conversions and When... ● What conversions should be made in the IO ? ● Some interpretation should be made in the IO object, to make it efficient. ● Leverage existing code. ● What conversions should be made after IO ? ● Separate what we need to know at: ● Compile time ● Run time

What Portion of Data To Read... ● Support stream ● Provide an API to select subsets (sub-regions) ● Specific resolutions (or set of resolutions) ● Subset of channels

ITK - IO Array of Images Delivers Images On Demand Query Methods Components TimePoint Dimensions Resolutions Image Container Image ● Component K ● TimePoint P ● Region R ● Resolution L itk::ImageSet

ITK - IO Array of Images Abstract Service Image ● Component K ● TimePoint P ● Region R ● Resolution L

ITK - IO Array of Images Abstract Service Image ● Component K ● TimePoint P ● Region R ● Resolution L DICOM Server

ITK - IO Array of Images Abstract Service Image ● Component K ● TimePoint P ● Region R ● Resolution L Image Database

ITK - IO Array of Images Abstract Service Image ● Component K ● TimePoint P ● Region R ● Resolution L Web Service

ITK - IO Array of Images Abstract Service Image ● Component K ● TimePoint P ● Region R ● Resolution L Interprocess Communication (Slicer 3D / MRML-Id-IO)

Suggested API – Image Interpretation Image Interpretation Abstract Service ● Query ● What is in these files / direc ● Get native data layout ● Describe Data Layout ● To reinterpret the data ● Set(“attribute”,int dimension) ● Data Extraction ● Give me a subset of the images → Stored in an ImageSet

Suggested API – Image Set Image Set ● Data container ● In memory ● Has a collection of heterogeneous ● e.g. different resolution – Different image size ● But same pixel type ● Different number of channels – With the same component type

Suggested API – Image Set Image Set ● Query methods ● Ask what is inside ● Is it redundant ? ● Extract methods ● Give me image X and put it in a Image ● The extraction could produce an ImageSet ● Must support multi-resolution ● High performance for data access ImageSet char,int,..., double

Suggested API – Set Set ● SpatialObjects ?

Image Set – Filter Manager Image Set Image Set Filter Manager Filter X

Image Set – Data Layout Reorganizer Image Set Image Set Filter Reorganizer Do the Filter Reorganizer has to be template-less ?

Image Set – Example Scenario Imag e Set Filter Reorganizer Imag e Set Filter Manager Filter X Pipeline Manager ?

Suggested API – Image Time Series Image Video With ring buffer inside ● Move across time ● Notion of streaming ● Across time ● Large overlaps ● Pixel type can be known at compilation time (maybe a template parameter)

Microscopy Cases ● 2D Images ( about 2K x 2K ) ● Eight Dimensions ● 3D – Z (100~200) (may be uneven) ● Channels (3~5) ● Time Points (0~2000) (may be uneven) ● Tiled in Space (x,y,z) (with overlap) ( ~5 each dim) ● Row, Columns (embryos in a plate) (~16 x ~6) ● Plates (~100)

Remote Sensing Case ● 2D Images ( about 2K x 2K → 40K x 40 K) ● May be a non-rectilinear grid ● 2.5 D Image (elevation) ● Bands (~4 → 250 (spectral)) (uneven sampling of spectrum) ● SAR (synthetic aperture radar) ● Complex images ● Horiz/Vertical polarization ● Up to 4 bands each one with complex values ● Multi-Resolutions ● 1km resolution in a band, and 50m resolution in other. ● Time Points (uneven) ● Heterogeneous Size (of overlapping regions) ● Heterogeneous Modalities

Video Cases ● 2D Frames ● Evenly space in time ● Grayscale image ● Multiple channels (RGB,HSV) ● Uneven space in time ● 3D + t ● 3D Ultrasound