Knowledge Systems Lab JN 8/10/2002 Fusion of Multi-Modality Volumetric Medical Imagery Mario Aguilar and Joshua R. New Knowledge Systems Laboratory MCIS Department Jacksonville State University Jacksonville, AL 36265

Knowledge Systems Lab JN 8/10/2002 Outline What is the neurophysiological motivation of the fusion architecture? How is fusion extended to a three- dimensional operator? How is this operator applied to volumetric data sets? What are the results?

Knowledge Systems Lab JN 8/10/2002 Neurophysiologically Motivated Architecture

Knowledge Systems Lab JN 8/10/2002 Fusion Architecture

Knowledge Systems Lab JN 8/10/2002 2D Image Fusion Example GADPDSPECTT2 Color Fuse Result

2D Image Fusion Animation

Knowledge Systems Lab JN 8/10/2002 Extensions to 3D 2D Fusion3D Fusion Where: A – decay rate B – maximum activation level (set to 1) D – minimum activation level (set to 1) I C – excitatory input I S – lateral inhibitory input C and G s are as follows: Where: A – decay rate B – maximum activation level (set to 1) D – minimum activation level (set to 1) I C – excitatory input I S – lateral inhibitory input C, G c and G s are as follows: + - 2D Shunt Operator Symbol3D Shunt Operator Symbol

Knowledge Systems Lab JN 8/10/2002 3D Fusion Architecture T1 Images T2 Images R G B Color Remap Color Fuse Result Noise cleaning & registration if needed Contrast Enhancement Between-band Fusion and Decorrelation

Knowledge Systems Lab JN 8/10/2002 3D Shunt Results Original 2D Shunt 3D Shunt

Knowledge Systems Lab JN 8/10/2002 3D Image Fusion Example GADPDSPECTT2

Knowledge Systems Lab JN 8/10/2002 3D Fusion Results 3D Explorer Views – 3D vs. 2D shunt

Knowledge Systems Lab JN 8/10/2002 Conclusions Development of the 3D shunting operator is a natural extension of the 2D operator Application of the 3D shunting operator provides better definition of image details in volumetric data sets 3D fusion extensions developed in the context of Med-LIFE, a visualization and pattern recognition tool.

ABSTRACT Ongoing efforts at our laboratory have targeted the development of techniques for fusing medical imagery of various modalities (i.e. MRI, CT, PET, SPECT, etc.) into single image products. Past results have demonstrated the potential for user performance improvements and workload reduction. While these are positive results, a need exists to address the three-dimensional nature of most medical image data sets. In particular, image fusion of three-dimensional imagery (e.g. MRI slices) must account for information content not only within the given slice but also across adjacent slices. In this paper, we describe extensions made to our 2D image fusion system that utilize 3D convolution kernels to determine locally relevant fusion parameters. Representative examples are presented for fusion of MRI and SPECT imagery. We also present these examples in the context of a GUI platform under development aimed at improving user-computer interaction for exploration and mining of medical data.

Neurophysiologically Motivated Architecture Retinal Circuitry We can model the neural interactions in the retina by using the shunting neural network, which is based on the following equation: Fusion Architecture System is based on the fusion of color wavelengths in human and primate retinal circuits. When applied to one band, it performs contrast enhancement and adaptive normalization. When applied to more than one band, it performs decorrelation and information fusion. 2D Shunt Operator: + -

FUSION SYSTEM EXTENSION 2D Fusion3D Fusion Shunting Neural Network Equation: Where: A – decay rate B – maximum activation level (set to 1) D – minimum activation level (set to 1) I C – excitatory input I S – lateral inhibitory input C and G s are as follows: Shunting Neural Network Equation: Where: A – decay rate B – maximum activation level (set to 1) D – minimum activation level (set to 1) I C – excitatory input I S – lateral inhibitory input C, G c and G s are as follows: + - 2D Shunt Operator Symbol3D Shunt Operator Symbol

Results Original 2D Shunt 3D Shunt

Knowledge Systems Lab JN 8/10/2002 Acknowledgements This work was supported by a Faculty Research Grant awarded to the first author by the faculty research committee and Jacksonville State University. Opinions, interpretations, and conclusions are those of the authors and not necessarily endorsed by the committee or Jacksonville State University. For additional information, please visit