A multispectral image enhancement approach to visualize tissue structures Pinky A. Bautista 1, Tokiya Abe 1, Yukako Yagi 1, John Gilbertson 1, Masahiro Yamaguchi 2, and Nagaaki Ohyama 2 1 Massachusetts General Hospital 2 Technology Institute of Technology

Multispectral Imaging (MSI)  Originally developed for space-based imaging  Multiple grey-level images are captured at different wavelengths  Allows extraction of additional information which the human eye fails to capture. Filter sensitivities 3 grey-level images N>3 grey-level images RGB imagingMutispectral imaging MSI allows greater flexibility for image analysis as compared to RGB imaging 3 broadband filters N Narrowband filters

Objectives  To digitally enhance an H&E stained multispectral image such that collagen fiber can easily be differentiated from the rest of the eosin stained tissue components.  Show the capability of multispectral imaging to differentiate tissue structures with minute colorimetric difference.

Multispectral Microscope Imaging system  Olympus BX-62 optical microscope controlled by a PC  16 interference filters  2kx2k pixel CCD camera *Used in the experiment

Enhancement Method Reference: Masanori Mitsui, Yuri Murakami, Takashi Obi, et.al, “ Color Enhancement in Multispectral Image Using the Karhunen- Loeve Transform,” Optical Review Vol.12, no.2, pp.60-75, 2005 Original spectral transmittance at location x,y (16-band) Enhanced version estimated spectral transmittance using M (M<N) KL vectors derived from the transmittance data of the selected tissue components Spectral residual error W NxN weighting factor Matrix, i.e. N=16 Controls the color of enhanced areas

Experiment 1. Training Phase 2. Testing Phase  Collection of 16-band transmittance spectra samples of the identified H&E stained tissue components  Derivation of the KL vectors  Identification of the appropriate number of KL vectors, i.e. M-KL vectors  Perform multispectral enhancement on 16-band images using the M-KL vectors derived in the training phase  Transform the multispectral enhanced image into its equivalent RGB format for visualization Examine the spectral residual error characteristics of the different tissue components

Derivation of KL vectors KL vectors were derived from the transmittance of these tissue components Transmittance spectra of the different tissue components Training data fiber Subject for enhancement Not Subject for enhancement RGB format of the 16-band MS image of a Heart tissue Nucleus Cytoplasm RBCs, etc.

Tissue components transmittance spectra Each tissue component is represented with 200 transmittance spectra samples. structures found in white areas

Spectral Residual Error The spectral residual error for fiber peaks at band 8 structures found in white areas Appropriate number of KL vectors was investigated…… 5-KL vectors were found to produce distinct peaks on the spectral residual error of collagen fiber

Result (heart tissue) H&E stained Digitally enhanced Striated muscle and Collagen fiber which are both stained with Eosin in an H&E stained slide are impressed with different shades of color when digitally enhanced Collagen fiber Striated muscle Striated Muscle Collagen fiber  2kx2k pixels  20x magnification

Results H&E stained MT stained Digitally enhanced Serial Section reference Tissue areas highlighted in the digitally enhanced image correspond to areas emphasized by the MT stain

Result (Magnified) not clearly differentiated differentiated Original H&E stained Enhanced H&E stained image MT stained Tissue structures with minute color difference is differentiated using Multispectral information reference

RGB and Multispectral Enhanced using RGB information Enhanced using Multispectral information Original H&E stained image MT stained image Serial Section

RGB and Multispectral Enhanced using RGB information Enhanced using Multispectral information Original H&E stained image Not clearly differentiatedClearly differentiated

Spectral transmittance There is a slight difference in the spectral configurations between the labeled fiber1 and fiber2 areas

Conclusion  With multispectral imaging it is possible to differentiate tissue structures with minute colorimetric difference  The current enhancement scheme makes it possible to differentiate tissue structures that are less likely differentiated with RGB imaging Future work  Work with more tissue images to validate the current result  Investigate further the meaning of spectral residual error in relation to tissue differentiation  Investigate possible application of the residual error configurations to select important bands to classify/segment specific tissue structures

We thank CAP foundation for making it possible for us to attend this conference. THANK YOU….

Weighting matrix Variation H&E stained Digitally enhanced Color of target areas can be varied by manipulating the weighting matrix W Spectral enhancement

Results H&E stained MT stained Digitally enhanced Serial Section reference Tissue areas highlighted in the digitally enhanced image correspond to areas emphasized by the MT stain

Result (kidney tissue) Training data were extracted from another MS image of a kidney tissue; training and test images belong to the same slide H&E stainedDigitally enhanced

Result (kidney tissue) H&E stainedDigitally enhancedMT stained