1. SPIE Medical Imaging 2004 Reflectance and fluorescence hyperspectral
elastic image registration
Holger Lange*, Ross Baker,
Johan Håkansson, Ulf Gustafsson
Science and Technology International® (STI),
733 Bishop Street, Suite 3100
Honolulu, HI 96813, USA * 1

2. Overview Application Hyperspectral Imaging
Hyperspectral Registration Problem
Application-Specific Problems
Methodology
Validate Elastic Image Registration Algorithm
Embed Reflectance Image in Fluorescence HSI
Preliminary Results
Future Work
Conclusion 2

3. Application Science & Technology International® (STI) Medical Systems
develops a new hyperspectral medical imaging modality
for the early detection of uterine cervical cancer:
HyperSpectral Diagnostic Imaging (HSDI®) cervical instrument
Science and Technology International® (STI) is developing a new medical imaging modality, HyperSpectral Diagnostic Imaging (HSDI®) for the early detection of uterine cervical, colorectal, dermatological, esophageal and oral cancers. Hyperspectral refers to the instruments’ ability to collect many times more color information than a standard RGB (Red, Green, Blue color space) digital camera. This allows for discrimination between spectral features not normally available to a physician. STI’s current third generation HSDI cervical instrument is targeting uterine cervical cancer. The HSDI cervical instrument is developed to provide a cost effective and superior alternative to the subjective, time-consuming, and unpleasant colposcopic examination.
The HSDI cervical instrument acquires HyperSpectral Imagery (HSI) of tissue fluorescence and white-light tissue reflectance as well as RGB imagery. The tissue fluorescence is induced by non-ionizing, long-wavelength ultraviolet (UV) excitation. The RGB imagery is similar to that obtained from digital or video colposcopy. In a colposcopic examination an acetic acid treatment is applied to the cervix. The acquired data sets, shown in Fig. 1, consist of a pre-reflectance-scan RGB image, reflectance HSI, pre-fluorescence-scan RGB image, and fluorescence HSI taken sequentially before (pre) and after (post) the acetic acid treatment.
RGB Image
Reflectance HSI
RGB Image
Fluorescence HSI
HSI - HyperSpectral Imagery 3

4. Hyperspectral Imaging
“red”
“green”
“blue”
In hyperspectral imaging an entire scene is being imaged in a large number of spectral bands 2, 14 and 17. The HSDI instrument utilizes a technique called push-broom scanning. This technique is illustrated in Fig. 2. The hyperspectral sensor uses a progressive line scan to capture an entire image. For each scan line, the full spectrum for every spatial location (pixel) is provided. By taking a series of scan lines, a hyperspectral data cube is obtained. This hyperspectral data cube contains spatial information (pixels) in two dimensions and spectral information (bands) in the third dimension.
Hyperspectral
push-broom scanning 4

5. Hyperspectral Registration Problem
Computer-Aided-Diagnostic (CAD)
based on data fusion requires data registration
Problem: Reflectance and Fluorescence HSI Registration
Resemblance?
A Computer-Aided-Diagnostic (CAD) system is being developed using advanced computer algorithms to help the physician with the diagnosis of pre-cancerous and cancerous tissue regions. The CAD system will utilize the fusion of multiple data sources and algorithms to optimize its performance. A key enabling technology for the data fusion is the image registration of the different data sources. The difficult problem is the image registration of fluorescence and reflectance HSI data.
Using a push-broom scanning technology, the hypersprectral bands are already spatially aligned to each other and only the two spatial dimensions of the hyperspectral imagery needs to be registered. The two spatial dimensions can be represented by a 2D image from a single band or a 2D image calculated from any number of bands.
The registration of two images involves the matching of features present in both images, and from their spatial relationships, the calculation of the image transformation between them. The fluorescence and reflectance data by their nature exhibit different features. The ambiguity in multi-sensor image registration, due to the different features present in the images, typically only allows one to do image registration using an affine (rotation, translation and scale) image transformation.
The registration of two images involves the matching of features present in both images, analyzing the spatial relationships among these features, and then calculating the image transformation between them. Within this broad area of research, medical image registration has emerged as a particular active field 3, 4, 9, 10, 11, 13, 15, and 16. This activity is due in part to the many clinical applications including diagnosis, longitudinal studies, and surgical planning, and also due to the need for registration across different imaging modalities (e.g., MRI, CT, PET, X-RAY, etc.). Medical image registration, however, still presents many challenges. Several notable difficulties are (1) the transformation between images can vary widely and be highly nonlinear in nature; (2) the transformation between images acquired from different modalities may differ significantly in overall appearance and resolution; and (3) each imaging modality introduces its own unique challenges, making it difficult to develop a single generic registration algorithm. Those are the very same difficulties we face with the HSDI cervical instrument. 5

6. Application-Specific Problems
due to time interval between data acquisitions
to avoid problem reduce time:
- interleave refl. & fluor. scan lines
- multispectral
Soft Tissue
Movement
requires Elastic Image Registration
Glint
Blood
Mucous
In case of a sequential data acquisition, the patient and in particular the cervix is most likely to have moved between the fluorescence and reflectance HSI data acquisition. This soft tissue movement cannot be described with an affine image transformation and requires a more general elastic image transformation.
The registration problem can be avoided by reducing the time between the acquisition of the reflectance and fluorescence data and thereby reducing the magnitude of any possible movement. Using a push-broom scanning technology, this can be achieved by interleaving the acquisition of reflectance and fluorescence line scans 12. For each scan-line the fluorescence and reflectance hyperspectral data is acquired before moving to the next scan-line. The disadvantages to this method are an overall increased acquisition time and design complexity due to mechanical shutters and moving filters that need to be controlled for each line scan. The use of multi spectral imaging technology, instead of hyperspectral imaging technology, would also shorten the time between reflectance and fluorescence data acquisitions. The registration problem could also be avoided by using reflectance and fluorescence HSI data only when no movement has occurred. As we are able to register RGB images, RGB images taken before and after the reflectance and fluorescence HSI data acquisitions could be used to detect movements. Should movement occur, the data acquisitions need to be repeated, which makes this approach quite cumbersome and impractical for a product realization.
In our application, all images are taken at different moments in time. Therefore, the image registration algorithm must deal with the following problems, illustrated in Fig. 4: (1) Glint, (2) Mucous, (3) Blood and (4) Soft tissue movement. 6

7. Methodology Resolve Resemblance Problem
Deal w/ Application-Specific Problems
(2) Embed reflectance image
in fluorescence HSI
(3 Methods)
Validate (w/ RGB images)
elastic image registration
algorithm
Embedding a reflectance image in the fluorescence HSI data enables the image registration of fluorescence and reflectance HSI data to use an elastic image transformation. Both data sets then have a reflectance image showing exactly the same features. This helps resolve the ambiguity of the image registration, and the data sets can be registered by taking into account soft tissue movements.
The whole image registration process can be broken down into four steps, as shown in Fig. 3: (1) data acquisition, (2) feature image extraction, (3) pre-processing and (4) image registration. The data acquisition is about the physical process of acquiring the raw data with the sensor. This includes the design and use of the light source and the sensor. The feature image extraction takes the raw data from the sensor and generates an image that contains the common features in the scene that will be used for the registration process. The pre-processing transforms the representation of the common features to a common representation with similar characteristics. The image registration can then resolve the correspondence between the features (similar characteristics) in both images and from their spatial relationships calculate the transformation between the images.
The idea behind the registration of reflectance and fluorescence HSI data is to use a monochromatic reflectance image as the feature image. Reflectance feature images can easily be obtained from the RGB images and the reflectance HSI data. Without loss of generality, we can validate the capabilities of a chosen elastic image registration algorithm to deal with the application-specific problems by using monochromatic reflectance feature images extracted from the RGB images taken prior to the reflectance and fluorescent scans. The details are presented in the next paragraph. Once a suitable elastic image registration algorithm is developed that works well with monochromatic reflectance feature images, the next step involves identifying a means to embed a reflectance image into the fluorescence HSI data. This is primarily a question of defining a new data acquisition system for the fluorescence spectroscopy. Several methods of how to embed a reflectance image into the fluorescence HSI data are described in the second paragraph.
(3) Preliminary Results 7

8. Validate Elastic Image Registration Algorithm
Pre-
Reflectance
Pre-
Fluorescence
Deal w/ Application-Specific Problems
Data Acquisition: Refl. RGB Refl. HSI Fluor. HSI
RGB Images
Feature Image Extraction: GREEN Channel
Pre-Processing: (1) detect and eliminate GLINT
(2) reduce noise w/ Gauss filter
(3) mutual histogram equalization
Pre-Processed
Feature Images
We adopted the elastic image registration algorithm from Jan Kybic 7. He has demonstrated the performance of his algorithm under what we consider analogous conditions, the registration of Echo Planar Imaging (EPI) and Magnetic Resonance Imaging (MRI) images 8.
We evaluated the suitability of the algorithm for our application with data sets from a human subject clinical trial using STI’s second generation HSDI cervical instrument 1, 5 and 6. The focus of the evaluation was on algorithm performance in the presents of: (1) glint, (2) mucous, (3) blood and (4) soft tissue movement when registering pre-processed monochromatic reflectance feature images.
The data acquisition process uses an RGB camera to provide raw RGB images from a cervix illuminated with white light. RGB images are taken prior to the reflectance and fluorescence scans. The feature image extraction uses the green channel of the RGB image as the feature image.
The impact of the pre-processing on the overall performance needs to be emphasized. For the evaluation we used a simple, rather classic, pre-processing sequence: (1) detect and eliminate glint, (2) reduce noise with a gauss filter and (3) adjust the values in one feature image using a histogram equalization based on the histogram of the other feature image. Fig. 5 shows the pre-processed feature images for two RGB images to be registered. To illustrate the mismatch between the two images before image registration, the strongest contours of one feature image are overlaid on the other feature image. The glint is currently eliminated as part of the pre-processing, which still creates undesired artifacts. Polarization filters would eliminate the glint in the scene and thereby any problems associated with it.
Adopted Elastic Image Registration Algorithm from Jan Kybic
Elastic Image Registration using Parametric Deformation Models;
Ph.D. Thesis, Number 2439, EPFL, Lausanne, Switzerland, 2001
Elastic Image Transformation 8

9. Validate Elastic Image Registration Algorithm
overlay B contours on: A B A A A B B
The image registration algorithm performed well with the application-specific problems present. Fig. 6 shows the results of the elastic image registration. The elastic image transformation is visualized with a deformed grid and one feature image is warped using the image transformation to the spatial locations of the other feature image. To illustrate the performance of the image registration, the strongest contours of one feature image are overlaid on the other warped feature image.
Elastic Image
Transformation
A warped on B
Visualize Match 9

10. Embed reflectance image in fluorescence HSI
Resolve Resemblance Problem
embed
Three (3) methods:
Narrow-band light embedded in UV light
(2) Second-order diffracted UV light
(3) UV light reflectance
Reflectance
Fluorescence
Reflectance and fluorescence are two different modes of light interaction with matter. Reflectance is an elastic interaction process, which means that there is no change in energy of the incident and the emitted light, while fluorescence is an inelastic interaction process, which results in the emission of light with an energy different from that of the incident light. In reflectance mode, the tissue is illuminated using a broadband white light source in the visible light spectrum, and detects the reflected intensity in the same spectral region. In fluorescence mode, the tissue is excited with narrowband Ultraviolet (UV) light and collects fluorescence in the visible spectral region.
Three methods are described by which a reflectance image can be embedded into the fluorescence HSI data:
(1) Light from a narrow-band source embedded in Ultraviolet (UV) excitation light,
(2) Second-order diffracted Ultraviolet (UV) light, and
(3) Ultraviolet (UV) excitation light reflectance.
One challenge is to balance the fluorescence and the reflectance intensities. If the intensities are not balanced, the reflectance image will either be too bright (saturated) or too dark (noisy) making it impossible to accurately register the fluorescence and reflectance HSI data. The balancing of the fluorescence and the reflectance intensities can be addressed with the use of proper filters.
Examination of reflectance and corresponding fluorescence images in the same wavelength bands, illustrates the difficulties image registration algorithms are confronted with when trying to register pure reflectance and fluorescence HSI data, namely soft tissue movement and lack of resemblance between reflectance and fluorescence imagery. Fig. 10 shows a reflectance and the corresponding fluorescence image from the same “blue” light wavelength band. At first glance, it may appear that regions, either dark or bright, in the reflectance image correspond to either bright or dark regions in the fluorescence image. In principle, this problem can be resolved with a pre-processing that calculates the absolute gradient (region contours) of those images and thereby allows the image registration algorithm to use the presence of features contours rather than the feature region characteristics for the matching process. Unfortunately, upon examination, it can be seen that different features and feature boundaries are imaged in the reflectance and fluorescent images. This means that the feature contours cannot readily be used for the image registration. It seems that a robust registration of pure reflectance and fluorescence HSI data is very difficult to achieve. 10

11. (1) Narrow-band light embedded in UV light
Fig. 7 shows the block diagram of a hyperspectral imaging device that embeds a reflectance image into the fluorescence HSI by using light from a narrow-band source embedded in UV excitation light. In approach 1, the tissue is illuminated by the UV light and a narrow band light source in the visible spectrum, preferable centered in the upper visible spectrum where almost no fluorescence intensity is present. Both the UV light and the light source illuminate the tissue during the hyperspectral scan and the hyperspectral imaging device collects both the fluorescence and the reflected light. The reflected light is seen as a narrow peak in the fluorescence spectrum. The light from the narrow-band source might induce fluorescence in the tissue as well but as fluorescence is an in-elastic scattering process, this fluorescence light will be emitted at even longer wavelengths.
see preliminary results 11

12. (2) Second-order diffracted UV light
m=2
Second-Order Diffraction
m=1
Reflectance
Incident Light
m=0
Diffraction
Grating
Approaches 2 and 3 both use UV excitation light to embed a reflectance image in the fluorescence HSI. The ultraviolet light will excite tissue fluorescence but the light will also be reflected.
Fig 8 shows a block diagram of a hyperspectral imaging device that embeds a reflectance image into the fluorescence HSI by using second-order diffracted light. The hyperspectral imaging device provides spectral information through diffraction. Every diffractive optical element share a common feature in that it will diffract light in what is called different orders, first, second, third, etc. This means that light of the same wavelength will be diffracted in different angles depending on the order. It also means, due to the regular profile of a diffractive optical element, that the second-order light with wavelength l and the first-order light with wavelength 2l will be diffracted into the same angle. This phenomenon which is often an undesired consequence can be exploited here to embed the second-order reflected UV light in the longer wavelength region of the fluorescence HSI. As almost no fluorescence intensity is present at in this wavelength region, the information contained in the fluorescence data will not be compromised. 12

13. (3) UV light reflectance
Fig. 9 shows a block diagram of a hyperspectral imaging devices that embeds a reflectance image into the fluorescence HSI by using UV excitation light reflectance. In approach 3, the reflected UV light is collected in a band centered around its excitation wavelength. The fluorescence information will not be compromised as no fluorescence light is present at the same wavelength as the excitation light. 13

14. Preliminary Results Narrow-band light embedded in UV light details!
register
“Red”
details!
STI has performed a human subject clinical study using its second generation HSDI cervical instrument 1, 5 and 6. The second generation HSDI cervical instrument was designed to embed a reflectance image into the fluorescence HSI following the first approach. Near-infrared light was added to the UV excitation light, generating a reflectance image in the corresponding “red” light wavelength bands.
Looking at the hyperspectral reflectance images at different wavelength bands, as illustrated with the example of two bands in Fig. 11, reveals that the reflectance images at the lower wavelength bands, corresponding to “blue” and “green” light bands, exhibit more discriminating features (details) than the images at higher wavelength bands, corresponding to “red” light bands. This can be explained by the general “red” color characteristics of the cervical tissue. Discriminating features are important for a robust image registration; therefore wavelength bands in the “blue” and “green” light bands seem to be best suited for embedding a reflectance image. Unfortunately this is in the middle of the valuable fluorescence wavelength bands.
We next present an example of the reflectance and fluorescence HSI registration using light from a narrow-band source embedded in UV excitation light to embed a reflectance image in the fluorescence HSI (approach 1), shown in Fig. 12. It can be seen, as previously discussed, that the reflectance image in the “red” light bands, which were used to embed the reflectance image in the fluorescence HSI data, lack discriminating features (details) required for a robust registration process. This is a shortcoming of this approach and needs to be addressed. Nevertheless, by embedding the reflectance image into the HSI data we can now use an elastic image registration algorithm in an automated fashion for the registration of reflectance and fluorescence HSI data.
Reflectance
Fluorescence 14

15. Preliminary Results A B A A A B B Elastic Image Transformation
overlay B contours on: A B A A A B B
The monochromatic reflectance feature image and “embedded” reflectance feature image were extracted from the reflectance and fluorescence HSI by simply averaging the reflectance images in the “red” light bands that were used to embed the reflectance image. The pre-processing sequence used was the same as for the RGB imagery: (1) detect and eliminate glint, (2) reduce noise with a gauss filter and (3) adjust the values in one feature image using a histogram equalization based on the histogram of the other feature image.
The image registration algorithm performed well with the application-specific problems present. Fig. 6 shows the results of the elastic image registration. The elastic image transformation is visualized with a deformed grid and one feature image is warped using the image transformation to the spatial locations of the other feature image. To illustrate the performance of the image registration, the strongest contours of one feature image are overlaid on the other warped feature image.
Elastic Image
Transformation
A warped on B
Visualize Match 15

16. Future Work Improvements and Optimizations:
- Method (2) Second-order diffracted UV light
Current STI’s third generation HSDI cervical instrument
- Method (3) UV light reflectance
Lab evaluation
- Optimize Pre-Processing
- Optimize Elastic Image Registration
Further experiments are currently being performed using the third generation HSDI cervical instrument, which has been enabled to embed a reflectance image into the fluorescence HSI following the second approach, the use of second-order diffracted UV light.
The design of the third approach, using the UV excitation light reflectance, will be evaluated in a lab setting before being used in a human subject clinical trial.
The following design goals are used to guide the development of a suitable fluorescence spectroscopy system that embeds a reflectance image into the fluorescence HSI data:
Maximize the number of common discriminating features (details) in the reflectance images; considering the reflectance images in the “blue” and “green” light bands as sufficient,
Minimize the destruction and/or degradation of relevant fluorescence HSI data as a consequence of embedding a reflectance image; the definition of “relevant” fluorescence HSI data is determined by how the fluorescence HSI data is processed, and
Maximize the resemblance (same characteristics) of the common features in the reflectance and “embedded” reflectance feature images. 16

17. Conclusion Problem: Reflectance and Fluorescence HSI Registration
Solution: Embed Reflectance Image in Fluorescence HSI
Narrow-band light embedded in UV light
lack of detail > landmarks (“ink” markings)
(2) Second-order diffracted UV light
(3) UV light reflectance
Enables Elastic Image Registration
Further improvements and optimizations to come …
When using hyperspectal imaging for the detection of uterine cervical cancer, the registration of reflectance and fluorescence HyperSpectral Imagery (HSI) can be a difficult problem due to the occurrence of soft tissue movement and the limited resemblance of these types of imagery.
The proposed solution is based on the idea of embedding a reflectance image in the fluorescence HSI. Having a reflectance image in both data sets resolves the resemblance problem and thereby enables the use of elastic image registration algorithms required to compensate for soft tissue movements.
We have shown the suitability of an elastic image registration algorithm (adopted from Jan Kybic 7) to deal well with the application-specific problems using RGB images taken prior to the reflectance and fluorescence scans.
Three methods have been described by which a reflectance image can be embedded into the fluorescence HSI data: (1) Light from a narrow-band source embedded in Ultraviolet (UV) excitation light, (2) Second-order diffracted Ultraviolet (UV) light, and (3) Ultraviolet (UV) excitation light reflectance.
Science and Technology International®‘s (STI) second generation Hyperspectral Diagnostic Imaging (HSDI®) cervical instrument was designed to embed a reflectance image into the fluorescence HSI by adding near-infrared light to the UV excitation light, generating a reflectance image in the corresponding “red” light wavelength bands. Even so this method suffers from a limited number of discriminating features (details) in the “red” light bands of cervical imagery, we can now perform an elastic image registration of reflectance and fluorescence HSI data in an automatic fashion. The lack of detail can be compensated by adding landmarks in form of “ink” markings on the cervix.
Future work will examine the two other methods and optimizations to the pre-processing sequence and elastic image registration algorithm. 17