Medical Image Registration Yujun Guo Dept.of CS Kent State University

Outline Why registration Registration basics Rigid registration Non-rigid registration Applications

Modalities in Medical Image Computed Tomography (CT), Magnetic Resonance (MR) imaging, Ultrasound, and X-ray give anatomic information. Positron Emission Tomography (PET) and Single Photon Emission CT (SPECT) give functional information.

Registration Monomodality: Multimodality: A series of same modality images (CT/CT, MR/MR, Mammogram pairs,…). Images may be acquired weeks or months apart; taken from different viewpoints. Aligning images in order to detect subtle changes in intensity or shape Multimodality: Complementary anatomic and functional information from multiple modalities can be obtained for the precise diagnosis and treatment. Examples:PET and SPECT (low resolution, functional information) need MR or CT (high resolution, anatomical information) to get structure information.

Registration Problem Definition q = (912,632) q = T(p;a) p = (825,856) Homologous pixel location in second image Pixel location in first image Pixel location mapping function

Example Mapping Function q = (912,632) p = (825,856) Pixel scaling and translation

Image Registration Define a transform T that will map one image onto another image of the same object such that some image quality criterion is maximized. A mapping between two images both spatially and with respect to intensity I2 = g (T(I1))

Registration Scheme

Components Feature Space Search Space or transformation Similarity Metric Search Strategy There are plenty of publications dedicated to medical image registration. Various techniques, models, algorithms and clinical applications are described and reviewed. The aspects of image registration may differ greatly among various approaches, but an underlying framework is common to all techniques. Registration methods can be viewed as different combinations of choices for the following four components.

Feature Space Geometric landmarks: Points Edges Contours Surfaces, etc. Intensities: Raw pixel values Feature space extracts the information in the images to be used for the matching. It can be a set of matching targets, such as… Feature-based Intensity-based 35 56

Image transformations Input image Output Rigid transformation Original shape Affine transformation Homogeneous coordinates Polynomial transformations Rigid Non-rigid

Similarity Metric Absolute difference SSD (Sum of Squared Difference) Correlation Coefficient Mutual Information / Normalized Mutual Information Similarity metric determines the relative merit for each test.

Search Strategy Powell’s direction set method Downhill simplex method Dynamic programming Relaxation matching Hierarchical techniques Search Strategy, or optimization algorithm, decides how to choose the next transformation from this space, to be tested in the search for the optimal transformation. According to the similarity measure. Powell and Downhill are used when local distortion is not present, and the global transformation can be put into one function. Dynamic programming is used to register images where a local transformation is needed. Relaxation matching is most often used when a global transformation is needed, but local distortion is present. Hierarchical techniques can speed up different approaches by guiding search strategy through progressively finer solutions.

Multi-modality Brain image registration Intensity-based 3D/3D Rigid transformation, DOF=6 (3 translations, 3 rotations) Maximization of Normalized Mutual Information Simplex Downhill Multi-resolution Dataset: Vanderbilt University http://www.vuse.vanderbilt.edu/~image/registration/results.html

Mutual Information as Similarity Measure Mutual information is applied to measure the statistic dependence between the image intensities of corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned.

Normalized Mutual Information Extension of Mutual Information Maes et. al.: Studholme et. Al.: Compensate for the sensitivity of MI to changes in image overlap

Geometry Transformation Image Coordinate transform: The features (dimension, voxel size, slice spacing, gantry tilt, orientation) of images, which are acquired from different modalities, are not the same. From voxel units (column, row, slice spacing) to millimeter units with its origin in the center of the image volume.

Target Image & Template Image Target Image Grid j i Template Image Grid j i y x Target Image Physical Coordinates y’ x’ Template Image Physical Coordinates Space Transform

Images from the same patient 256 x 256 pixels MRI-T2 128 x 128 pixels PET Target Image ? Template Image ? Images provided as part of the project: “Retrospective Image Registration Evaluation”, NIH, Project No. 8R01EB002124-03, Principal Investigator, J. Michael Fitzpatrick, Vanderbilt University, Nashville, TN.

Interpolation Nearest Neighbor Tri-linear Interpolation Partial-Volume Interpolation Higher order partial-volume interpolation

Evaluating similarity measure for each transformation x x Template Image Target Image

Optimization Powell’s Direction Set method Downhill Simplex method

Multi-resolution Why Multi-resolution Methods for detecting optimality can not guarantee that a global optimal value will be found. Time to evaluate the registration criterion is proportional to the number of voxels. The result at coarser level is used as the starting point for the finer level. Currently multi-resolution approaches: Sub-sampling Averaging Wavelet

Registration Result (I) A typical superposition of CT-MR images. Left : before registration Right: after registration.

Rigid transformation (II) A typical superposition of MR-PET images. Left : before registration Right: after registration.

Mammography Breast cancer is the second leading cause of death among women in USA. Detected in its early stage, breast cancer is most treatable. Mammography is the main tool for detection and diagnosis of breast malignances. It reduces breast cancer mortality by 25% to 30% for women in the 50 to 70 age group

Mammogram Registration Temporal/bilateral mammograms vary Breast compression Breast position Imaging Technique Change in Breast

Mammogram registration techniques Whole breast area vs. regional Nipple location Control-point location Rigid & non-rigid registration

Non-rigid Mammogram Registration Intensity-based Elastic transformation Multi-resolution Demons algorithm (Thirion, 1996)

Demons Transform Scene (Target) Model (Template)

Demons (Cont.) Transform Scene Forces Model

Demons (Cont.) Current Estimation Intensity Space Gradient Desired Displacement Scene

Demons From Optical Flow Scene: f, Model: g Assumption: The intensity of a moving object is constant with time (1) One model image: G One scene image: F Fit model to the scene One solution to the first equation is to consider that… Leads to equation 2 (g-f)^2 is added to make the equation stable when ||->0 (2)

Description of the Approach Select demon points. Compute the force u on the model at each of the selected demons Determine a global transformation based on the computed u and apply it to the model If the model images is now registered to the scene image, stop. Else, go to Step 2. The description of the demon-based approach is given here.

Registration Components Image Intensities Non-rigid transformation, one displacement vector for each pixel Bilinear interpolation Absolute difference as similarity metric Multi-resolution Dataset: MIAS,DDSM

Demons Results (I) Synthetic Images Template: image T Target: Image C The second experiment is designed to test the performance of the multi-resolution techniques. Level=2 Level=3 Level=5 Level=4

Demons Result (II) MIAS Original images Before registration After rigid registration After non-rigid registration

Ongoing registration topics Trade-off of computation and accuracy Evaluation of registration results Visualization of registration

Applications: Change Detection Images taken at different times Following registration, the differences between the images may be indicative of change Deciding if the change is really there may be quite difficult

Other Applications Multi-subject registration to develop organ variation atlases. Used as the basis for detecting abnormal variations Object recognition - alignment of object model instance and image of unknown object (segmentation)

References Maes F,Collignon A, et al. “Multimodality image registration by maximization of mutual information.” IEEE Trans. Med. Imaging. 1997, V16,pp187-198 L.G.Brown, “A survey of image registration techniques,” ACM Computing Surveys, vol. 24, no. 4, pp. 325–376, 1992. Jean-Philippe Thirion, “Non-Rigid Matching Using Demons,” IEEE Conference on Computer Vision and Pattern Recognition,1996