MPHY8149: Image Registration David Sterling, MS, DABR Fall 2018

What is Image Registration? Sometimes called Image Fusion, Image Registration is the overlaying of different imaging studies on one another Secondary datasets are used to superimpose specific helpful information onto the primary dataset.

What is Image Registration? Planar x-ray CT PET SPECT MRI Ultrasound Useful Patient Data!

What is Image Registration Imaging modalities provide different information Anatomic Metabolic Functionality Uptake of tracer Motion Positioning Accuracy Combining this data is much more powerful than one scan on its own Can be many for a single patient

Example: PET scan Patient given radiotracer (e.g. FDG) Patient receives CT scan Patient receives PET scan Patient receives CT Scan for two reasons

Example: PET Reason 1: Attenuation Correction https://www.aapm.org/meetings/08SS/documents/Kinahan.pdf

Example: PET scan Reason 2: Anatomical Information

Image Registration Even images of the same modality can provide information when fused MRI T1 vs. T2 With and without contrast Image acquisition settings Timing Over time to check for changes Before and after surgery

How is it used clinically? Example 1: PET we already reviewed Misalignment or miscalibration in CT/PET/Couch sometime means that CT and PET do not line up well and must be manually adjusted

Fusion

Image Registration Categories Manual Purely automatic Feature-based image registration Intensity-based image registration Semi-manual Purely automatic followed by manual tweaking

Automatic Image Registration Techniques Feature-based image registration Point-based matching (usually 2D images). Contour-based matching (based on manual or automatic segmentation). Chamfer (edge) matching, using edge detection software. point-based contour-based point-based

y C x z y x z A B y x w z

Feature-based image registration: Contour-based matching Matches one contour to another (in 2D or 3D), so as to minimize the “difference” between them. (For example, difference could be the sum of square of distances from each point on one contour to the nearest point on the other contour). Procedure is more laborious (hence slower) than point matching, since it requires human interaction for contour drawing. Can be made less human-dependent by using automatic segmentation algorithms. However, automatic segmentation can be very error prone!

Feature-based image registration: Contour-based matching Scenario: Patient is planned with CT image. Contours of interest are drawn on planning CT image. Images are acquired prior to each treatment (using either cone-beam computed tomography or CT-on-rails). Contours are drawn on these images. Contour-to-contour matching is performed between the planning CT and pre-treatment CT. Patient is shifted based on the contour matching, and then treated. bony matching or “soft tissue” (prostate) matching (based on your protocol). Bony matching is more common.

SIEMENS CT-ON-RAILS VARIAN CBCT SYSTEM

Feature-based image registration: Contour-based matching Planning image = bony matching = soft tissue matching

Feature-based image registration: Chamfer-based matching Use edge detection software to extract only the ridges/edges in the image. Match the edges/ridges to each other. Decreases the complexity from a full image to just a set of features.

PET CT Misalignment Examples https://www.researchgate.net/figure/51400698_fig1_Figure-1-Examples-for-mis-registration-in-different-regions-of-whole-body-PETCT-scans

PET CT Misalignment If these errors are present when PET and CT are taken at the same time, the errors can obviously be much greater when taken at different times

What to do? We can adjust the registration to ensure that the area we are concerned about lines up well Sometimes, this requires multiple fusions for a single pair of datasets (e.g., H&N patient with PET) Sometimes “eyeballing it” on different screens is actually better than a bad fusion Deformable registration Not in the scope of this talk (though I can take questions at the end), but don’t believe the hype

How much does a misalignment matter?

2 degree head tilt and 2mm shift

Optimization is used to move one image with respect to another, until the chosen criterion (above) is optimized (minimized/maximized) For rigid body shift, there are 6 variables: 3 translations and 3 rotations. Optimal MI Local Optimum Mutual information (MI) Optimal shift movement As you can see, there is the danger of optimization techniques getting “stuck” at local optima.

How good is a fusion? Evaluating a fusion can be subjective Two people fusing the same two images might decide that different shifts are needed Having an objective, calculated number can be helpful

Intensity Based Image Registration Most clinical Treatment Planning Systems offer automated image registration Rather than using points, contours, or edges, these algorithms use the fact that images are data to find relationships between them Let’s look at three types: Local Correlation/ Squared Error Cross Correlation Mutual Information

Intensity Based Image Registration Squared error minimizes the sum of squares of differences between intensity of images at corresponding pixel values Can only be used within the same imaging modality. Correlation Maximizes the correlation in coincidence of image intensities. Can be used between dissimilar imaging modalities, provided grey scales have same relative order. Mutual information Maximizes the mutual information between two images. Can be used between dissimilar imaging modalities.

Images are data maps Image with colors Image with pixel values 1 1 1 1 2 2 2 2 2 2 2 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1

Squared Error Example: assume that the image on the right is the target image………… 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 1 1 2 3 3 3 3 3 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 1 1 2 3 4 4 4 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 2 2 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 = 1 2 2 1 - 1 2 3 4 5 5 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 5 5 4 3 2 1 -1 -1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 -1 -2 -2 -1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 4 4 4 3 2 1 -1 -2 -2 -2 -2 -1 1 1 1 1 1 1 1 1 1 1 1 2 3 3 3 3 3 3 2 1 -1 -2 -2 -2 -2 -2 -2 -1 1 2 2 2 2 2 2 2 2 1 -1 -2 -2 -2 -2 -2 -2 -2 -2 -1 1 1 1 1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 moving image target image SE = 156

Intensity-based image registration: squared error (minimization) If an image is perfectly registered with itself, SE = 0 1 2 downward shift An optimization algorithm would make these shifts “intelligently”. For example, gradient descent algorithms would sequentially choose shift directions that lie close to the steepest descent path.

Cross Correlation In general, the cross-correlation between two variables x and y (equal number of entries for each variable) is: expectation mean of y std dev of y list of pixel values in moving image list of pixel values in target image

Intensity-based image registration: cross-correlation (maximization) For a 0 pixel shift ……….. moving image target image 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 1 1 2 3 3 3 3 3 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 4 4 4 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 4 4 4 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 3 3 3 3 3 2 1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1

Mutual Information image A (moving) image B (target) intensity level in image A

Mutual Information Mutual Information compares the distribution of pixel intensities, NOT the location of the intensities, or the actual values

Mutual information is the reduction in uncertainty of one image, knowing another image.

For a 2 pixel shifted moving image ……….. Intensity-based image registration: Mutual Information (MI) (maximization) For a 2 pixel shifted moving image ……….. moving image target image 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 1 1 2 3 3 3 3 3 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 4 4 4 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 4 4 4 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 3 3 3 3 3 2 1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 4 5

moving target

MI = 0.4082

For a 0 pixel shift: MI = 0.4082 For a 1 pixel shift: MI = 0.6241 For a 2 pixel shift: MI = 1.4293 For 2 images that are exactly the same (same modality, same pixel intensity distribution), the maximum MI value is dependent on the way the image “looks”, i.e., there is no predefined maximum. This is different from cross-correlation, where the maximum CC value is = 1 for the same imaging modality.

Project Description Import the CT_organ_delineation spreadsheet and change the organs to the following values Display this image in gray scale (This is your “target” image) 1-Body = 1000 2-Femurs = 1300 3-ProsatePTV = 1030 4-Bladder = 970 5-Rectum = 1060

**You do not need to actually make them move** Project Description Create a “moving”image by moving the fixed image across an empty matrix one pixel at a time. **You do not need to actually make them move** For each step, compute and plot the Squared Error, Cross Correlation, and Mutual Information as a function of number of steps.

Project Description Step 4 Repeat the steps 1-3, but this time add random noise in the range of 0-50. Repeat this adding noise step 100 times (each time the random values will be different), and plot SE, CC, and MU as a function of steps for each randomization superimposed on the same plot Create a table with the mean and standard deviation of SE, CC, and MI for each step Example plots (Ignore values)

Project Description

Project Description Repeat steps 1-3 (do not add noise) after changing the pixel values to: This simulates an image from a non-CT modality How are the 3 metrics (SE, CC, MI) affected when the images are from different modalities? 1-Body =150 2-Femurs = 220 3-ProsatePTV = 90 4-Bladder = 20 5-Rectum = 70

Project Description Discussing strategies is OK, sharing code is not Don’t overly stress about getting all values to be exactly correct. General shapes of plots and trends of data are more important than values Email me with questions