1. MPHY8149: Image Registration
David Sterling, MS, DABR
Fall 2018

2. What is Image Registration?
Sometimes called Image Fusion, Image Registration is the overlaying of different imaging studies on one another
Secondary datasets are used to superimpose specific helpful information onto the primary dataset.

3. What is Image Registration?
Planar x-ray CT
PET
SPECT
MRI
Ultrasound
Useful Patient Data!

4. What is Image Registration
Imaging modalities provide different information
Anatomic
Metabolic
Functionality
Uptake of tracer
Motion
Positioning Accuracy
Combining this data is much more powerful than one scan on its own
Can be many for a single patient

5. Example: PET scan Patient given radiotracer (e.g. FDG)
Patient receives CT scan
Patient receives PET scan
Patient receives CT Scan for two reasons

6. Example: PET Reason 1: Attenuation Correction

7. Example: PET scan
Reason 2: Anatomical Information

9. Image Registration
Even images of the same modality can provide information when fused
MRI
T1 vs. T2
With and without contrast
Image acquisition settings
Timing
Over time to check for changes
Before and after surgery

10. How is it used clinically?
Example 1: PET we already reviewed
Misalignment or miscalibration in CT/PET/Couch sometime means that CT and PET do not line up well and must be manually adjusted

11. Fusion

14. Image Registration Categories
Manual
Purely automatic
Feature-based image registration
Intensity-based image registration
Semi-manual
Purely automatic followed by manual tweaking

15. Automatic Image Registration Techniques
Feature-based image registration
Point-based matching (usually 2D images).
Contour-based matching (based on manual or automatic segmentation).
Chamfer (edge) matching, using edge detection software.
point-based
contour-based
point-based

16. y C x z y x z A B y x w z

17. Feature-based image registration: Contour-based matching
Matches one contour to another (in 2D or 3D), so as to minimize the “difference” between them. (For example, difference could be the sum of square of distances from each point on one contour to the nearest point on the other contour).
Procedure is more laborious (hence slower) than point matching, since it requires human interaction for contour drawing.
Can be made less human-dependent by using automatic segmentation algorithms. However, automatic segmentation can be very error prone!

18. Feature-based image registration: Contour-based matching
Scenario:
Patient is planned with CT image. Contours of interest are drawn on planning CT image.
Images are acquired prior to each treatment (using either cone-beam computed tomography or CT-on-rails). Contours are drawn on these images.
Contour-to-contour matching is performed between the planning CT and pre-treatment CT.
Patient is shifted based on the contour matching, and then treated.
bony matching or “soft tissue” (prostate) matching (based on your protocol). Bony matching is more common.

19. SIEMENS CT-ON-RAILS
VARIAN CBCT SYSTEM

20. Feature-based image registration: Contour-based matching
Planning image
= bony matching
= soft tissue matching

21. Feature-based image registration: Chamfer-based matching
Use edge detection software to extract only the ridges/edges in the image.
Match the edges/ridges to each other.
Decreases the complexity from a full image to just a set of features.

22. PET CT Misalignment Examples

23. PET CT Misalignment
If these errors are present when PET and CT are taken at the same time, the errors can obviously be much greater when taken at different times

24. What to do?
We can adjust the registration to ensure that the area we are concerned about lines up well
Sometimes, this requires multiple fusions for a single pair of datasets (e.g., H&N patient with PET)
Sometimes “eyeballing it” on different screens is actually better than a bad fusion
Deformable registration
Not in the scope of this talk (though I can take questions at the end), but don’t believe the hype

25. How much does a misalignment matter?

26. 2 degree head tilt and 2mm shift

27. Optimization is used to move one image with respect to another, until the chosen criterion (above) is optimized (minimized/maximized)
For rigid body shift, there are 6 variables: 3 translations and 3 rotations.
Optimal MI
Local Optimum
Mutual information (MI)
Optimal shift
movement
As you can see, there is the danger of optimization techniques getting “stuck” at local optima.

28. How good is a fusion? Evaluating a fusion can be subjective
Two people fusing the same two images might decide that different shifts are needed
Having an objective, calculated number can be helpful

29. Intensity Based Image Registration
Most clinical Treatment Planning Systems offer automated image registration
Rather than using points, contours, or edges, these algorithms use the fact that images are data to find relationships between them
Let’s look at three types:
Local Correlation/ Squared Error
Cross Correlation
Mutual Information

30. Intensity Based Image Registration
Squared error
minimizes the sum of squares of differences between intensity of images at corresponding pixel values
Can only be used within the same imaging modality.
Correlation
Maximizes the correlation in coincidence of image intensities.
Can be used between dissimilar imaging modalities, provided grey scales have same relative order.
Mutual information
Maximizes the mutual information between two images.
Can be used between dissimilar imaging modalities.

31. Images are data maps Image with colors Image with pixel values 1 1 1 12 2 2 2 2 2 2 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1

32. Squared Error
Example: assume that the image on the right is the target image………… 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 1 1 2 3 3 3 3 3 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 1 1 2 3 4 4 4 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 2 2 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 = 1 2 2 1 - 1 2 3 4 5 5 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 5 5 4 3 2 1 -1 -1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 -1 -2 -2 -1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 4 4 4 3 2 1 -1 -2 -2 -2 -2 -1 1 1 1 1 1 1 1 1 1 1 1 2 3 3 3 3 3 3 2 1 -1 -2 -2 -2 -2 -2 -2 -1 1 2 2 2 2 2 2 2 2 1 -1 -2 -2 -2 -2 -2 -2 -2 -2 -1 1 1 1 1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
moving image
target image
SE = 156

33. Intensity-based image registration: squared error (minimization)
If an image is perfectly registered with itself, SE = 0 1 2
downward shift
An optimization algorithm would make these shifts “intelligently”. For example, gradient descent algorithms would sequentially choose shift directions that lie close to the steepest descent path.

34. Cross Correlation
In general, the cross-correlation between two variables x and y (equal number of entries for each variable) is:
expectation
mean of y
std dev of y
list of pixel values in moving image
list of pixel values in target image

35. Intensity-based image registration: cross-correlation (maximization)
For a 0 pixel shift ………..
moving image
target image 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 1 1 2 3 3 3 3 3 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 4 4 4 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 4 4 4 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 3 3 3 3 3 2 1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1

36. Mutual Information image A (moving) image B (target)
intensity level in image A

37. Mutual Information
Mutual Information compares the distribution of pixel intensities, NOT the location of the intensities, or the actual values

38. Mutual information is the reduction in uncertainty of one image, knowing another image.

39. For a 2 pixel shifted moving image ………..
Intensity-based image registration: Mutual Information (MI) (maximization)
For a 2 pixel shifted moving image ………..
moving image
target image 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 1 1 2 3 3 3 3 3 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 4 4 4 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 4 4 4 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 3 3 3 3 3 3 2 1 1 2 3 4 5 5 4 3 2 1 1 2 2 2 2 2 2 2 2 1 1 2 3 4 4 4 4 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 3 3 3 3 3 2 1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 4 5

40. moving
target

41. MI =

42. For a 0 pixel shift: MI =
For a 1 pixel shift: MI =
For a 2 pixel shift: MI =
For 2 images that are exactly the same (same modality, same pixel intensity distribution), the maximum MI value is dependent on the way the image “looks”, i.e., there is no predefined maximum.
This is different from cross-correlation, where the maximum CC value is = 1 for the same imaging modality.

44. Project Description
Import the CT_organ_delineation spreadsheet and change the organs to the following values
Display this image in gray scale (This is your “target” image)
1-Body = 1000
2-Femurs = 1300
3-ProsatePTV = 1030
4-Bladder = 970
5-Rectum = 1060

45. **You do not need to actually make them move**
Project Description
Create a “moving”image by moving the fixed image across an empty matrix one pixel at a time.
**You do not need to actually make them move**
For each step, compute and plot the Squared Error, Cross Correlation, and Mutual Information as a function of number of steps.

46. Project Description Step 4
Repeat the steps 1-3, but this time add random noise in the range of 0-50.
Repeat this adding noise step 100 times (each time the random values will be different), and plot SE, CC, and MU as a function of steps for each randomization superimposed on the same plot
Create a table with the mean and standard deviation of SE, CC, and MI for each step
Example plots (Ignore values)

47. Project Description

48. Project Description
Repeat steps 1-3 (do not add noise) after changing the pixel values to:
This simulates an image from a non-CT modality
How are the 3 metrics (SE, CC, MI) affected when the images are from different modalities?
1-Body =150
2-Femurs = 220
3-ProsatePTV = 90
4-Bladder = 20
5-Rectum = 70

49. Project Description Discussing strategies is OK, sharing code is not
Don’t overly stress about getting all values to be exactly correct. General shapes of plots and trends of data are more important than values
me with questions