1. Medical J. Michael Fitzpatrick, Department of Electrical Engineering and Computer Science Vanderbilt University, Nashville, TN Course on Medical Image Registration, Nov 3-Nov 24, 2008 Institute für Robotic, Leibniz Universität Hannover, Germany Image Registration

2. Schedule Nov 3: Overview of Medical Image Registration Nov 10: Point-based, rigid registration Nov 17: Intensity-based registration Nov 24: Non-rigid registration

3. C omputed T omography (1972) Siemens CT Scanner (Somatom AR)

4. 3D Cross-sectional Image “voxels” (“volume elements”)

5. M agnetic R esonance Imaging GE MR Scanner (Signa 1.5T)

6. P ositron E mission T omography GE PET Scanner

7. Physician has 3 or more views. CT (bone) MR (wet tissue) PET (biological activity)

8. Combining multiple images requires image registration

9. Image Registration: Definition Determination of corresponding points in two different views

10. Motion relative to the scanners can be three-dimensional.

11. Slice orientations vary widely. transversesagittalcoronal

12. Views may be very different.

13. But all orientations and all views can be combined if we have the 3D point mapping.

14. Combining Registered Images = “Image Fusion” MR + PET CT + MR CT MR PET

15. Rigid Registration: Definition Rigid Registration = Registration using a “rigid” transformation

16. Rigid Transformation RigidNon-rigid Distances between all points remain constant. 6 degrees of freedom

17. Nonrigid Transformations can be very complex! [Thompson, 1996]

18. Non-rigid example

19. Registration Dichotomy “Retrospective” methods (nothing attached to patient before imaging)  Match anatomical features: e.g., surfaces  Maximize similarity of intensity patterns “Prospective” methods (something attached to patient before imaging)  Non-invasive: Match skin markers  Invasive: Match bone-implanted markers

20. Most Common Approaches Intensity-based* (not for surgical guidance) Surface-based (requires identified surfaces) Point-based (requires identified points) Stereotactic frames (for surgical guidance) *Sometimes called “voxel-based”

21. The Most Successful Intensity-Based Method: Mutual Information

22. 2D Intensity Histogram (Hill94) CT MR CT intensity MR intensity

23. Misregistration Blurs It 0 cm 2 cm 5 cm MR CT MR PET Hill, 1994

24. A measure of histogram sharpness Most popular “intensity” method Assumes a search method is available Stochastic, multiresolution search common Requires a good starting pose May not find global optimum Not useful for surgical guidance Mutual Information (Viola, Collignon, 1996)

25. Example: Mutual Information Studholme, Hill, Hawkes, 1996, “Automated 3D registration of MR and CT images of the head”, MIA, 1996 (Open movie with QuickTime)

26. The Most Successful Surface-Based Method: The Iterative Closest-Point Algorithm

27. Minimizes a positive distance function Assumes surfaces have been delineated Guaranteed to converge Requires a good starting pose May not find global optimum Can be used for surgical guidance Iterative Closest-Point Method (Besl and McKay, 1992)

28. Start with two surfaces

29. Reorient one (somehow)

32. Pick points on moving surface

34. Remove moving surface

35. Points become proxy for surface

36. Find closest points on stationary surface

37. Measure the total distance

38. Remove stationary surface

39. Points become proxy for surface

40. Register point sets (rigid)

42. Restore stationary surface

43. Find (new) closest points

45. Remove stationary surface

47. Register Points

48. Register Points, and so on…

49. Iterative Closest-Point Algorithm: Find closest points Measure total distance Register points Stop when distance change is small.

50. ICP: Image-to-Image Dawant et al.

51. ICP: Image to Patient The BrainLab VectorVision surgical guidance system uses surface-based registration.

52. ICP requires surface delineation, which is a problem in Image Segmentation Example: Level Set Segmen- tation (Dawant et al.) http://www.vuse.vanderbilt.edu/~dawant/levelset_examples/

53. The fiducial marker is used in prospective registration for image- guided surgery. The Most Common Application of The Point-based Method: The Fiducial Marker

54. Image-Guided Surgery...and the other is the patient. One view is an image.... Just another image registration problem.

55. Acustar ™ Allen, Maciunas, Fitzpatrick, and Galloway 1988-1995 (J&J  Z-Kat) are implanted into the skull. Posts [Maurer, et al., TMI, 1997]

56. Acustar ™ Allen, Maciunas, Fitzpatrick, and Galloway 1988-1995 (J&J  Z-Kat) [Maurer, et al., TMI, 1997] Liquid in marker shows up in image Divot cap is localizable in OR

57. Acustar ™ Allen, Maciunas, Fitzpatrick, and Galloway 1988-1995 (J&J  Z-Kat) [Maurer, et al., TMI, 1997] Marker center and cap center occupy the same position relative to the post

58. Acustar ™ Allen, Maciunas, Fitzpatrick, and Galloway 1988-1995 (J&J  Z-Kat) [Maurer, et al., TMI, 1997] Marker center and cap center occupy the same position relative to the post

59. Find corresponding “fiducial” points Point-based, Rigid Registration View 2 = “Space” 2 View 1 = “Space” 1 Rigid transformation Align corresponding fiducials “targets” are also aligned Find all corresponding “fiducial” points

60. Measures of Error View 1 Registered Views View 2 Fiducial Localization Error (FLE) Target Registration Error (TRE) Fiducial Registration Error (FRE)

61. The Most Successful Point-based Method (by far!): Minimization of Sum of Squares of Fiducial Registration Errors

62. Minimizes a positive distance function Most popular point method Assumes points have been localized Guaranteed to converge Does not require a good starting pose Always finds global optimum Can be used for surgical guidance Minimization of Sum of FRE 2 (Shönemann, Farrell, 1966)

63. Accuracy: State of the Art The best accuracy is probably achieved for the head…

64. Retrospective Registration of Head: Image-to-Image Median Maximum CT-MR : 0.6 mm 3.0 mm PET-MR: 2.5 mm 6.0 mm TRE

65. Prospective Registration of Head: mean TRE ≤ 1 mm (CT) [Hill, JCAT, 1998, Maurer, TMI, 1997]

66. Error Theory for Minimization of Mean-square FRE

67. End of Overview

68. How to Do Minimization of Sum of Squares of Fiducial Registration Errors

69. Sum of Squares: Step 1 Center the points: Centered

70. Step 2 (Shönemann, Farrell, 1966) Determine the Rotation: Centered Centered and Rotated

71. Step 3 (Farrell, 1966) Determine the Translation: Before rotation After rotation, but before translation

72. Error Analysis

73. Start with Assumptions about FLE Independent, normal, isotropic, zero mean Space 1 Space 2

74. “Effective” FLE Space 1 Space 2

75. FRE Statistics: Sibson 1979 Approximate Solution: Configuration doesn’t matter !

76. Principal axes Configuration does matter. d1d1 d2d2 d3d3 [Fitzpatrick, West, Maurer, TMI, ’98] TRE statistics, 1998 Approximate Solution:

77. Got to here Nov 10, 2008

78. 4mm 3mm 2mm 1mm 2mm 1mm FRE = 1mm TRE for FLE of 1mm Marker Placement [West et al., Neurosurgery, April, 2001]

79. A distribution would be better TRE 2 95% level Probability density

80. And what about direction?

81. TRE statistics, 2001 Approximate Solution: TRE 1 TRE 2 TRE 3 [Fitzpatrick and West., TMI, Sep 2001]

82. Some Remaining Problems

83. Isotropic Scaling [Actually now solved: Batchelor, West, Fitzpatrick, Proc. of Med. Im. Undstnd. & Anal., Jul 2002]

84. Anisotropic Scaling (Iterative Solution Only)

85. Register M points sets simultaneously View 1 View 2 ; View 3 View M The “Generalized” Procrustes Problem (Iterative Solution Only)

86. Anisotropic FLE (Iterative Solutions Only)

87. Other Unsolved Problems What is the statistical effect on TRE of dropping or adding a fiducial? Does anisotropy in FLE always, sometimes, or never makes TRE worse? How do we configure markers on a given surface so as to minimize TRE over a given region? Is there a correlation between FRE and TRE? It’ solved: There is no correlation! Fitzpatrick, SPIE Medical Imaging Symposium, to be presented Feb 2009.

88. Extension to perspective transformations. Extension to surface matching. Other Unsolved Problems (cont.)

89. Rigid Registration of the Head State of the Art

90. CT MR-T1 MR-T2 Finding Points = “Localization”

91. Acustar v. Leibinger: Leibinger Grows Up!

92. Retrospective Registration of Head Images: The State of the Art Median Maximum (Acustar) Best CT-MR : 0.6 mm 3.0 mm (0.5 mm) Poor CT-MR: 5.4 mm 61 mm (0.5 mm) Best PET-MR: 2.5 mm 6.0 mm (1.7 mm) Poor PET-MR: 5.3 mm 15 mm (1.7 mm) And how do we know?…

93. R etrospective I mage R egstration E valuation Access: 150+ participants in 20 countries Evaluation: 57 participants in 17 countries External site Vanderbilt 1995-2007

94. End Additional slides follow

95. Categories within error prediction Number of point sets: Two or more Scaling: Isotropic or anisotropic Point-wise weighting: equal or unequal Anisotropic weighting Cost function: squared error or other Point-wise FLE: equal or unequal Spatial FLE: isotropic or anisotropic... Key: Approximate, Negligible progress

96. Anisotropic Scaling R, t = rotation, translation w i 2 = point weighting S = diag( s x, s y, s z ) Given {x i y i w i } find R, t, S to minimize mean FRE 2 Iterative Algorithm: sysy szsz sxsx Search space Problem Statement:

97. Scaling: Anisotropic II R, t = rotation, translation w i 2 = point weighting S = diag( s x, s y, s z ) Given {x i y i w i } find R, t, S to minimize mean FRE 2 Iterative Algorithm:Problem Statement:

98. Spatial Weighting R, t = rotation, translation w i 2 = point weighting S = diag( s x, s y, s z ) A = diag( a x, a y, a z ) Given {x i y i w i } find R, t, S to minimize mean FRE 2 Iterative Algorithm:Problem Statement: Partial Solution:

99. Generalized Procrustes Problem Cost function Iterative method (only)

100. Add Isotropic Scaling Approximate Solution: FRE 2 = sum of squared fiducial registration errors

101. FRE: Generalized + Scaling Approximate Solution: FRE 2 = sum of squared fiducial registration errors

102. TRE statistics with scaling Approximate Solution: TRE 2 = target registration error

103. Applications of TRE Statistics

104. Surgical Paths Radiation Isodose Contours Error Bounds

105. Probe Design Tip = “target” IREDs are fiducials FLE TRE

106. Fiducial-Specific FRE Poor fiducial alignment tends to occur where target registration is good!!

107. Four Solution Methods ( All work equally well [Eggert91]! )

108. Generalized Procrustes Problem (We’ve already done it for M=2.) Problem Statement: Illustration:

109. Generalized Procrustes Problem Iterative Algorithm: Illustration: *Subject to S (m) normalization

110. Approximation Method (due to Sibson, 1979)

111. Approximation Method (cont.)

112. FRE Statistics Problem Statement: Approximate Solution:

113. TRE statistics with scaling Problem Statement: Approximate Solution:

114. What do “solved” and “unsolved” mean? “Solved”, working definition:  Reduced to solving algebraic equations  Iterative algorithm that converges to solution  Approximate solution accurate to “Unsolved”:  Not solved

115. Point-wise weighting: Equal or Unequal (We’ve just looked at this one.) R, t = rotation, translation w i 2 = point weighting Given {x i y i w i } find R, t to minimize mean FRE 2 Problem Statement: See previous slides again! Solution:

116. 1. Performing a Registration x i = point in “from” set; y i = point in “to” set. t = translation vector. R = 3x3 rotation matrix (therefore R t R = I ). Rx i + t Given {x i y i w i } find R, t to minimize mean FRE 2 xixi yiyi ( usually w i =1) a.k.a. The “Orthogonal Procrustes Problem” Problem Statement:

117. 2. Predicting Registration Error View 1 Registered Views View 2 Input --- fiducial positions target position, r FLE distribution Output --- statistics for TRE Output --- statistics for FRE

118. Isotropic Scaling R, t = rotation, translation w i 2 = point weighting s = isotropic scaling Given {x i y i w i } find R, t, s to minimize mean FRE 2 Problem Statement: Solution:

119. Acknowledgements Benoit M. Dawant, PhD, EECS Robert L. Galloway, PhD, BME William C. Chapman, MD, Surgery Jeannette L. Herring, PhD, EECS Jim Stefansic, PhD, Psychology Diane M. Muratore, MS, BME David M. Cash, MS, BME Steve Hartman, MS, BME W. Andrew Bass, BME NSF NIH Matthew Wang, PhD, IBM Jay B. West, PhD, Accuray, Inc. Derek L. G. Hill, PhD Kings College Calvin R. Maurer, Jr., PhD, Stanford U.

120. What could we choose to optimize? Mean-square “Fiducial Registration Error” (FRE 2 )  Known as the “Orthogonal Procrustes Problem” in statistics since 1950s. Robust estimators (median, M-estimators)  Less sensitive to “outliers” Color key: Major problems solved, Much less done