1. Medical Image Registration Dept. of Biomedical Engineering Biomedical Image Analysis www.bmia.bmt.tue.nl

2. Image registration definition ‘‘ Image registration is about determining a spatial transformation – or mapping – that relates positions in one image, to corresponding positions in one or more other images’’ 3D - 3D 3D - 2D 3D/2D - patient Source image Target image

3. Example from our group Medtronic Polestar N20 Intra-operative MRI

7. Pre-Operative Intra- Operative

8. Student Project Wenxin Wang: REGISTRATION

10. Many more examples of imaging modalities X-rays CTAngiographyMRI UltrasoundSPECTPET

11. Application of image registration Same modality, same patient - monitoring and quantifying disease progression over time, - evaluation of intra-operative brain deformation, etc… Different modalities, same patient - correction for different patient position between scans, - linking between structural and functional images, etc… Same modality, different patients - atlas construction - studies of variability between subjects, etc…

12. Temporal registration PET

13. Fusion of images MRI CT Colored overlay

14. PET - CT

15. Region of interest (ROI) selection & color display Fusion of images CT scan of a thyroid gland Fusion of SPECT and CT

16. Fusion of images Protein localization Different spectral bands for optical biomarkers

17. Fusion of images Mapping of calculated probability maps

18. Fusion of images Functional MRI maps on Anatomical MRI fMRI

19. Weighted intensity combination Fusion of images CT MRI Also possible with intermittend presentation (flicker)

20. Fusion of images Checkerboard fusion

21. Fusion of images Linked cursor

22. Fusion of images Radiotherapy planning Iso-dosis contours on CT

23. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,... Matching with pointbased methods Matching with surface based methods Matching with intensity based methods

24. CT images Dynamic series WorkstationPerfusion images o o o o CT Perfusion: matching over time Marcel Quist Philips Medical Systems Medical IT – Advanced Development infarct tumor properties blood perfusion

25. o o o o CT Perfusion Marcel Quist Philips Medical Systems Medical IT – Advanced Development CT images Dynamic series WorkstationPerfusion images infarct tumor properties blood perfusion

26. Blood volume Blood current Time to maximum Aver. passage time Courtesy: Charité, Berlin Functional perfusion images Registration

27. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,... Matching with pointbased methods Matching with surface based methods Matching with intensity based methods

28. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

29. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

30. Image markers

32. Point-Based Registration Coordinates for the fiducials can be found on multiple images One set of fiducials can be lined up with another. Fiducials

33. Device position tracking 2 cameras

34. Finding the Fiducials

35. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

36. 2D Affine Transforms Translations by t x and t y x 1 = a x 0 + b y 0 + t x y 1 = c x 0 + d y 0 + t y Rotation around the origin by  radians x 1 = cos(  ) x 0 + sin(  ) y 0 y 1 = -sin(  ) x 0 + cos(  ) y 0 Zooms by s x and s y x 1 = s x x 0 y 1 = s y y 0 zShear zx 1 = x 0 + h y 0 zy 1 = y 0 http://www.dt.org/html/meshwarp.html

37. 3D Rigid-body Transformations A 3D rigid body transform is defined by: 3 translations - in X, Y & Z directions 3 rotations - about X, Y & Z axes The order of the operations matters TranslationsPitch about x axis Roll about y axis Yaw about z axis

38. Geometrical transformations Rigid preserves straightness of lines intra-patient, rigid anatomy rotation, translation, zoom, skew Curved inter-patient atlas tissue deformation

39. Image Metrics Fixed Image Moving Image Metric Transform Interpolator Value Parameters

40. Distance measures link to pdf

41. Image Metrics – similarity measures 1.Subtraction: 2.Mean squared differences: 3.Correlation coefficient: if the intensities are linearly related. Demo

42. Entropy A measure of dispersion or disorder. High entropy  high disorder. Mutual information A measure of how well one random variable (image intensities) “explains” another. High mutual information  high similarity Similarity Based on Information Theory

43. Mutual Information Correct registration Large mis-registration Wachowiak et al., Proc. SPIE Medical Imaging, 2003 Entropy Mutual information Normalized mutual information

44. MR – MR (identical images) Translation 2 and 5 mm. Mutual Information

45. MR – CT Translation 2 and 5 mm.

46. Demo

47. Two images are similar if changes of intensity occur at the same locations. Gradient Field Normalized Gradient Field: Regularized Normalized Gradient Field: Registration Distance Measure (1): Normalized Gradient Field Distance measure of NGF: Normalized Gradient Field

48. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

49. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

51. Optimization Optimization involves finding some “best” parameters according to an “objective function”, which is either minimised or maximised The “objective function” is often related to a probability based on some model Value of parameter Objective function Most probable solution (global optimum) Local optimum

52. Plotting the Metric Mean Squared Differences Transform Parametric Space Sensitivity analysis

53. The Best Transform Parameters Evaluation of the full parameter space is equivalent to performing optimization by exhaustive search Very Safe but Very Slow Better Optimization Methods: for example: Gradient Descent

54. Optimization in Image Registration Main goal: To determine the transformation parameters that result in the minimum value of a ‘distance measure’. Transformation parameters: Translations Rotations Scaling Find the “best”, or optimum value of an objective (cost) function. Very large research area. Multitude of applications.

55. Image Registration Framework Fixed Image Moving Image Metric Transform Interpolator Optimizer Parameters

56. Applications of Optimization Engineering design Business and industry Radiotherapy planning Biology and medicine Economics Systems biologyManagement Design of materials Manufacturing design Bioinformatics Proteomics Image registration Finance Simulation and modeling

57. Global and local optimization

58. Local Optimization Start

59. End Local Optimization

60. Start Global Optimization

61. End Global Optimization

62. Gradient Descent Optimizer f( x, y ) S = L ∙ G( x, y ) f( x, y ) ∆ G( x, y ) = S = Step L = Learning Rate

63. Gradient Descent Optimizer f( x, y ) S = L ∙ G( x, y ) f( x, y ) ∆ G( x, y ) =

64. Registration Framework Reference Image Template Image Calculate Distance Measure Condition Met? Transformed Template Image Optimize Transformation Parameters Transform Template Image Yes NO

65. Multi-Resolution Registration Framework Registration Fixed ImageMoving Image

66. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

67. Multi-Modality Registration Fixed Image Moving Image Registered Moving Image

68. Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

69. Visual Integration Platform for Enhanced Reality (VIPER) Collaboration with Dr. Wieslaw Nowinski, Cerefy Atlas, A*Star, Singapore

70. Substantia Nigra Nucleus Subthalami Motor Tract Atlas

71. Substantia Nigra Nucleus Subthalami Motor Tract Atlas

72. Cerefy Anat.Brain Atlas Wieslaw Nowinski, Singapore Anatomy atlas vs. function atlas (fMRI)

73. Manual marking of recognizable landmarks in both atlas and high resolution data. Example of slice TT88s / L+5mm Registration of reference data by landmarks Select points on conditions: Clearly visible in both atlas and reference data; Distribution in whole brain volume; Number of landmarks is unlimited. E. Bennink J. Korbeeck