Medical Image Registration Dept. of Biomedical Engineering Biomedical Image Analysis

Image registration definition ‘‘ Image registration is about determining a spatial transformation – or mapping – that relates positions in one image, to corresponding positions in one or more other images’’ 3D - 3D 3D - 2D 3D/2D - patient Source image Target image

Example from our group Medtronic Polestar N20 Intra-operative MRI

Pre-Operative Intra- Operative

Student Project Wenxin Wang: REGISTRATION

Many more examples of imaging modalities X-rays CTAngiographyMRI UltrasoundSPECTPET

Application of image registration Same modality, same patient - monitoring and quantifying disease progression over time, - evaluation of intra-operative brain deformation, etc… Different modalities, same patient - correction for different patient position between scans, - linking between structural and functional images, etc… Same modality, different patients - atlas construction - studies of variability between subjects, etc…

Temporal registration PET

Fusion of images MRI CT Colored overlay

PET - CT

Region of interest (ROI) selection & color display Fusion of images CT scan of a thyroid gland Fusion of SPECT and CT

Fusion of images Protein localization Different spectral bands for optical biomarkers

Fusion of images Mapping of calculated probability maps

Fusion of images Functional MRI maps on Anatomical MRI fMRI

Weighted intensity combination Fusion of images CT MRI Also possible with intermittend presentation (flicker)

Fusion of images Checkerboard fusion

Fusion of images Linked cursor

Fusion of images Radiotherapy planning Iso-dosis contours on CT

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,... Matching with pointbased methods Matching with surface based methods Matching with intensity based methods

CT images Dynamic series WorkstationPerfusion images o o o o CT Perfusion: matching over time Marcel Quist Philips Medical Systems Medical IT – Advanced Development infarct tumor properties blood perfusion

o o o o CT Perfusion Marcel Quist Philips Medical Systems Medical IT – Advanced Development CT images Dynamic series WorkstationPerfusion images infarct tumor properties blood perfusion

Blood volume Blood current Time to maximum Aver. passage time Courtesy: Charité, Berlin Functional perfusion images Registration

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,... Matching with pointbased methods Matching with surface based methods Matching with intensity based methods

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

Image markers

Point-Based Registration Coordinates for the fiducials can be found on multiple images One set of fiducials can be lined up with another. Fiducials

Device position tracking 2 cameras

Finding the Fiducials

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

2D Affine Transforms Translations by t x and t y x 1 = a x 0 + b y 0 + t x y 1 = c x 0 + d y 0 + t y Rotation around the origin by  radians x 1 = cos(  ) x 0 + sin(  ) y 0 y 1 = -sin(  ) x 0 + cos(  ) y 0 Zooms by s x and s y x 1 = s x x 0 y 1 = s y y 0 zShear zx 1 = x 0 + h y 0 zy 1 = y 0

3D Rigid-body Transformations A 3D rigid body transform is defined by: 3 translations - in X, Y & Z directions 3 rotations - about X, Y & Z axes The order of the operations matters TranslationsPitch about x axis Roll about y axis Yaw about z axis

Geometrical transformations Rigid preserves straightness of lines intra-patient, rigid anatomy rotation, translation, zoom, skew Curved inter-patient atlas tissue deformation

Image Metrics Fixed Image Moving Image Metric Transform Interpolator Value Parameters

Distance measures link to pdf

Image Metrics – similarity measures 1.Subtraction: 2.Mean squared differences: 3.Correlation coefficient: if the intensities are linearly related. Demo

Entropy A measure of dispersion or disorder. High entropy  high disorder. Mutual information A measure of how well one random variable (image intensities) “explains” another. High mutual information  high similarity Similarity Based on Information Theory

Mutual Information Correct registration Large mis-registration Wachowiak et al., Proc. SPIE Medical Imaging, 2003 Entropy Mutual information Normalized mutual information

MR – MR (identical images) Translation 2 and 5 mm. Mutual Information

MR – CT Translation 2 and 5 mm.

Demo

Two images are similar if changes of intensity occur at the same locations. Gradient Field Normalized Gradient Field: Regularized Normalized Gradient Field: Registration Distance Measure (1): Normalized Gradient Field Distance measure of NGF: Normalized Gradient Field

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

Optimization Optimization involves finding some “best” parameters according to an “objective function”, which is either minimised or maximised The “objective function” is often related to a probability based on some model Value of parameter Objective function Most probable solution (global optimum) Local optimum

Plotting the Metric Mean Squared Differences Transform Parametric Space Sensitivity analysis

The Best Transform Parameters Evaluation of the full parameter space is equivalent to performing optimization by exhaustive search Very Safe but Very Slow Better Optimization Methods: for example: Gradient Descent

Optimization in Image Registration Main goal: To determine the transformation parameters that result in the minimum value of a ‘distance measure’. Transformation parameters: Translations Rotations Scaling Find the “best”, or optimum value of an objective (cost) function. Very large research area. Multitude of applications.

Image Registration Framework Fixed Image Moving Image Metric Transform Interpolator Optimizer Parameters

Applications of Optimization Engineering design Business and industry Radiotherapy planning Biology and medicine Economics Systems biologyManagement Design of materials Manufacturing design Bioinformatics Proteomics Image registration Finance Simulation and modeling

Global and local optimization

Local Optimization Start

End Local Optimization

Start Global Optimization

End Global Optimization

Gradient Descent Optimizer f( x, y ) S = L ∙ G( x, y ) f( x, y ) ∆ G( x, y ) = S = Step L = Learning Rate

Gradient Descent Optimizer f( x, y ) S = L ∙ G( x, y ) f( x, y ) ∆ G( x, y ) =

Registration Framework Reference Image Template Image Calculate Distance Measure Condition Met? Transformed Template Image Optimize Transformation Parameters Transform Template Image Yes NO

Multi-Resolution Registration Framework Registration Fixed ImageMoving Image

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

Multi-Modality Registration Fixed Image Moving Image Registered Moving Image

Classification of registration algorithms: Image dimensionality2D, 3D, time,... Registration basispoint sets, markers, surfaces,... Geometrical transformationsaffine, perspective,... Degree of interactionuser initialization, automatic Optimization proceduremax distance, gradient descent Modalitiesmulti-modal, intra-modal,... Subjectinter-patient, atlas,... Objecthead, vertebra, liver,...

Visual Integration Platform for Enhanced Reality (VIPER) Collaboration with Dr. Wieslaw Nowinski, Cerefy Atlas, A*Star, Singapore

Substantia Nigra Nucleus Subthalami Motor Tract Atlas

Substantia Nigra Nucleus Subthalami Motor Tract Atlas

Cerefy Anat.Brain Atlas Wieslaw Nowinski, Singapore Anatomy atlas vs. function atlas (fMRI)

Manual marking of recognizable landmarks in both atlas and high resolution data. Example of slice TT88s / L+5mm Registration of reference data by landmarks Select points on conditions: Clearly visible in both atlas and reference data; Distribution in whole brain volume; Number of landmarks is unlimited. E. Bennink J. Korbeeck