Multimodal Registration of Medical Data Prof. Leo Joskowicz School of Computer Science and Engineering The Hebrew University of Jerusalem

Intensity-based rigid registration (1) Use intensity information to define the measure of similarity between two data sets Rationale: the closer the data sets are, the more similar their intensity values are. No segmentation is necessary! The entire data set is used. Slow, especially for 3D data sets. The parametric space of transformations is searched incrementally from an initial configuration. The search space is six- dimensional (3 rotations and 3 translations)

Intensity-based registration (2) Similarity measures –cross correlation –histogram correlation –mutual information –intensity values Uses: brain CT/MRI, Xray/CT Example: fluoroscopic Xray to CT

Xray/CT registration Problem definition: given –preoperative CT data set of rigid structure –intraoperative Xray images from a calibrated camera at relatively known spatial configurations Find a rigid transformation that matches the CT data set to the intraoperative Xrays so that if Xrays of the CT were taken from the transformed camera positions, the resulting Xray images would be identical to the intraoperative ones.

Xray/CT registration setup 2D/3D registration problem! Ref C-arm Ref patient Ct slices Ref ct volume DRR Fluoroscopic image

Xray to CT registration algorithm Input: preop CT, intraop Xray I fluoro, intrinsic Xray camera parameters, initial guess p 0 for camera pose 1. generate simulated Xray I DRR (called digitally reconstructed radiograph, or DRR) at camera pose p i 2. Compute dissimilarity between I DRR and I fluoro by comparing their intensities 3. Compute a new camera pose p i+1 = p i + d that best reduces the dissimilarity between I DRR,and I fluoro ) repeat until no progress can be made

Digitally reconstructed radiographs

Generating DRRs For each pixel in the DRR plane, construct the ray emanating from the camera focal point. Sum up the intensities of the CT voxel values according to the Xray attenuation formula to obtain the gray level value of the DRR pixel. DRR CT Camera

Generated DRRs

Real X-ray vs DRR

Similarity measure Pairwise comparison of normalized pixel intensity values: I DRR (i,j) and I fluoro (i,j) are the pixel values I DRR and I fluoro are the average image values T is the region of interest

Examples of initial poses registrations (DRRs only)

Actual use: radiation therapy with the Cyberknife (radiation therapy)

Cyberknife: system setup

Frameless radiation therapy Stereotactic setup Track head with Xrays before each dose application

Matching skull X-ray and DRR Match only regions

CyberKnife system Description The acquired radiographs are masked to isolate the same regions of interest. Sobel Edge detection filter finds the point where the radial ray through the center of the region crosses the skull edge. Interpolating over several pixels,to better resolve the maximum. All feature vector components carried equal weight.

CyberKnife system Description(4) The iteration are well describes by Eulerian rotation convertion. Rotation of the skull, are modeled by rotating the camera. Using Semiempirical algorithms to find next iteration.

CyberKnife system Description(5) Resolving outer edged of the skull by adjusting its integration step length according to the local gradient of the Hounsfield numbers. Compensating Residual differences in contrast between DRRs and radiographs by fitting a gamma function that matches brightness hystograms, and applying this function to subsequent DRRs.

CyberKnife system Results (1) The tests were performed using an anthropomorphic head phantom consisting of a human skull encased in plastic. The phantom was held by a fixture, that allowed it to be translated and rotated with six degrees of freedom.

CyberKnife system Results (2)

CyberKnife system Results (3)

CyberKnife system Results (4)

CyberKnife system Results (5)

CyberKnife system Conclusions(1) The numerical offset of a point in the skull may be large due to large target site’s distance from the rotational axes. Empirical mean radial error was only 0.7 mm, indicating that the uncertainties in the six degrees of freedom are correlated (expected).

CyberKnife system Conclusions(2) No systematic errors. No linkage (except edge cases), between the least square statistic and the angle error. No simulation or real trial has suggested any possibility that LSS could mistakenly converge to a good minimum, where we are far from the true position.

Intensity-based registration Advantages –no segmentation, automatic –selective regions –potentially accurate Disadvantages –large seach space, many local minima –slow

Deformable registration: scope Necessary for soft tissue organs and for cross- patient comparisons –brain images before and during surgery –anatomical structures at different times or from patients: tumor growth, heart beating, compare –matching to atlases Much more difficult than rigid registration! –problem is ill-posed; solution is not unique –error measurements and comparisons are difficult –local vs. global deformations?

Deformable registration: properties Mapping transformation can be –global, e.g., a bi- or tri-variate polynomial –local, e.g.a fine grid with displacement vectors Define an energy function that should be minimized to make the data sets match. Usually comes after rigid registration to get an approximate position estimate. Both geometry based and intensity-based techniques exist.

rigid affinetriliear quadratic transform Mathematics of deformations Global transformations Local spline deformation

Square shift Global deformation transformations Scale

Local grid-based deformable registration image 1 image 2image 1+2 deformation map

Example: MRI slice matching image 1image 2 after registration difference image without deformation difference image with deformation

Brain tumor matching - 2D map

Brain tumor matching - 3D map sourcetarget match

Initial configuration After rigid registration After deformable registration with local splines Example: spine matching

Deformable registration techniques Too many to list here! –Optical flow model –Physics-based: elastic and fluid models Use an elastic or deformable model Validation is difficult

Commercial products Medical image processing software packages include some registration capabilities (manual or semi-automatic feature selection) Contact-based rigid registration of CT and optical tracker in orthopaedics and neurosurgery (half a dozen companies) Intraoperative Open MR to tracker rigid registration

The future: research directions In many areas, the problem is far from solved: similar to image segmentation! Much clinical validation is needed. More coverage of other anatomy (60% focus on brains!) Interleave segmentation and registration Difficult data sets: 2D and 3D ultrasound images, video sequences, portal images Model-based techniques are the most likely to be sufficiently robust for clinical use Integration requirements are very important.

Bibliography (1) Two chapters on registration in Computer-Integrated Surgery, Taylor et al, MIT Press, Medical Image Registration, Hajnal et al, CRC Press 2001 “A survey of medical image registration”, Maintz and Viergever, Medical Image Analysis Journal, 2(1), Oxford University Press 1998 (over 150 references!) “A method for registration of 3D shapes”, Besl and McKay, IEEE Trans. on Pattern Analysis, 14(2), Special issue on Biomedical Image Registration, Image and Vision Computing, Vol 19(1-2), “Deformable models in medical image analysis: a survey”, McInerney and Terzopolous, Medical Image Analysis 1(2), 1996.

Bibliography (2) “Retrospective registration of tomographic brain images”, J. Mainz, PhD Thesis, Utrecht U., “Localy affine registration of free-form surfaces”, J. Feldmar and N. Ayache, Proc. IEEE CVPR, “Matching 3D anatomical surfaces with non-rigid deformations using octree splines”, R. Szeliski and S. Lavallee, Int. Journal of Computer Vision 18(2), “Fast intensity-based non-rigid matching”, P. Thirion, Proc. Conf. Medical Robotics and CAS, “Multimodal volume registration by maximization of mutual information, W.Wells, P.Viola, R.Kikinis. (idem)