1. Multimodal Registration of Medical Data Prof. Leo Joskowicz School of Computer Science and Engineering The Hebrew University of Jerusalem

2. Outline of the talk Introduction Classification of registration methods Rigid registration methods Deformable registration methods Challenges and research directions This is only an introductory overview!

3. What is medical multimodal registration? “The process of establishing a common geometric reference frame between two or more data sets from different modalities taken at different times for the purpose of improving preoperative and intraoperative information for diagnosis and navigation”

4. Multimodal registration Registration PreoperativeIntraoperative X-rays US NMR CTMRIFluoro CAD Tracking US Open MR Special sensors Video Combined Data

5. Why multimodal integration? Combine different types of information: CT/MRI, MRI/NMR,... Track relative position of instruments and anatomy during surgery: CT or MRI/tracker. Compare “before” and “during” information: MRI/Ultrasound, CT/Xray,... Supplement the quality/field of view of preoperative info with intraoperative info Clinical applications usually require more than one registration: registration chains.

6. Registration of MRI and NMR Ref_MRIRef_NMR

7. Registration Ultrasound and Doppler images

8. Registration of preoperative CT and intraoperative tracker data

9. 2D/3D X-ray/CT registration preoperative CT slices intraoperative X-ray images

10. CT images fluoroscopic images optical tracker instruments patient Registration chain computer 3D surface model

11. Not one but many registration problems! Many two, three, and n-way multimodal integration problems! Great differences depending on –the type of data to be matched –the anatomy that is being imaged –the specific clinical requirements of procedures Accuracy, assumptions, and technical requirements vary greatly from type to type!

12. data set 1 data set 2 combined data sets Transformation Generic registration problem

13. Generic registration procedure 1. Distortion correction and camera calibration for each modality while dissimilarity > 0 and improvement do 2. Feature extraction 3. Feature pairing 4. Similarity formulation and outlier removal 5. Dissimilarity reduction (optimization) Great differences in each step depending on images and task!

14. Feature pairing data set 1 data set 2 combined data sets Transformation Feature extraction Generic registration problem Outlier removal Similarity formulation Dissimilarity reduction Transformation combined data sets Optimization result

15. Classification of registration methods Dimensionality Type of registration basis Nature and domain of the transformation Interaction Optimization procedure Modalities Subject and anatomy

16. Dimensionality Spatial –2D/2D: slices of MRI, CT, NMR, portal images –2D/3D: Xray/CT, US/CT, video/CT –3D/3D: MRI/CT, NMR/MRI, Temporal –slow: comparison of data sets, e.g., bone growth –fast: beating heart, angiography, injected imaging agents

17. Registration basis Image extrinsic: objects attached to the patient –invasive: stereotactic frame, fiducials (screws) –non-invasive: frame, dental adapter, skin fiducials Image intrinsic: image content only –landmark based: anatomical or geometric –segmentation based: rigid or deformable models –voxel based: reduction (scalars, vectors), image contents Non-image: data from other sources –trackers, laser scanners, robot arms

18. Registration transformation

19. Interaction Interactive (manual) –initialization supplied –no initialization supplied Semi-automatic –user initialization –user steering/correcting –both Automatic

20. Optimization procedure Parameters computed Parameters searched for Mathematical characteristics –optimization function: linear, nonlinear –solution method: SVD, Lavenberg-Marquard Multistep approach: –fast but approximate for coarse registration, followed more expensive but more precise for fine registration

21. Modalities Monomodal –CT, MR, PET, Xray, US, video, portal Multimodal –CT/MR, CT/NMR, MR/NMR –Xray/CT, video/CT Modality to model –model can be atlas, CAD model, etc. Patient to modality –tracker data, robot arm, etc.

22. Subject and anatomy Subject –intrasubject, intersubject, atlas Anatomy –head: brain and skull, eye, maxillofacial –thorax: entire, cardiac, breast –abdomen: general, kidney, liver –pelvis and perineum –limbs: femur and tibia, humerus, hand –spine and vertebra

23. Rigid registration Rigid transformation Applicable to rigid structures which change their position but not their shape: – bones of the same patient –implanted fiducials, stereotactic frames –approximation for quasi rigid structures (brain) –as a first step to deformable registration Widely used in: –orthopaedic aplications –data from CT, Xray, trackers

24. Deformable registration General curved mapping Necessary for matching soft tissue organs and for cross-patient comparisons –brain images before and during surgery –anatomical structures at different times or from patients: tumor growth, heart beating, compare –matching to atlases Much more difficult than rigid registration! –problem is ill-posed; solution is not unique –error measurements and comparisons are difficult

25. Rigid registration techniques technical classification Two main approaches Geometric approach: use spatial disparity between selected features to reduce difference –distance between two matching points Intensity-based approach: use the pixel intensity values to reduce difference –intensity gradient between two pixels or voxels –mutual information: maximize image correlation

26. General rigid registration problems geometric approach 3D/3D point to point registration based on least squares minimization 2D/3D line to point registration Iterative Closest Point (ICP) algorithm: automatic feature paring Octree splines: hierarchical data representation actual use: landmarks + cloud of points

27. Rigid registration: basic concepts Features: points, lines, surfaces Feature pairings: predefined or automatic –point/point, point/line, spline/spline Similarity measure: sum of distances between pairwise features Dissimilarity reduction: minimize sum of distances

28. Rigid registration: mathematics (1) Attach coordinate systems to each data set S 1, S 2 Define the rigid transformation P from one data set to the other. Transformation: rotation and translation Goal: for all points in data sets

29. Rigid registration: mathematics (2) Rotation matrices –Euler angles –Quaternions

30. Registration: mathematics (3) n pairs of points: (p i, q i ) Distance between pairwise points: Difference metric: sum of pairwise distances Dissimilarity reduction: minimize sum of paiwise distances

31. Registration: mathematics (4) Solving the minimization problem: –closed-form solution for three points –closed form solution of min problem (Horn) –nonlinear optimization methods: Powell, Lavenberg-Marquard (numerical recipes in C) –quadratic optimization (NNLS) of approximation for small angle vectors  Robust estimation: establish threshold for distance between pairs and eliminate those with distance higher than threshold.

32. Three points closed form solution Match three points in two coordinate systems left p L1, p L2, p L3 right p R1, p R2, p R3 1.Choose p 1 to be the origin. 2.Construct x axis: 3.Construct y axis:

33. Three points closed form solution(2) Construct z axis: z = x x y Build rotation matrices for two points sets: R L =[x L, y L, z L ] and R R =[x R, y R, z R ] The rotation between “right” and “left” is: R=R L R R T The translation is: t=p L1 – R(p R1 )

34. Three points closed form solution(3) Problems with this solution: –This method does not use the information about each of the three points equally –It cannot use the information of more than three points when available. –Numerical stability problems.

35. Horn closed form solution (1) Given n points in two coordinate systems: right p Ri and left p Li. Error for each point: e i = p Ri – R(p Li ) – t Find R and t that minimize the sum of squared errors Translate all points to their centroids

36. Horn closed form solution (2) New error term: e i = p ’ Ri – R(p ’ Li ) – t ’, t ‘ = t – c R + R(c L ) The sum of the square errors: Middle sum expression equals to 0 Last sum expression is minimized when t = c R – R(c L ) (desired translation)

37. Horn closed form solution(3) We should minimize then (for R): First and last terms are constants independent of R We should maximize the second term We represent R using unit quaternions, so we get:

38. Horn closed form solution(4) Using quaternion properties, the expression can be written as Quaternion products can be expressed using matrices:

39. Horn closed form solution(5) From the sum, we get: q T Nq. The vector q which maximizes q T Nq is the eigenvector corresponding to the most positive eigenvalue of the matrix N. Define:

40. Horn closed form solution(6) We can express N as: And we return the rotation matrix R represented by the unit quaternion q, and the translation vector t, calculated after we have R.

41. Horn closed form solution(7) Advantages of the Horn closed-form solution Best possible solution is achieved by one step without iteration. No need for initial good guess to bring us close. All the information in the data sets is used. Symmetry of solution (it gives the exact inverse of the best transformation in the other direction).

42. Iterative closest point algorithm (1) Besel and McKay, 1992 The main problem: which features to pair? Heuristic: –pick a set of predefined features in one data set –choose the closest feature to each in the other data set. –solve the problem, bringing the data sets closer –repeat the pairing selectiob until the distance is minimized.

43. Iterative closest point algorithm (2) ICP always converges monotonically to a local minimum with respect to the mean- square distance objective function. Works when the data sets are “reasonably” close --> requires a good initial guess. Closest point operation is the most expensive operation --> data structure for fast access (octrees, see later).

44. Rigid registration examples 3D/3D cloud of points to cloud of points –CT/CT, laser scanner/CT 3D/3D ridge lines to ridge lines –CT/CT 3D/3D tracker cloud of points to CT points –registration for intraoperative navigation 2D/3D contour lines to CT points –anatomical image-based registration Octree splines

45. 3D/3D countour point registration points from CAD model points from 3D contour

46. 3D/3D tracking points to CT data cloud of points from tracker points from CT

47. 3D/3D ridge lines registration ridge lines Advantage: very few features to match!

48. 2D/3D line/point registration 3D data set 2D View 1 2D View 2

49. Octree spline subdivision Example of quadtree subdivision Hierarchical space subdivision Reduces query time from O(n) to O(log n)

50. 2D/3D registration of simulated image and femur octree

51. 3d/3d registration with octree: vertebra