1. Intensity-based deformable registration of 2D fluoroscopic X- ray images to a 3D CT model Aviv Hurvitz Advisor: Prof. Leo Joskowicz

2. 2 Navigation in orthopedic surgery Applications Determine position of surgical tools relative to anatomy Position surgical robots Match pre-operative model to anatomy [medtronic.com][hss.edu]

3. 3 Navigation in orthopedic surgery Related registration methods Registration to fiducials –Implanted fiducials –On skin Contact based registration (Point-cloud to surface registration) Registration of fluoroscopic X-ray to CT

4. 4 [LaRose 2001; Tomazevic 2003; Knaan 2003; Livyatan 2003; and others] Preoperatively: 1.Acquire CT of anatomy of interest Intraoperatively: 1.Fix a tracking marker to the bone 2.Acquire 2-5 fluoroscopic X-ray images from various camera poses 3.For each image, record: T(tracker  camera) - camera position at acquisition time T(tracker  bone) - bone marker position at acquisition time 4.Obtain an initial transformation estimate T(bone  CT) 5.Estimate transformation T(bone  CT) with the algorithm we describe next Tracker updates T(tracker  bone) continuously, enabling real- time navigation.

5. 5 Registration of fluoroscopic X-ray to CT Input: 1.For 2-5 X-ray images: a.T(tracker  camera) at acquisition time b.T(tracker  bone) at acquisition time 2.Initial T(bone  CT) estimate Repeat until convergence: 1.For each camera position: a.Define a virtual camera positioned relative to CT b.Create DRR 2.Rate similarity of DRRs to X-ray images by a similarity metric 3.Determine the next T(bone  CT) by an optimization algorithm

6. 6 DRR creation [Knaan et al., 03]

7. 7 Is it possible to do the registration without a CT? Advantages: Save time before operation Save costs Decrease radiation exposure Disadvantages: A patient-specific CT provides more information

8. 8 Is it possible to create a patient- specific model without a CT? Approach Use prior information – everyone’s bones are similar Deduce exact shapes of bones from the fluoroscopic images In absence of CT, need to search for both bone pose (6 d.o.f.) and the patient-specific bone shape

9. 9 Image analysis-by-synthesis Define a model of the bone which is parameterized by pose and by shape. Repeat until convergence: 1.Simulate X-ray imaging process to create DRRs 2.Rate similarity of DRRs to fluoroscopic images by similarity metric 3.Modify pose and shape parameters by optimization algorithm When the DRRs match the fluoroscopic images  We found the pose and the shape of the bone  We have registration: a transformation from every point in the model to its corresponding point in the patient anatomy.  We can map data from the model to the patient anatomy. E.g., map the surface of the bone.

10. 10 Active Appearance Model (AAM) [Cootes et al., 98; Matthews and Baker, 04] Model shape variations and appearance variations compactly (with few parameters) Can be trained from a dataset of samples A generative model – can generate new images which are similar to the training images Image analysis-by-synthesis [Edwards, Taylor and Cootes, 98]

11. 11 Each shape is represented by a vector of landmark coordinates Shape Manually label landmark points on training images Define a triangular mesh between the landmarks [Matthews and Baker, 04]

12. 12 Procrustes analysis - align meshes to cancel global variations in translation, rotation and scale PCA on shape vectors

13. 13 Appearance Warp each training image to the base mesh s o. This creates a set of shape-normalized images. Represent pixels inside the base mesh s o as vectors PCA on shape-normalized image vectors:

14. 14 The resulting model [Matthews and Baker, 04]

15. 15 Instantiating the model Input: –Shape parameters –Appearance parameters Computation: 1.Instance’s shape-normalized appearance 2.Instance’s shape mesh

16. 16 Instantiating the model [Matthews and Baker, 04]

17. 17 Fitting AAMs to images Input: –Image I(x) –AAM Goal: find shape p and appearance λ that yield an AAM image similar to I(x). Minimize SSD over λ and p: Sum differences over all pixels x in base mesh

18. 18 The optimization is nonlinear in p and linear in λ. The parameters are found by iterative gradient descent. Standard gradient descent is slow because it requires calculating the partial derivatives at every iteration. There are tricks to estimate gradient quickly [Cootes, 98], or use a pre-computed gradient. [Matthews and Baker, 04]. After improvements, algorithm is fast enough for real-time face tracking

19. 19 Using AAMs for 2D-3D registration The training dataset is CTs (3D images). ROIs of femoral head approximately 100x100x100 pixels. The shapes are defined by tetrahedral meshes (approx. 3,000 nodes and 20,000 tetras). It is too difficult to find landmarks manually.

20. 20 Finding landmarks automatically Select one CT arbitrarily as a template. Define a tetrahedral mesh on the template. Perform 3D-3D deformable registration of each CT to the template. Defines a transformation T i :Template  CT i Apply transformation to template mesh  transfer mesh to the corresponding landmarks in each CT.

21. 21 Finding landmarks automatically 3D-3D registration performed with Elastix software (an ITK wrapper) [Klein and Staring, 07] Registration procedure 1.Preparation: match histogram to template 2.Rigid registration: Initialize by aligning centerpoints of ROIs Metric: Normalized Cross Correlation Optimizer: Standard gradient descent 3.Deformable registration Transform: 3D B-spline. (Grid spacing 8x8x8 voxels) Metric: Sum of squared differences (SSD) Optimizer: Conjugate gradient Use image pyramids (3 levels) Unsupervised Each registration runs approx. 10 mins. Compute all registrations in parallel on bmos cluster (mosix)

22. 22 Building AAM of femur CTs Input: –14 CTs and 14 corresponding meshes Output: –Mean shape mesh and 6 significant basis vectors –Mean appearance image and 10 significant basis vectors

23. 23 Mean volume image - A 0

24. 24 Shape model s 0 + 2 std s 1 s 0 + 2 std s 2 S0S0

25. 25 Using the AAM in 2D-3D registration Repeat until convergence: Generate 2D DRRs out of AAM using current pose, shape, and appearance parameter estimates. Rate similarity of DRRs to fluoroscopic images by similarity metric Modify AAM shape parameters and the pose parameters by optimization algorithm. (We ignore appearance variations)

26. 26 Generating DRRs from AAM – Algorithm 1 1.Generate a 3D image of the AAM instance for each pixel in 3D image: –find tetrahedron containing pixel center (use KD-tree) –find corresponding point in base mesh (an affine transform) –read value from appearance image 2.Proceed with standard ray casting of 3D image [Knaan et al., 03]

27. 27 Generating DRRs from AAM – Algorithm 2 Cast rays directly through tetrahedral mesh of AAM instance Sample points at uniform intervals on ray Find corresponding points in base mesh Read values from appearance image Requires identifying all ray-tetrahedron intersection points. We implemented algorithm of [Marmitt and Slusallek, 06] Instance mesh Base mesh Shape-normalized volume image

28. 28 Parameter search issues Accuracy and Robustness –Search from various initial locations –Genetic algorithm –Downhill simplex optimizer (ITK amoeba) Speed –Multi-resolution –Limit DRR generation to ROIs

29. 29 Current status The naïve search converges to unsatisfactory result (local minimum) Need to improve both accuracy and speed –Search from multiple initialization points –ROIs Think about a smarter optimization algorithm Future: –Experiment on real fluoroscopic images

30. 30 Intensity-based registration of 2D X-ray images to a 3D deformable model

31. 31 Principal Component Analysis (PCA)