1. Feature based deformable registration of neuroimages using interest point and feature selection Leonid Teverovskiy Center for Automated Learning and Discovery Carnegie Mellon University

2. Description of the problem Our task is to align given neuroimages so that their corresponding anatomical structures have the same coordinates

3. Description of the problem Our task is to align given neuroimages so that their corresponding anatomical structures have the same coordinates

4. Existing approaches Landmark based registration. Deformation between images is calculated based on user defined correspondences between certain points, curves or surfaces on the neuroimages. - Not fully automatic. - Transformation of non-landmark points is interpolated from the transformation of landmark points.

5. Landmark based registration

6. Existing approaches Registration driven by a similarity measure. Deformation model is parameterized and then a numerical optimization procedure is used to find parameters that maximize some similarity measure. - Automatic, but prone to local maxima. - The more degrees of freedom deformation model has, the harder it is to find optimal parameters for it.

7. Affine registration driven by sum of square differences of the images SSD: 0.0085 Sometimes it works

8. SSD: 0.0063 Affine registration driven by sum of square differences of the images Sometimes it works

9. SSD: 0.0289 Affine registration driven by sum of square differences of the images Sometimes it works

10. SSD: 0.0188 Affine registration driven by sum of square differences of the images Sometimes it does not

11. Existing approaches Feature based registration. A feature vector is computed for each voxel. Correspondences between voxels in the reference image and voxels in the input image are estimated based on the simliarity of their feature vectors. - Best results among existing methods. - Existing systems have many hand tuned parameters, including components of feature vectors.

12. Feature based registration

14. Our goals. Fully automatic method that selects which features to use depending on - modality of the images; - anatomical structures we care to register the most No restriction on the degrees of freedom of the deformation model

15. A few questions… Would it be a hard task to register these two images for a human?

16. Not a hard task A few questions…

17. OK, then how about registering this image with a rotated copy of itself? A few questions…

18. OK, then how about registering this image with a rotated copy of itself? Looks much harder We can get some idea about how difficult a registration task will be even without seeing the other image! A few questions…

19. Not an easy task indeed A few questions…

20. If there were some points that “stood out”, we could easily find what the rotation was… A few questions…

21. If there were some points that “stood out”, we could easily find what the rotation was… provided we can determine correspondences correctly. A few questions…

22. We can solve both problems using the same mechanism We are facing two different problems: How to find interesting points in the reference image automatically. How to find corresponding points in the input image.

23. Feature Extraction. … We compute various rotationally invariant features at different scales.

24. [Feature Vector] h(Pi|F) Probability that given feature vector “belongs” to a certain pixel in the reference image A pixel in the input image Most likely correspondences in the reference image If we knew h(Pi|F) we could do this :

25. We could find h(Pi|F) … …if we knew what g(F|Pi) and q(Pi) were.  Probability of observing feature F at the pixel Pi Prior

26. Prior q(Pi) if we have a reason to believe that certain pixels in the reference image are more likely to correspond to the given pixel in the input image, we can express our beliefs through prior. we will use uniform prior for now.

27. We can estimate g(F|Pi)! We have applied about 1560 affine transforms to the reference image and computed features for each pixels in each of the resulting 1560 images. Thus we obtain 1560 feature vectors for each anatomical location in the reference image. We assume that components of feature vector are independent of each other and distributed according to a gaussian distribution. We find MLE of mean and variance for each gaussian g(F|Pi) is a product of these gaussians.

28. [Feature Vector] h(Pi|F) A pixel in the input image Blue dots represent correspondences with probability of 0.4999. Probability of other correspondences is negligibly small We are almost done; we need to have a way of distinguishing good correspondences …

29. [Feature Vector] h(Pi|F) A pixel in the input image Blue dots represent correspondences with probability of 0.4999. Probability of other correspondences is negligibly small … from bad correspondences

30. Risk=∑h(Pi|F)L(Pi, Po) L(Pi, Po) – loss, which is a geometric distance between estimated corresponding pixel Pi and the correct corresponding pixel Po. When Po is unknown, we use MAP estimate of Po instead. Correspondences with low risk are “good” correspondences Risk

31. Where are we now? We can find correspondences between pixels in the input image and pixels in the reference image using feature vectors computed on the pixels of the input image. And we can also determine interesting points by finding correspondences between pixels in the reference image and pixels in the … reference image!

32. Feature Selection Select feature subset for determining interesting points. Select feature subset for determining correspondences. Use sum of square differences (quality of registration) as a means of evaluating feature subsets.

33. Interesting point feature subset Reference image Input image Correspondence feature subset Interesting points CorrespondencesRANSAC h(Pi|F) Affine Transform TPS transform Driving voxels Registration quality Birds eye view. feedback Feature pool

34. Experimental results. Reference Image Input Image

35. Interesting points

36. Best correspondences

37. Driving voxels

38. Reference imageRegistered input image Registration results SSD: 0.0047

39. Difference image

40. More experiments 1.Select random set of interest points; select random subset of features to find correspondences 2.Select random set of interest points; use forward feature selection to find subset of features to be used for estimating correspondences 3.Select random subset of features to find interest points; select random subset of features to find correspondences

41. More experiments 4. Select random subset of features to find interesting points; use forward selection to choose subset of features used for determining the correspondences. 5. Select random subset of features to find interesting points; use forward selection to choose subset of features used for determining the correspondences. This time start from the subset used to find interesting points without one feature.

42. More experiments 6. Select random subset of features to find interesting points. Then employ forward selection for choosing subset of features to be used for determining the correspondences. Find a new set of interest points using this subset of features and iterate. 7. Select random subset of features to find interesting points. Then employ forward selection for choosing subset of features to be used for determining the correspondences. This time start from the subset used to find interesting points without one feature. Find a new set of interest points using this selected subset of features and iterate

43. More experiments. For each feature selection strategy we run registration eight times, each time restarting at a random point. Each run continues for 20 iterations.

44. Feature pool. 22 features, all at the finest scale (for now). 4. First derivative (D1) 9. Second derivative (D2) 14. Third derivative (D3) 19. Fourth derivative (D4) 24. Fifth derivative (D5) 29. Gabor_0_3 (G1) 34. Gabor_0_5 (G2) 39. Gabor_2_7 (G3) 44. Gabor_3_7 (G4) 49. Gabor_4_9 (G5) 54. Laplacian (L) 59. Harris (H) 64. Intensity_1_mean (M1) 69. Intensity_1_std(S1) 74. Intensity_2_mean(M2) 79. Intensity_2_std(S2) 84. Intensity_4_mean(M3) 89. Intensity_4_std(S3) 94. Intensity_8_mean(M4) 99. Intensity_8_std(S4) 104. Intensity_16_mean(M5) 109. Intensity_16_std(S5) “Intensity_n_mean” is the mean of the pixel intensities inside a ring with inner radius log 2 (n) and outer radius n, centered at the given pixel. “Intensity_n_std” is the standard deviation of the pixel intensities inside a ring with inner radius log 2 (n) and outer radius n, centered at the given pixel.

45. Feature selection significantly improves registration results. Here, a 100 interest points were used

46. Random IPRandom edge IPIP selection Registration error when 30 interesting points are used. Interesting points are selected 1) at random from all the image pixels, 2) at random from image pixels that lie on edges, 3) using interesting point selection; Interest point selection has greater positive effect on the registration accuracy when number of interesting points is decreased.

47. Typical graph for the case when feature selection strategy number 7 is used. Brown line shows registration error if affine deformation is used, green line – when thin plate spline deformation is used.

48. Feature pool consists of 22 features. 8 features appear to be enough for good registration results.

49. Histogram of selected interesting point features when feature selection strategy number 7 is used

50. Histogram of selected correspondence features when feature selection strategy number 7 is used

51. Histogram of selected interesting point features when feature selection strategy number 6 is used

52. Histogram of selected correspondence features when feature selection strategy number 6 is used

53. Reference slice Input slice Reference slice and input slice are midsagittal slices of neuroimages of different subjects. In addition, input slice was affinely transformed. Registration results at each step of feature subset selection

54. {M4}{M4, S1}{M4, S1, S5} {M4, S1, S5, M5} {M4, S1, S5, M5, H} {M4, S1, S5, M5, H, G2} SSD: 0.04719 SSD: 0.04043SSD: 0.01954 SSD: 0.01644SSD: 0.01777SSD: 0.01625

55. Thank you