1. CSci 6971: Image Registration Lecture 5: Feature-Base Regisration January 27, 2004 Prof. Chuck Stewart, RPI Dr. Luis Ibanez, Kitware Prof. Chuck Stewart, RPI Dr. Luis Ibanez, Kitware

2. Image RegistrationLecture 5 2 Overview  What is feature-based (point-based) registration?  Feature points  The correspondence problem  Solving for the transformation estimate  Putting it all together: ICP  Discussion and conclusion  What is feature-based (point-based) registration?  Feature points  The correspondence problem  Solving for the transformation estimate  Putting it all together: ICP  Discussion and conclusion

3. Image RegistrationLecture 5 3 What is Feature-Based Registration?  Images are described as discrete sets of point locations associated with a geometric measurement  Locations may have additional properties such as intensities and orientations  Registration problem involves two parts:  Finding correspondences between features  Estimating the transformation parameters based on these correspondences  Images are described as discrete sets of point locations associated with a geometric measurement  Locations may have additional properties such as intensities and orientations  Registration problem involves two parts:  Finding correspondences between features  Estimating the transformation parameters based on these correspondences

4. Image RegistrationLecture 5 4 Feature Examples: Range Data  Range image points:  (x,y,z) values  Triangulated mesh  Surface normals are sometimes computed  Notice:  Some information (locations) is determined directly by the sensor (“raw data”)  Some information is inferred from the data  Range image points:  (x,y,z) values  Triangulated mesh  Surface normals are sometimes computed  Notice:  Some information (locations) is determined directly by the sensor (“raw data”)  Some information is inferred from the data

5. Image RegistrationLecture 5 5 Feature Examples: Vascular Landmarks  Branching points pulmonary images:  Lung vessels  Airway branches  Retinal image branches and cross-over points  Typically augmented (at least) with orientations of vessels meeting to form landmarks  Branching points pulmonary images:  Lung vessels  Airway branches  Retinal image branches and cross-over points  Typically augmented (at least) with orientations of vessels meeting to form landmarks

6. Image RegistrationLecture 5 6 Points Along Centers of Vessels and Airways  Airways and vessels modeled as tubular structures  Sample points spaced along center of tubes  Note that the entire tube is rarely used as a unit  Augmented descriptions:  Orientation  Radius  Airways and vessels modeled as tubular structures  Sample points spaced along center of tubes  Note that the entire tube is rarely used as a unit  Augmented descriptions:  Orientation  Radius

7. Image RegistrationLecture 5 7 “Interest” Points  Locations of strong intensity variation in all directions  Augmented with summary descriptions (moments) of surrounding intensity structures  Recent work in making these invariant to viewpoint and illumination.  We’ll discuss interest points during Lectures 16 and 17  Locations of strong intensity variation in all directions  Augmented with summary descriptions (moments) of surrounding intensity structures  Recent work in making these invariant to viewpoint and illumination.  We’ll discuss interest points during Lectures 16 and 17 Brown and Lowe, Int. Conf. On Computer Vision, 2003

8. Image RegistrationLecture 5 8 Feature Points: Discussion  Many different possible features  Problem is reliably extracting features in all images  This is why more sophisticated features are not used  Feature extraction methods do not use all intensity values  Use of features dominates range-image registration techniques where “features” are provided by the sensor  Many different possible features  Problem is reliably extracting features in all images  This is why more sophisticated features are not used  Feature extraction methods do not use all intensity values  Use of features dominates range-image registration techniques where “features” are provided by the sensor

9. Image RegistrationLecture 5 9 Preamble to Feature-Based Registration: Notation  Set of moving image features  Set of fixed image features  Each feature must include a point location in the coordinate system of its image. It may include more  Set of correspondences  Set of moving image features  Set of fixed image features  Each feature must include a point location in the coordinate system of its image. It may include more  Set of correspondences

10. Image RegistrationLecture 5 10  Error objective function depends on unknown transformation parameters and unknown feature correspondences  Each may depend on the other!  Transformation may include mapping of more than just locations  Distance function, D, could be as simple as the Euclidean distance between location vectors.  We are using the forward transformation model.  Error objective function depends on unknown transformation parameters and unknown feature correspondences  Each may depend on the other!  Transformation may include mapping of more than just locations  Distance function, D, could be as simple as the Euclidean distance between location vectors.  We are using the forward transformation model. Mathematical Formulation

11. Image RegistrationLecture 5 11 Correspondence Problem  Determine correspondences before estimating transformation parameters  Based on rich description of features  Error prone  Determine correspondences at the same time as estimation of parameters  “Chicken-and-egg” problem  For the next few minutes we will assume a set of correspondences is given and proceed to the estimation of parameters  Then we will return to the correspondence problem  Determine correspondences before estimating transformation parameters  Based on rich description of features  Error prone  Determine correspondences at the same time as estimation of parameters  “Chicken-and-egg” problem  For the next few minutes we will assume a set of correspondences is given and proceed to the estimation of parameters  Then we will return to the correspondence problem

12. Image RegistrationLecture 5 12 Example: Estimating Parameters  2d point locations:  Similarity transformation:  Euclidean distance:  2d point locations:  Similarity transformation:  Euclidean distance:

13. Image RegistrationLecture 5 13 Putting This Together

14. Image RegistrationLecture 5 14 What Do We Have?  Least-squares objective function  Quadratic function of each parameter  We can  Take the derivative with respect to each parameter  Set the resulting gradient to 0 (vector)  Solve for the parameters through matrix inversion  We’ll do this in two forms: component and matrix/vector  Least-squares objective function  Quadratic function of each parameter  We can  Take the derivative with respect to each parameter  Set the resulting gradient to 0 (vector)  Solve for the parameters through matrix inversion  We’ll do this in two forms: component and matrix/vector

15. Image RegistrationLecture 5 15 Component Derivative (a)

16. Image RegistrationLecture 5 16 Component Derivative (b) At this point, we’ve dropped the leading factor of 2. It will be eliminated when this is set to 0.

17. Image RegistrationLecture 5 17 Component Derivatives t x and t y

18. Image RegistrationLecture 5 18 Gathering  Setting each of these equal to 0 we obtain a set of 4 linear equations in 4 unknowns. Gathering into a matrix we have:

19. Image RegistrationLecture 5 19 Solving  This is a simple equation of the form  Provided the 4x4 matrix X is full-rank (evaluate SVD) we easily solve as  This is a simple equation of the form  Provided the 4x4 matrix X is full-rank (evaluate SVD) we easily solve as

20. Image RegistrationLecture 5 20 Matrix Version  We can do this in a less painful way by rewriting the following intermediate expression in terms of vectors and matrices:

21. Image RegistrationLecture 5 21 Matrix Version (continued)  This becomes  Manipulating:  This becomes  Manipulating:

22. Image RegistrationLecture 5 22 Matrix Version (continued)  Taking the derivative of this wrt the transformation parameters (we didn’t cover vector derivatives, but this is fairly straightforward):  Setting this equal to 0 and solving yields:  Taking the derivative of this wrt the transformation parameters (we didn’t cover vector derivatives, but this is fairly straightforward):  Setting this equal to 0 and solving yields:

23. Image RegistrationLecture 5 23 Comparing the Two Versions  Final equations are identical (if you expand the symbols)  Matrix version is easier (once you have practice) and less error prone  Sometimes efficiency requires hand- calculation and coding of individual terms  Final equations are identical (if you expand the symbols)  Matrix version is easier (once you have practice) and less error prone  Sometimes efficiency requires hand- calculation and coding of individual terms

24. Image RegistrationLecture 5 24 Resetting the Stage  What we have done:  Features  Error function of transformation parameters and correspondences  Least-squares estimate of transformation parameters for fixed set of correspondences  Next:  ICP: joint estimation of correspondences and parameters  What we have done:  Features  Error function of transformation parameters and correspondences  Least-squares estimate of transformation parameters for fixed set of correspondences  Next:  ICP: joint estimation of correspondences and parameters

25. Image RegistrationLecture 5 25 Iterative Closest Points (ICP) Algorithm  Given an initial transformation estimate  0  t = 0  Iterate until convergence:  Establish correspondences:  For fixed transformation parameter estimate,  t, apply the transformation to each moving image feature and find the closest fixed image feature  Estimate the new transformation parameters,  For the resulting correspondences, estimate  t+1  Given an initial transformation estimate  0  t = 0  Iterate until convergence:  Establish correspondences:  For fixed transformation parameter estimate,  t, apply the transformation to each moving image feature and find the closest fixed image feature  Estimate the new transformation parameters,  For the resulting correspondences, estimate  t+1 ICP algorithm was developed almost simultaneous by at least 5 research groups in the early 1990’s.

26. Image RegistrationLecture 5 26 Finding Correspondences  Map feature into coordinate system of I f  Find closest point  Map feature into coordinate system of I f  Find closest point

27. Image RegistrationLecture 5 27 Finding Correspondences (continued)  Enforce unique correspondences  Avoid trivial minima of objective function due to having no correspondences  Spatial data structures needed to make search for correspondences efficient  K-d trees  Digital distance maps  More during lectures 11-15…  Enforce unique correspondences  Avoid trivial minima of objective function due to having no correspondences  Spatial data structures needed to make search for correspondences efficient  K-d trees  Digital distance maps  More during lectures 11-15…

28. Image RegistrationLecture 5 28 Initialization and Convergence  Initial estimate of transformation is again crucial because this is a minimization technique  Determining correspondences and estimating the transformation parameters are two separate processes  With Euclidean distance metrics you can show they are working toward the same minimum  In general this is not true  Convergence in practice is sometimes problematic and the correspondences oscillate between points.  Initial estimate of transformation is again crucial because this is a minimization technique  Determining correspondences and estimating the transformation parameters are two separate processes  With Euclidean distance metrics you can show they are working toward the same minimum  In general this is not true  Convergence in practice is sometimes problematic and the correspondences oscillate between points.

29. Image RegistrationLecture 5 29 2d Retinal Example  White = vessel centerline points from one image  Black = vessel centerline points from second image  Yellow line segments drawn between corresponding points  Because of the complexity of the structure, initialization must be fairly accurate  White = vessel centerline points from one image  Black = vessel centerline points from second image  Yellow line segments drawn between corresponding points  Because of the complexity of the structure, initialization must be fairly accurate

30. Image RegistrationLecture 5 30 Comparison  For a given transformation estimate, we can only find a new, better estimate, not the best estimate, based on the gradient step.  We then need to update the constraints and re- estimate  For a given transformation estimate, we can only find a new, better estimate, not the best estimate, based on the gradient step.  We then need to update the constraints and re- estimate Intensity-BasedFeature-Based  For given set of correspondences, we can directly (least- squares) estimate the best transformation  BUT, the transformation depends on the correspondences, so we generally need to re- establish the correspondences.

31. Image RegistrationLecture 5 31 Summary  Feature-based registration  Feature types and properties  Correspondences  Least-squares estimate of parameters based on correspondences  ICP  Comparison  Feature-based registration  Feature types and properties  Correspondences  Least-squares estimate of parameters based on correspondences  ICP  Comparison

32. Image RegistrationLecture 5 32 Looking Ahead to Lecture 6  Introduction to ITK and the ITK registration framework.