2. CSci 6971: Image Registration Lecture 16: View-Based Registration March 16, 2004 Prof. Chuck Stewart, RPI Dr. Luis Ibanez, Kitware Prof. Chuck Stewart, RPI Dr. Luis Ibanez, Kitware

3. Image RegistrationLecture 16 2 Overview  Retinal image registration  The Dual-Bootstrap ICP algorithm  Covariance matrix  Covariance propagation  Model selection  View-based registration  Software design  Warning: mathematically, this lecture is a little “rich.” You will not be responsible for knowing the details  Retinal image registration  The Dual-Bootstrap ICP algorithm  Covariance matrix  Covariance propagation  Model selection  View-based registration  Software design  Warning: mathematically, this lecture is a little “rich.” You will not be responsible for knowing the details

4. Image RegistrationLecture 16 3 Retinal Image Registration: Applications  Mosaics  Multimodal integration  Blood flow animation  Change detection  Mosaics  Multimodal integration  Blood flow animation  Change detection

5. Image RegistrationLecture 16 4 Mosaics

6. Image RegistrationLecture 16 5 Multimodal Integration

7. Image RegistrationLecture 16 6 Fluorescein Angiogram Animation

8. Image RegistrationLecture 16 7 Change Visualization

9. Image RegistrationLecture 16 8 Retinal Image Registration - Preliminaries  Features  Transformation models  Initialization  Features  Transformation models  Initialization

10. Image RegistrationLecture 16 9 landmarks vascular centerlines Features  Vascular centerline points  Discrete locations along the vessel contours  Described in terms of pixel locations, orientations, and widths  Vascular landmarks  Pixel locations, orientations and width of vessels that meet to form landmarks  Vascular centerline points  Discrete locations along the vessel contours  Described in terms of pixel locations, orientations, and widths  Vascular landmarks  Pixel locations, orientations and width of vessels that meet to form landmarks

11. Image RegistrationLecture 16 10 Transformation Models ModelParameter MatrixDoFAccuracy (pixels) Similarity45.05 Affine64.58 Reduced quadratic 62.41 Full quadratic120.64

12. Image RegistrationLecture 16 11 Initializing Registration  Form list of landmarks in each image  Form matches of one landmark from each image  The selection of these matches will be discussed in Lectures 18 and 19  Choose matches, one at a time  For each match:  Compute an initial similarity transformation in the small image region surrounding the landmarks  Apply Dual-Bootstrap ICP procedure to see if the initial alignment can be successfully grown into an accurate, image- wide alignment  End when one match leads to success, or all matches are exhausted  Form list of landmarks in each image  Form matches of one landmark from each image  The selection of these matches will be discussed in Lectures 18 and 19  Choose matches, one at a time  For each match:  Compute an initial similarity transformation in the small image region surrounding the landmarks  Apply Dual-Bootstrap ICP procedure to see if the initial alignment can be successfully grown into an accurate, image- wide alignment  End when one match leads to success, or all matches are exhausted

13. Image RegistrationLecture 16 12 Dual-Bootstrap - Overview  Match and refine estimate in each region  Bootstrap the model:  Low-order for small regions;  High-order for large  Automatic selection  Bootstrap the region:  Covariance propagation gives uncertainty Iterate until convergence

14. Image RegistrationLecture 16 13 Matching and Estimation in Each Region  Matching - standard stuff:  Vascular centerline points from within current region of moving image  Mapped using current transform estimate  Find closest point using Borgefors digital distance map  Estimation:  Fix scale estimate  Run IRLS  Matching - standard stuff:  Vascular centerline points from within current region of moving image  Mapped using current transform estimate  Find closest point using Borgefors digital distance map  Estimation:  Fix scale estimate  Run IRLS

15. Image RegistrationLecture 16 14 Covariance Matrix of Estimate  Measures uncertainty in estimate of transformation parameters  Basis for region growth and model selection  The next few slides will give an overview of computing an approximate covariance matrix  We’ll start with linear regression  Measures uncertainty in estimate of transformation parameters  Basis for region growth and model selection  The next few slides will give an overview of computing an approximate covariance matrix  We’ll start with linear regression

16. Image RegistrationLecture 16 15 Problem Formulation in Linear Regression  Independent (non-random) variable values:  Dependent (random) variable values  Linear relationship based on k+1 dimensional parameter vector a:  Independent (non-random) variable values:  Dependent (random) variable values  Linear relationship based on k+1 dimensional parameter vector a:

17. Image RegistrationLecture 16 16 Least-Squares Formulation  Least-squares error term:  Here:  Least-squares error term:  Here:

18. Image RegistrationLecture 16 17 Estimate and Covariance Matrix  Estimate:  Residual error variance (square of “scale”):  Parameter estimate covariance  Estimate:  Residual error variance (square of “scale”):  Parameter estimate covariance

19. Image RegistrationLecture 16 18 Aside: Line Fitting in 2D  Form of the equation:  If the x i values are centered:  Then the parameters are independent with variances for the linear and constant terms, respectively  Form of the equation:  If the x i values are centered:  Then the parameters are independent with variances for the linear and constant terms, respectively

20. Image RegistrationLecture 16 19 Hessians and Covariances  Back to k dimensions, re-consider the objective function:  Compute the Hessian matrix:  Observe the relationship  Back to k dimensions, re-consider the objective function:  Compute the Hessian matrix:  Observe the relationship

21. Image RegistrationLecture 16 20 Hessians and Covariances  This is exact for linear regression, but serves as a good approximation for non-linear least-squares  In general, the Hessian will depend on the estimate (in regression it doesn’t because the problem is quadratic), so the approximate relationship is  This is exact for linear regression, but serves as a good approximation for non-linear least-squares  In general, the Hessian will depend on the estimate (in regression it doesn’t because the problem is quadratic), so the approximate relationship is

22. Image RegistrationLecture 16 21 Hessian in Registration  Recall the weighted least-squares objective function:  Keeping the correspondences and the weights fixed, where D k gives the error of the k-th correspondence  Inverting this gives the covariance approximation.  This approximation is only good when the estimate is fairly accurate  Recall the weighted least-squares objective function:  Keeping the correspondences and the weights fixed, where D k gives the error of the k-th correspondence  Inverting this gives the covariance approximation.  This approximation is only good when the estimate is fairly accurate

23. Image RegistrationLecture 16 22 Back to Dual-Bootstrap ICP  Covariance is used in two ways in each DB-ICP iteration:  Determining the region incorporates enough constraints to switch to a more complex model  Similarity => Affine => Reduced Quadratic => Quadratic  Determining the growth of the dual-bootstrap region:  More stable transformation estimates lead to faster growth  Covariance is used in two ways in each DB-ICP iteration:  Determining the region incorporates enough constraints to switch to a more complex model  Similarity => Affine => Reduced Quadratic => Quadratic  Determining the growth of the dual-bootstrap region:  More stable transformation estimates lead to faster growth

24. Image RegistrationLecture 16 23 Model Selection  What model should be used to describe a given set of data?  Classic problem in statistics, and many methods have been proposed  Most trade-off the fitting accuracy of higher-order models with the stability (or lower complexity) of lower-order models  What model should be used to describe a given set of data?  Classic problem in statistics, and many methods have been proposed  Most trade-off the fitting accuracy of higher-order models with the stability (or lower complexity) of lower-order models

25. Image RegistrationLecture 16 24 Model Selection in DB-ICP  Use correspondence set  Estimate the IRLS parameters and covariance matrices for each model in current set  For each model (with d m parameters) this generates a set of weights and errors and a covariance matrix:  Choose the model maximizing the model selection equation (derived from Bayesian modeling):  The first two terms increase with increasingly complex models; the last term decreases  Use correspondence set  Estimate the IRLS parameters and covariance matrices for each model in current set  For each model (with d m parameters) this generates a set of weights and errors and a covariance matrix:  Choose the model maximizing the model selection equation (derived from Bayesian modeling):  The first two terms increase with increasingly complex models; the last term decreases

26. Image RegistrationLecture 16 25 Region Growth in DB-ICP  Grow each side independently  Grow is inversely proportional to uncertainty in mapping of boundary point on the center of each side  New rectangular region found from the new positions of each of the boundary points  Grow each side independently  Grow is inversely proportional to uncertainty in mapping of boundary point on the center of each side  New rectangular region found from the new positions of each of the boundary points

27. Image RegistrationLecture 16 26 Aside: Covariance Propagation and Transfer Error  Given mapping function:  We will treat  as a random variable, but not g k  Uncertainty in  makes g k ’ a random variable.  What then is the covariance of g k ’?  We solve this using standard covariance propagation techniques:  Compute the Jacobian of the transformation, evaluated at g k :  Pre- and post-multiply to obtain the covariance of g k ’  In computer vision, this is called the “transfer error”  Given mapping function:  We will treat  as a random variable, but not g k  Uncertainty in  makes g k ’ a random variable.  What then is the covariance of g k ’?  We solve this using standard covariance propagation techniques:  Compute the Jacobian of the transformation, evaluated at g k :  Pre- and post-multiply to obtain the covariance of g k ’  In computer vision, this is called the “transfer error”

28. Image RegistrationLecture 16 27 Outward Growth of a Side  Let  k be the outward normal of the side, and let r k be the distance of the side from the center of the region  Project the transfer error covariance onto  k to obtain a scalar variance  k  The outward growth (along normal  k ) is  where  controls the maximum growth rate, which occurs when  k < 1  Let  k be the outward normal of the side, and let r k be the distance of the side from the center of the region  Project the transfer error covariance onto  k to obtain a scalar variance  k  The outward growth (along normal  k ) is  where  controls the maximum growth rate, which occurs when  k < 1

29. Image RegistrationLecture 16 28 Putting It All Together - The Example, Revisited

30. Image RegistrationLecture 16 29 Turning to the Software  A “view” is a definition or snapshot of the registration problem.  A “view” contains:  An image region (current region, plus goal region)  A current transformation estimate and estimator  A current stage (resolution) of registration  Views work in conjunction with multistage registration  A “view” is a definition or snapshot of the registration problem.  A “view” contains:  An image region (current region, plus goal region)  A current transformation estimate and estimator  A current stage (resolution) of registration  Views work in conjunction with multistage registration

31. Image RegistrationLecture 16 30 View-Based Registration - Procedural  The following is repeated for each initial estimate  For each stage:  Do  Match  Compute weights  Estimate scale  For each model  Run IRLS to estimate parameters and covariances  Re-estimate scale  Generate next view  For DB-ICP view generator this choose the best model and grows the region  Until region has converged and highest order model used  Prepare for next stage  The following is repeated for each initial estimate  For each stage:  Do  Match  Compute weights  Estimate scale  For each model  Run IRLS to estimate parameters and covariances  Re-estimate scale  Generate next view  For DB-ICP view generator this choose the best model and grows the region  Until region has converged and highest order model used  Prepare for next stage

32. Image RegistrationLecture 16 31 Implementation  rgrl_view  Store the information about the view  rgrl_view_generator  Generate the next view  rgrl_view_based_registration  Mirrors rgrl_feature_based_registration with modifications based on the outline on previous slide  Example  rgrl/example/registration_retina.cxx  rgrl_view  Store the information about the view  rgrl_view_generator  Generate the next view  rgrl_view_based_registration  Mirrors rgrl_feature_based_registration with modifications based on the outline on previous slide  Example  rgrl/example/registration_retina.cxx

33. Image RegistrationLecture 16 32 Summary  Retina registration:  Models, features and initialization  DB-ICP:  Matching, estimation and covariances  Model selection  Region growing  Generalization to view-based registration and its implementation in the toolkit.  Retina registration:  Models, features and initialization  DB-ICP:  Matching, estimation and covariances  Model selection  Region growing  Generalization to view-based registration and its implementation in the toolkit.

34. Image RegistrationLecture 16 33 Looking Ahead to Lecture 17  Discussion of toolkits:  What’s easy, what’s hard  What’s missing  Project discussion  Requirements  Topics and initial steps  Discussion of toolkits:  What’s easy, what’s hard  What’s missing  Project discussion  Requirements  Topics and initial steps