CSci 6971: Image Registration Lecture 16: View-Based Registration March 16, 2004 Prof. Chuck Stewart, RPI Dr. Luis Ibanez, Kitware Prof. Chuck Stewart, RPI Dr. Luis Ibanez, Kitware

Image RegistrationLecture 16 2 Overview  Retinal image registration  The Dual-Bootstrap ICP algorithm  Covariance matrix  Covariance propagation  Model selection  View-based registration  Software design  Warning: mathematically, this lecture is a little “rich.” You will not be responsible for knowing the details  Retinal image registration  The Dual-Bootstrap ICP algorithm  Covariance matrix  Covariance propagation  Model selection  View-based registration  Software design  Warning: mathematically, this lecture is a little “rich.” You will not be responsible for knowing the details

Image RegistrationLecture 16 3 Retinal Image Registration: Applications  Mosaics  Multimodal integration  Blood flow animation  Change detection  Mosaics  Multimodal integration  Blood flow animation  Change detection

Image RegistrationLecture 16 4 Mosaics

Image RegistrationLecture 16 5 Multimodal Integration

Image RegistrationLecture 16 6 Fluorescein Angiogram Animation

Image RegistrationLecture 16 7 Change Visualization

Image RegistrationLecture 16 8 Retinal Image Registration - Preliminaries  Features  Transformation models  Initialization  Features  Transformation models  Initialization

Image RegistrationLecture 16 9 landmarks vascular centerlines Features  Vascular centerline points  Discrete locations along the vessel contours  Described in terms of pixel locations, orientations, and widths  Vascular landmarks  Pixel locations, orientations and width of vessels that meet to form landmarks  Vascular centerline points  Discrete locations along the vessel contours  Described in terms of pixel locations, orientations, and widths  Vascular landmarks  Pixel locations, orientations and width of vessels that meet to form landmarks

Image RegistrationLecture Transformation Models ModelParameter MatrixDoFAccuracy (pixels) Similarity45.05 Affine64.58 Reduced quadratic Full quadratic120.64

Image RegistrationLecture Initializing Registration  Form list of landmarks in each image  Form matches of one landmark from each image  The selection of these matches will be discussed in Lectures 18 and 19  Choose matches, one at a time  For each match:  Compute an initial similarity transformation in the small image region surrounding the landmarks  Apply Dual-Bootstrap ICP procedure to see if the initial alignment can be successfully grown into an accurate, image- wide alignment  End when one match leads to success, or all matches are exhausted  Form list of landmarks in each image  Form matches of one landmark from each image  The selection of these matches will be discussed in Lectures 18 and 19  Choose matches, one at a time  For each match:  Compute an initial similarity transformation in the small image region surrounding the landmarks  Apply Dual-Bootstrap ICP procedure to see if the initial alignment can be successfully grown into an accurate, image- wide alignment  End when one match leads to success, or all matches are exhausted

Image RegistrationLecture Dual-Bootstrap - Overview  Match and refine estimate in each region  Bootstrap the model:  Low-order for small regions;  High-order for large  Automatic selection  Bootstrap the region:  Covariance propagation gives uncertainty Iterate until convergence

Image RegistrationLecture Matching and Estimation in Each Region  Matching - standard stuff:  Vascular centerline points from within current region of moving image  Mapped using current transform estimate  Find closest point using Borgefors digital distance map  Estimation:  Fix scale estimate  Run IRLS  Matching - standard stuff:  Vascular centerline points from within current region of moving image  Mapped using current transform estimate  Find closest point using Borgefors digital distance map  Estimation:  Fix scale estimate  Run IRLS

Image RegistrationLecture Covariance Matrix of Estimate  Measures uncertainty in estimate of transformation parameters  Basis for region growth and model selection  The next few slides will give an overview of computing an approximate covariance matrix  We’ll start with linear regression  Measures uncertainty in estimate of transformation parameters  Basis for region growth and model selection  The next few slides will give an overview of computing an approximate covariance matrix  We’ll start with linear regression

Image RegistrationLecture Problem Formulation in Linear Regression  Independent (non-random) variable values:  Dependent (random) variable values  Linear relationship based on k+1 dimensional parameter vector a:  Independent (non-random) variable values:  Dependent (random) variable values  Linear relationship based on k+1 dimensional parameter vector a:

Image RegistrationLecture Least-Squares Formulation  Least-squares error term:  Here:  Least-squares error term:  Here:

Image RegistrationLecture Estimate and Covariance Matrix  Estimate:  Residual error variance (square of “scale”):  Parameter estimate covariance  Estimate:  Residual error variance (square of “scale”):  Parameter estimate covariance

Image RegistrationLecture Aside: Line Fitting in 2D  Form of the equation:  If the x i values are centered:  Then the parameters are independent with variances for the linear and constant terms, respectively  Form of the equation:  If the x i values are centered:  Then the parameters are independent with variances for the linear and constant terms, respectively

Image RegistrationLecture Hessians and Covariances  Back to k dimensions, re-consider the objective function:  Compute the Hessian matrix:  Observe the relationship  Back to k dimensions, re-consider the objective function:  Compute the Hessian matrix:  Observe the relationship

Image RegistrationLecture Hessians and Covariances  This is exact for linear regression, but serves as a good approximation for non-linear least-squares  In general, the Hessian will depend on the estimate (in regression it doesn’t because the problem is quadratic), so the approximate relationship is  This is exact for linear regression, but serves as a good approximation for non-linear least-squares  In general, the Hessian will depend on the estimate (in regression it doesn’t because the problem is quadratic), so the approximate relationship is

Image RegistrationLecture Hessian in Registration  Recall the weighted least-squares objective function:  Keeping the correspondences and the weights fixed, where D k gives the error of the k-th correspondence  Inverting this gives the covariance approximation.  This approximation is only good when the estimate is fairly accurate  Recall the weighted least-squares objective function:  Keeping the correspondences and the weights fixed, where D k gives the error of the k-th correspondence  Inverting this gives the covariance approximation.  This approximation is only good when the estimate is fairly accurate

Image RegistrationLecture Back to Dual-Bootstrap ICP  Covariance is used in two ways in each DB-ICP iteration:  Determining the region incorporates enough constraints to switch to a more complex model  Similarity => Affine => Reduced Quadratic => Quadratic  Determining the growth of the dual-bootstrap region:  More stable transformation estimates lead to faster growth  Covariance is used in two ways in each DB-ICP iteration:  Determining the region incorporates enough constraints to switch to a more complex model  Similarity => Affine => Reduced Quadratic => Quadratic  Determining the growth of the dual-bootstrap region:  More stable transformation estimates lead to faster growth

Image RegistrationLecture Model Selection  What model should be used to describe a given set of data?  Classic problem in statistics, and many methods have been proposed  Most trade-off the fitting accuracy of higher-order models with the stability (or lower complexity) of lower-order models  What model should be used to describe a given set of data?  Classic problem in statistics, and many methods have been proposed  Most trade-off the fitting accuracy of higher-order models with the stability (or lower complexity) of lower-order models

Image RegistrationLecture Model Selection in DB-ICP  Use correspondence set  Estimate the IRLS parameters and covariance matrices for each model in current set  For each model (with d m parameters) this generates a set of weights and errors and a covariance matrix:  Choose the model maximizing the model selection equation (derived from Bayesian modeling):  The first two terms increase with increasingly complex models; the last term decreases  Use correspondence set  Estimate the IRLS parameters and covariance matrices for each model in current set  For each model (with d m parameters) this generates a set of weights and errors and a covariance matrix:  Choose the model maximizing the model selection equation (derived from Bayesian modeling):  The first two terms increase with increasingly complex models; the last term decreases

Image RegistrationLecture Region Growth in DB-ICP  Grow each side independently  Grow is inversely proportional to uncertainty in mapping of boundary point on the center of each side  New rectangular region found from the new positions of each of the boundary points  Grow each side independently  Grow is inversely proportional to uncertainty in mapping of boundary point on the center of each side  New rectangular region found from the new positions of each of the boundary points

Image RegistrationLecture Aside: Covariance Propagation and Transfer Error  Given mapping function:  We will treat  as a random variable, but not g k  Uncertainty in  makes g k ’ a random variable.  What then is the covariance of g k ’?  We solve this using standard covariance propagation techniques:  Compute the Jacobian of the transformation, evaluated at g k :  Pre- and post-multiply to obtain the covariance of g k ’  In computer vision, this is called the “transfer error”  Given mapping function:  We will treat  as a random variable, but not g k  Uncertainty in  makes g k ’ a random variable.  What then is the covariance of g k ’?  We solve this using standard covariance propagation techniques:  Compute the Jacobian of the transformation, evaluated at g k :  Pre- and post-multiply to obtain the covariance of g k ’  In computer vision, this is called the “transfer error”

Image RegistrationLecture Outward Growth of a Side  Let  k be the outward normal of the side, and let r k be the distance of the side from the center of the region  Project the transfer error covariance onto  k to obtain a scalar variance  k  The outward growth (along normal  k ) is  where  controls the maximum growth rate, which occurs when  k < 1  Let  k be the outward normal of the side, and let r k be the distance of the side from the center of the region  Project the transfer error covariance onto  k to obtain a scalar variance  k  The outward growth (along normal  k ) is  where  controls the maximum growth rate, which occurs when  k < 1

Image RegistrationLecture Putting It All Together - The Example, Revisited

Image RegistrationLecture Turning to the Software  A “view” is a definition or snapshot of the registration problem.  A “view” contains:  An image region (current region, plus goal region)  A current transformation estimate and estimator  A current stage (resolution) of registration  Views work in conjunction with multistage registration  A “view” is a definition or snapshot of the registration problem.  A “view” contains:  An image region (current region, plus goal region)  A current transformation estimate and estimator  A current stage (resolution) of registration  Views work in conjunction with multistage registration

Image RegistrationLecture View-Based Registration - Procedural  The following is repeated for each initial estimate  For each stage:  Do  Match  Compute weights  Estimate scale  For each model  Run IRLS to estimate parameters and covariances  Re-estimate scale  Generate next view  For DB-ICP view generator this choose the best model and grows the region  Until region has converged and highest order model used  Prepare for next stage  The following is repeated for each initial estimate  For each stage:  Do  Match  Compute weights  Estimate scale  For each model  Run IRLS to estimate parameters and covariances  Re-estimate scale  Generate next view  For DB-ICP view generator this choose the best model and grows the region  Until region has converged and highest order model used  Prepare for next stage

Image RegistrationLecture Implementation  rgrl_view  Store the information about the view  rgrl_view_generator  Generate the next view  rgrl_view_based_registration  Mirrors rgrl_feature_based_registration with modifications based on the outline on previous slide  Example  rgrl/example/registration_retina.cxx  rgrl_view  Store the information about the view  rgrl_view_generator  Generate the next view  rgrl_view_based_registration  Mirrors rgrl_feature_based_registration with modifications based on the outline on previous slide  Example  rgrl/example/registration_retina.cxx

Image RegistrationLecture Summary  Retina registration:  Models, features and initialization  DB-ICP:  Matching, estimation and covariances  Model selection  Region growing  Generalization to view-based registration and its implementation in the toolkit.  Retina registration:  Models, features and initialization  DB-ICP:  Matching, estimation and covariances  Model selection  Region growing  Generalization to view-based registration and its implementation in the toolkit.

Image RegistrationLecture Looking Ahead to Lecture 17  Discussion of toolkits:  What’s easy, what’s hard  What’s missing  Project discussion  Requirements  Topics and initial steps  Discussion of toolkits:  What’s easy, what’s hard  What’s missing  Project discussion  Requirements  Topics and initial steps