3D Rigid/Nonrigid RegistrationRegistration 1)Known features, correspondences, transformation model – feature basedfeature based 2)Specific motion type, unknown correspondences – feature basedfeature based 3) Known transformation model, unknown correspondences – region basedregion based 4) Specific motion model – feature basedfeature based 5) Unknown motion model, unknown correspondences – region based

Visual Motion Jim Rehg (G.Tech)

Motion (Displacement) of Environment Image plane Scene Flow Motion Field Visual motion results from the displacement of the scene with respect to a fixed camera (or vice-versa). Motion field is the 2-D velocity field that results from a projection of the 3-D scene velocities

Examples of Visual Motion

Applications of Motion Analysis Visual tracking Structure recovery Robot (vehicle) navigation

Applications of Motion Analysis Visual tracking Structure recovery Robot (vehicle) navigation

Motion Segmentation Where are the independently moving objects (and how many are there)?

Optical Flow 2-D velocity field describing the apparent motion in an image sequence A vector at each pixel indicates its motion (between a pair of frames). Ground truthHorn and Schunk

Optical Flow and Motion Field In general the optical flow is an approximation to the motion field. When the scene can be segmented into rigidly moving objects (for example) the relationship between the two can be made precise. We can always think of the optical flow as summarizing the temporal change in an image sequence.

Computing Optical Flow Courtesy of Michael Black

Cost Function for Optical Flow Courtesy of Michael Black

Lucas-Kanade Method Brute-force minimization of SSD error can be inefficient and inaccurate Many redundant window evaluations Answer is limited to discrete u, v pairs

Lucas-Kanade Method Problems with brute-force minimization of SSD error Many redundant window evaluations Answer is limited to discrete u, v pairs Related to Horn-Schunk optical flow equations Several key innovations Early, successful use of patch-based model in low-level vision. Today these models are used everywhere. Formulation of vision problem as non-linear least squares optimization, a trend which continues to this day.

Optical Flow Estimation

Optical Flow Constraint

Optimization

Quality of Image Patch Eigenvalues of the matrix contain information about local image structure Both eigenvalues (close to) zero: Uniform area One eigenvalue (close to) zero: Edge No eigenvalues (close to) zero: Corner

Contributions of Lucas-Kanade Basic idea of patch or template is very old (goes back at least to Widrow) But in practice patch models have worked much better than the alternatives: Point-wise differential equations with smoothness Edge-based descriptions Patchs provide a simple compact enforcement of spatial continuity and support (robust) least-squares estimators.

Lets Talk Applications

Alain Pitiot, Ph.D. Siemens Molecular Imaging - Advanced Applications Medical Image Registration (a short overview) Summer School 2005

SOME APPLICATIONS Medical Image Registration

Motivation Advances in imaging technology  novel modalities see beyond: inside (non-invasive), during (dynamic processes), at small scale (increased resolution) Understanding and correlating structure & function - automated/aided diagnosis - image guided surgery/radio-therapy - treatment/surgery planning - medical atlases - longitudinal studies: disease progression, development

Definitions Def. #1: put two images into spatial correspondence goal: extract more/better information Def. #2: maximize similarity between transformed source image & target image CT (thorax)PET (thorax) source image target image + transformed target image Anatomical Functional

Taxonomy Nature of application Subject - intrasubject - intersubject - atlas Homer Simpson (MRI, coronal section)

Nature of application Subject - intrasubject - intersubject - atlas Homer Simpson (rest position) Homer Simpson (monkey position)  very similar shapes Taxonomy

Nature of application Subject - intrasubject - intersubject - atlas Homer Simpson Homo sapiens sapiens brain  expect larger differences Taxonomy

Nature of application Subject - intrasubject - intersubject - atlas Homer Simpson (MRI) Homer Simpson (labelled atlas) Taxonomy

Nature of application Subject - intrasubject - intersubject - atlas Registration basis - extrinsic - intrinsic Taxonomy

Nature of application Subject - intrasubject - intersubject - atlas Registration basis - extrinsic - intrinsic stereotactic frame  fast, explicit computation  prospective, often invasive, often rigid transf. only Taxonomy

Nature of application Subject - intrasubject - intersubject - atlas Registration basis - extrinsic - intrinsic  versatile, minimally invasive  no ground truth PET scintillography Taxonomy

Registration basis extrinsic intrinsic - landmark based - segmentation based - voxel based  fast  accuracy limited by localization precision CT PET Taxonomy | Nature of Application

Registration basis extrinsic intrinsic - landmark based - segmentation based - voxel based segmented corpora callosa  fast  accuracy limited by segmentation  combine with voxel based Taxonomy | Nature of Application

Registration basis extrinsic intrinsic - landmark based - segmentation based - voxel based cryo. section myelin-stained histological section  most flexible approach  resource intensive  combine with previous techniques Taxonomy | Nature of Application

Nature of input images Modality Combination: - mono-modal: same modality for source and target - multi-modal: different modality Dimensionality - spatial: 2-D/2-D, 2-D/3-D, 3-D/3-D - temporal a few imaging modalities Taxonomy

Constraints fusion  maximize similarity between transformed source & target Transformation space - flexibility rigid, affine, parameterized, free-form - support local, global  choose space that fits anatomy and/or application global local rigid affine parameterized fluid/elastic Taxonomy

Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing Taxonomy

Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing Taxonomy

Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing Taxonomy

Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing Heteroclite bag of tricks - progressive refinement - multi-scale (multi-resolution) Taxonomy

Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing bag of tricks - progressive refinement - multi-scale (multi-resolution) Taxonomy

Issues Validation No ground truth in general case (ill-posed problem) Precision, robustness, reliability, etc. Semi-automated registration Is fully-automated desirable ? Which compromise between fully and semi ?

Specific Application

Image Guided Surgery

Conventional Surgery: Seeing surfaces Provided by Nakajima, Atsumi et al.

Computer Assisted Surgery: seeing through surfaces

Goal: Assist Surgeons Surgical Planning & Simulation Maximize Tumor Removal Minimize Damage to Critical Structures Intraoperative Visualizations via 3D Slicer

Pre-Operative Image Processing Construct 3D Models Semi-Automated Segmentation DTMRI Tract Tracing Register all pre-operative data

Integrated Preoperative Data F. Talos

Patient-specific models Gering_fmri

Segmentation of Neural Structures

Intraoperative Image Processing Acquire one or more volumetric (interventional) MRI (iMRI) images Determine non-rigid registration of Pre- and Intra- operative data

Construct Intraoperative Visualization transmit image data and 3D models thru volumetric deformation integrate with iMRI images and models display with 3D Slicer LCD screen in front of surgeon in iMRI coordinate visualization with intraoperative instruments

3D Slicer: tool for Visualization Registration Segmentation Measurements Realtime Integration Provided by D. Gering

3D Slicer Demo...

More Examples

More examples of correspondence: Motion (tracking) beating heart We have to establish correspondence between specific points on the object boundary from frame to frame

Template matching In matching we estimate “position” of a rigid template in the image “Position” includes global location parameters of a rigid template: - translation, rotation, scale,… Face template image

Flexible template matching In flexible template matching we estimate “position” of each rigid component of a template

3D Doctor Multimodal registrationregistration

Warping example