1. 3D Rigid/Nonrigid RegistrationRegistration 1)Known features, correspondences, transformation model – feature basedfeature based 2)Specific motion type, unknown correspondences – feature basedfeature based 3) Known transformation model, unknown correspondences – region basedregion based 4) Specific motion model – feature basedfeature based 5) Unknown motion model, unknown correspondences – region based

2. Visual Motion Jim Rehg (G.Tech)

3. Motion (Displacement) of Environment Image plane Scene Flow Motion Field Visual motion results from the displacement of the scene with respect to a fixed camera (or vice-versa). Motion field is the 2-D velocity field that results from a projection of the 3-D scene velocities

4. Examples of Visual Motion

7. Applications of Motion Analysis Visual tracking Structure recovery Robot (vehicle) navigation

8. Applications of Motion Analysis Visual tracking Structure recovery Robot (vehicle) navigation

9. Motion Segmentation Where are the independently moving objects (and how many are there)?

10. Optical Flow 2-D velocity field describing the apparent motion in an image sequence A vector at each pixel indicates its motion (between a pair of frames). Ground truthHorn and Schunk

11. Optical Flow and Motion Field In general the optical flow is an approximation to the motion field. When the scene can be segmented into rigidly moving objects (for example) the relationship between the two can be made precise. We can always think of the optical flow as summarizing the temporal change in an image sequence.

12. Computing Optical Flow Courtesy of Michael Black

13. Cost Function for Optical Flow Courtesy of Michael Black

14. Lucas-Kanade Method Brute-force minimization of SSD error can be inefficient and inaccurate Many redundant window evaluations Answer is limited to discrete u, v pairs

15. Lucas-Kanade Method Problems with brute-force minimization of SSD error Many redundant window evaluations Answer is limited to discrete u, v pairs Related to Horn-Schunk optical flow equations Several key innovations Early, successful use of patch-based model in low-level vision. Today these models are used everywhere. Formulation of vision problem as non-linear least squares optimization, a trend which continues to this day.

16. Optical Flow Estimation

18. Optical Flow Constraint

19. Optimization

22. Quality of Image Patch Eigenvalues of the matrix contain information about local image structure Both eigenvalues (close to) zero: Uniform area One eigenvalue (close to) zero: Edge No eigenvalues (close to) zero: Corner

23. Contributions of Lucas-Kanade Basic idea of patch or template is very old (goes back at least to Widrow) But in practice patch models have worked much better than the alternatives: Point-wise differential equations with smoothness Edge-based descriptions Patchs provide a simple compact enforcement of spatial continuity and support (robust) least-squares estimators.

24. Lets Talk Applications

25. Alain Pitiot, Ph.D. Siemens Molecular Imaging - Advanced Applications Medical Image Registration (a short overview) Summer School 2005

26. SOME APPLICATIONS Medical Image Registration

27. Motivation Advances in imaging technology  novel modalities see beyond: inside (non-invasive), during (dynamic processes), at small scale (increased resolution) Understanding and correlating structure & function - automated/aided diagnosis - image guided surgery/radio-therapy - treatment/surgery planning - medical atlases - longitudinal studies: disease progression, development

28. Definitions Def. #1: put two images into spatial correspondence goal: extract more/better information Def. #2: maximize similarity between transformed source image & target image CT (thorax)PET (thorax) source image target image + transformed target image Anatomical Functional

29. Taxonomy Nature of application Subject - intrasubject - intersubject - atlas Homer Simpson (MRI, coronal section)

30. Nature of application Subject - intrasubject - intersubject - atlas Homer Simpson (rest position) Homer Simpson (monkey position)  very similar shapes Taxonomy

31. Nature of application Subject - intrasubject - intersubject - atlas Homer Simpson Homo sapiens sapiens brain  expect larger differences Taxonomy

32. Nature of application Subject - intrasubject - intersubject - atlas Homer Simpson (MRI) Homer Simpson (labelled atlas) Taxonomy

33. Nature of application Subject - intrasubject - intersubject - atlas Registration basis - extrinsic - intrinsic Taxonomy

34. Nature of application Subject - intrasubject - intersubject - atlas Registration basis - extrinsic - intrinsic stereotactic frame  fast, explicit computation  prospective, often invasive, often rigid transf. only Taxonomy

35. Nature of application Subject - intrasubject - intersubject - atlas Registration basis - extrinsic - intrinsic  versatile, minimally invasive  no ground truth PET scintillography Taxonomy

36. Registration basis extrinsic intrinsic - landmark based - segmentation based - voxel based  fast  accuracy limited by localization precision CT PET Taxonomy | Nature of Application

37. Registration basis extrinsic intrinsic - landmark based - segmentation based - voxel based segmented corpora callosa  fast  accuracy limited by segmentation  combine with voxel based Taxonomy | Nature of Application

38. Registration basis extrinsic intrinsic - landmark based - segmentation based - voxel based cryo. section myelin-stained histological section  most flexible approach  resource intensive  combine with previous techniques Taxonomy | Nature of Application

39. Nature of input images Modality Combination: - mono-modal: same modality for source and target - multi-modal: different modality Dimensionality - spatial: 2-D/2-D, 2-D/3-D, 3-D/3-D - temporal a few imaging modalities Taxonomy

40. Constraints fusion  maximize similarity between transformed source & target Transformation space - flexibility rigid, affine, parameterized, free-form - support local, global  choose space that fits anatomy and/or application global local rigid affine parameterized fluid/elastic Taxonomy

41. Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

42. Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

43. Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

44. Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

45. Constraints Similarity measure “intensities of matched images verify criterion” - to each hypothesis its measure: conservation affine functional statistical Taxonomy conservation of intensity  SSD affine relationship  correlation coefficient functional relationship  correlation ratio statistical dependence  mutual information source target

46. Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing Taxonomy

47. Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing Taxonomy

48. Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing Taxonomy

49. Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing Heteroclite bag of tricks - progressive refinement - multi-scale (multi-resolution) Taxonomy

50. Optimization Often iterative - deterministic gradient descent - stochastic simulated annealing bag of tricks - progressive refinement - multi-scale (multi-resolution) Taxonomy

51. Issues Validation No ground truth in general case (ill-posed problem) Precision, robustness, reliability, etc. Semi-automated registration Is fully-automated desirable ? Which compromise between fully and semi ?

52. Specific Application

53. Image Guided Surgery

54. Conventional Surgery: Seeing surfaces Provided by Nakajima, Atsumi et al.

55. Computer Assisted Surgery: seeing through surfaces

56. Goal: Assist Surgeons Surgical Planning & Simulation Maximize Tumor Removal Minimize Damage to Critical Structures Intraoperative Visualizations via 3D Slicer

57. Pre-Operative Image Processing Construct 3D Models Semi-Automated Segmentation DTMRI Tract Tracing Register all pre-operative data

58. Integrated Preoperative Data F. Talos

59. Patient-specific models Gering_fmri

60. Segmentation of Neural Structures

61. Intraoperative Image Processing Acquire one or more volumetric (interventional) MRI (iMRI) images Determine non-rigid registration of Pre- and Intra- operative data

62. Construct Intraoperative Visualization transmit image data and 3D models thru volumetric deformation integrate with iMRI images and models display with 3D Slicer LCD screen in front of surgeon in iMRI coordinate visualization with intraoperative instruments

63. 3D Slicer: tool for Visualization Registration Segmentation Measurements Realtime Integration Provided by D. Gering

64. 3D Slicer Demo...

65. More Examples

66. More examples of correspondence: Motion (tracking) beating heart We have to establish correspondence between specific points on the object boundary from frame to frame

67. Template matching In matching we estimate “position” of a rigid template in the image “Position” includes global location parameters of a rigid template: - translation, rotation, scale,… Face template image

68. Flexible template matching In flexible template matching we estimate “position” of each rigid component of a template

69. 3D Doctor Multimodal registrationregistration

70. Warping example