1. Medical Image Analysis Sebastian Thrun, Gary Bradski, Daniel Russakoff Stanford CS223B Computer Vision http://robots.stanford.edu/cs223b

2. Daniel Russakoff Stanford University CS223B Computer Vision Introduction n The practice of modern medicine incorporates an enormous amount of image data n Traditional computer vision relies on cameras and, more recently, range finders n Medicine uses, to name a few: –Computed Tomography (CT) –Magnetic Resonance Imaging (MRI) –X-ray fluoroscopy –Ultrasound

3. Daniel Russakoff Stanford University CS223B Computer Vision Modalities: CT © L. Joskowicz (HUJI)

4. Daniel Russakoff Stanford University CS223B Computer Vision Modalities: MRI © L. Joskowicz (HUJI)

5. Daniel Russakoff Stanford University CS223B Computer Vision Modalities: X-ray fluoroscopy

6. Daniel Russakoff Stanford University CS223B Computer Vision Modalities: Ultrasound © L. Joskowicz (HUJI)

7. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection –3D Reconstruction n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

8. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection –3D Reconstruction n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

9. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection –3D Reconstruction n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

10. Daniel Russakoff Stanford University CS223B Computer Vision Segmentation n Thresholding (normal and adaptive) n Level sets (2D and 3D) n Shape models n Level sets + shape models n And beyond…

11. Daniel Russakoff Stanford University CS223B Computer Vision Level sets n In medicine, 3D segmentation often proceeds as a boundary propagation problem along the 2D slices of the data n Starting point for contour in new slice comes from the final contour in the previous slice Tsai, et al.

12. Daniel Russakoff Stanford University CS223B Computer Vision Level sets n Can think of this problem as one of tracking a moving interface in time n What happens as the interface splits and rejoins? © L. Joskowicz (HUJI)

13. Daniel Russakoff Stanford University CS223B Computer Vision Level sets n Snakes have difficulty dealing with changing topology n Requires messy bookkeeping of control points Sethian (UC Berkeley)

14. Daniel Russakoff Stanford University CS223B Computer Vision Level sets n Level sets deal with this in a very clever way. n We add a dimension to the problem and propagate the “level set surface” instead of the curve n The boundary becomes the “zero level set” Sethian (UC Berkeley)

15. Daniel Russakoff Stanford University CS223B Computer Vision Level sets Sethian (UC Berkeley)

16. Daniel Russakoff Stanford University CS223B Computer Vision Level sets n Now the question remains, how do we propagate the level set function? n F is a term representing the speed of motion

17. Daniel Russakoff Stanford University CS223B Computer Vision Level sets n Typical level set speed function F n The 1 causes the contour to expand in the object n The -  (viscosity) term reduces the curvature of the contour n The final term (edge attraction) pulls the contour to the edges n Other terms possible depending on your application n Level sets trivially extend to 3D segmentation

18. Daniel Russakoff Stanford University CS223B Computer Vision Level sets Sethian (UC Berkeley) Results: femur segmentation

19. Daniel Russakoff Stanford University CS223B Computer Vision Shape models n Learn modes of variation of a structure n Use PCA to generate orthonormal basis of variation n Ex. prostate segmentation n Start with a training set of segmented prostates Tsai, et al.

20. Daniel Russakoff Stanford University CS223B Computer Vision Shape models Mean shape and  of 1 st 4 principal modes of variation Tsai, et al.

21. Daniel Russakoff Stanford University CS223B Computer Vision Shape models n New shape can be seen as a linear combination of the basis shapes Patient A Patient B Tsai, et al.

22. Daniel Russakoff Stanford University CS223B Computer Vision Shape models + Level sets n Can incorporate priors based on shape models into the F term in the level set equation. n Leads to robust segmentations of challenging objects without much initialization Leventon, et al.

23. Daniel Russakoff Stanford University CS223B Computer Vision And beyond… n ICCV 2003: Geodesic contours + Min Cuts Boykov, et al.

24. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

25. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

26. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

27. Daniel Russakoff Stanford University CS223B Computer Vision Computer-aided Detection (CAD) n “CAD may be defined as a diagnosis made by a physician who takes into account the computer output as a second opinion” -Dr. Kunio Doi (U. Chicago) n Currently in use for early detection of breast cancer in mammography (FDA approved) n On the way for lung nodule detection and colon polyp detection

28. Daniel Russakoff Stanford University CS223B Computer Vision Case study: polyp detection n Step 1: CT scan of patient n Step 2: Segmentation of colon Paik, et al.

29. Daniel Russakoff Stanford University CS223B Computer Vision Case study: polyp detection n Step 3: detection of polyp candidates –Hough transform (looking for spheres) Paik, et al.

30. Daniel Russakoff Stanford University CS223B Computer Vision Case study: polyp detection n Step 4: feature extraction n Step 5: classification –Take your pick of algorithms (SVM, ANN, etc.) Gokturk, et al.

31. Daniel Russakoff Stanford University CS223B Computer Vision Case study: polyp detection n Step 6: Flythrough colon giving information to physician for final diagnosis (not yet realized) Paik, et al.

32. Daniel Russakoff Stanford University CS223B Computer Vision Case study: polyp detection Paik, et al.

33. Daniel Russakoff Stanford University CS223B Computer Vision Future…

34. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

35. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

36. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

37. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

38. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

39. Daniel Russakoff Stanford University CS223B Computer Vision Two categories of interest n Applications of standard computer vision techniques into the medical domain –Segmentation –Computer-Aided Detection n New techniques from medical image analysis added to the vision toolbox –Multi-modal registration

40. Daniel Russakoff Stanford University CS223B Computer Vision Registration n “The process of establishing a common, geometric reference frame between two data sets.” n Previously used in vision to align satellite images, generate image mosaics, etc. Image 1Image 2Registered + =

41. Daniel Russakoff Stanford University CS223B Computer Vision Registration in medicine n Explosion of data, both 2D and 3D from many different imaging modalities have made registration a very important and challenging problem in medicine © L. Joskowicz (HUJI) Ref_MRIRef_NMR

42. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

43. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

44. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Registration Preoperative Intraoperative X-rays USNMR CTMRIFluoro CAD Tracking US Open MR Special sensors Video Combined Data © L. Joskowicz (HUJI)

45. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

46. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

47. Daniel Russakoff Stanford University CS223B Computer Vision Feature selection n Points-based –3D points calculated using an optical tracker n Surfaces –Extracted from images using segmentation algorithms n Intensities –Uses the raw voxel data itself

48. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

49. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

50. Daniel Russakoff Stanford University CS223B Computer Vision Optimization n Gradients –Gradient descent –Conjugate-gradient –Levenburg-Marquardt n No gradients –Finite-difference gradient + above –Best-neighbor search –Nelder-Mead –Simulated annealing

51. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

52. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

53. Daniel Russakoff Stanford University CS223B Computer Vision Transformations n Rigid (6 DOF) –3 rotation –3 translation n Affine (12 DOF) –6 from before –3 scale –3 skew n Non-rigid (? DOF) –As many control points as your favorite supercomputer can handle © T. Rohlfing (Stanford)

54. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

55. Daniel Russakoff Stanford University CS223B Computer Vision Multi-modal registration Data Set #1 Feature Selection Feature Selection T Similarity Measure Optimizer Transform Data Set #2

56. Daniel Russakoff Stanford University CS223B Computer Vision Similarity measures n Intra-modality –normalized cross-correlation –gradient correlation –pattern intensity –sum of squared differences n Inter-modality –mutual information (the industry standard)

57. Daniel Russakoff Stanford University CS223B Computer Vision Example: CT-DSA Native CT imagePost-contrast CT image © T. Rohlfing (Stanford)

58. Daniel Russakoff Stanford University CS223B Computer Vision Example: CT-DSA After affine registration B-spline with 10mm c.p.g. © T. Rohlfing (Stanford)

59. Daniel Russakoff Stanford University CS223B Computer Vision Example: CT-DSA After affine registration B-spline with 10mm c.p.g. © T. Rohlfing (Stanford)

60. Daniel Russakoff Stanford University CS223B Computer Vision Example: Liver motion Respiration gating during abdominal MR imaging Time © T. Rohlfing (Stanford)

61. Daniel Russakoff Stanford University CS223B Computer Vision Example: liver motion © T. Rohlfing (Stanford)

62. Daniel Russakoff Stanford University CS223B Computer Vision Irradiate tumor (T) with a series of directed beams avoiding critical structures (C) Example: CyberKnife T C

63. Daniel Russakoff Stanford University CS223B Computer Vision RD X Y Z The crux of the problem is to match up the coordinate frames of the CT and the radiation delivery device Example: CyberKnife X2X2X2X2 Y2Y2Y2Y2 Z2Z2Z2Z2 CT X2X2X2X2 Y2Y2Y2Y2 Z2Z2Z2Z2 CT

64. Daniel Russakoff Stanford University CS223B Computer Vision RD X Y Z CT X2X2X2X2 Y2Y2Y2Y2 Z2Z2Z2Z2 Using only 2D projection images! Example: CyberKnife RD CT X2X2 Y2Y2 Z2Z2 X Y Z

65. Daniel Russakoff Stanford University CS223B Computer Vision Example: CyberKnife virtual source RD X Y Z RD X Y Z

66. Daniel Russakoff Stanford University CS223B Computer Vision CT T1T1 Example: CyberKnife Digitally Reconstructed Radiograph virtual source RD X Y Z RD X Y Z

67. Daniel Russakoff Stanford University CS223B Computer Vision Example: CyberKnife CT T2T2 DRR virtual source RD X Y Z RD X Y Z

68. Daniel Russakoff Stanford University CS223B Computer Vision CT T*T* DRR virtual source RD X Y Z RD X Y Z Example: CyberKnife

69. Daniel Russakoff Stanford University CS223B Computer Vision Conclusions n Medicine is a fertile and active area for computer vision research n Application of existing vision tools to new, challenging domains n Development of new vision tools to assist in the practice of medicine

70. Daniel Russakoff Stanford University CS223B Computer Vision The End