1. MIT AI Labtkapur@ai.mit.edu Knowledge Based 3D Medical Image Segmentation Tina Kapur MIT Artificial Intelligence Laboratory http://www.ai.mit.edu/~tkapur

2. MIT AI Labtkapur@ai.mit.edu Outline Goal of Segmentation Applications Why is segmentation difficult? My method for segmentation of MRI Future Work

3. MIT AI Labtkapur@ai.mit.edu The Goal of Segmentation

4. MIT AI Labtkapur@ai.mit.edu The Goal of Segmentation

5. MIT AI Labtkapur@ai.mit.edu Applications of Segmentation Image Guided Surgery

6. MIT AI Labtkapur@ai.mit.edu Applications of Segmentation Image Guided Surgery

7. MIT AI Labtkapur@ai.mit.edu Applications of Segmentation Image Guided Surgery Surgical Simulation

8. MIT AI Labtkapur@ai.mit.edu Applications of Segmentation Image Guided Surgery Surgical Simulation

9. MIT AI Labtkapur@ai.mit.edu Applications of Segmentation Image Guided Surgery Surgical Simulation Neuroscience Studies Therapy Evaluation

10. MIT AI Labtkapur@ai.mit.edu Limitations of Manual Segmentation slow (up to 60 hours per scan) variable (up to 15% between experts) [Warfield 95, Kaus98]

11. MIT AI Labtkapur@ai.mit.edu The Automatic Segmentation Challenge An automated segmentation method needs to reconcile –Gray-level appearance of tissue –Characteristics of imaging modality –Geometry of anatomy

12. MIT AI Labtkapur@ai.mit.edu How to Segment? i.e. Issues in Segmentation of Anatomy

13. MIT AI Labtkapur@ai.mit.edu How to Segment? i.e. Issues in Segmentation of Anatomy Tissue Intensity Models

14. MIT AI Labtkapur@ai.mit.edu How to Segment? i.e. Issues in Segmentation of Anatomy Tissue Intensity Models –Parametric [Vannier] –Non-Parametric [Gerig] –Point distribution Models [Cootes] –Texture [Mumford]

15. MIT AI Labtkapur@ai.mit.edu How to Segment? i.e. Issues in Segmentation of Anatomy Tissue Intensity Models Imaging Modality Models

16. MIT AI Labtkapur@ai.mit.edu How to Segment? i.e. Issues in Segmentation of Anatomy Tissue Intensity Models Imaging Modality Models –MRI inhomogeneity [Wells]

17. MIT AI Labtkapur@ai.mit.edu How to Segment? i.e. Issues in Segmentation of Anatomy Tissue Intensity Models Imaging Modality Models Anatomy Models: Shape, Geometric/Spatial

18. MIT AI Labtkapur@ai.mit.edu How to Segment? i.e. Issues in Segmentation of Anatomy Tissue Intensity Models Imaging Modality Models Anatomy Models: Shape, Geometric/Spatial –PCA [Cootes and Taylor, Gerig, Duncan, Martin] –Landmark Based [Evans] –Atlas [Warfield]

19. MIT AI Labtkapur@ai.mit.edu Typical Pipeline for Segmentation of Brain MRI Pre-processing for noise reduction EM Segmentation Morphological or other post-processing pre-processing (noise removal)

20. MIT AI Labtkapur@ai.mit.edu Typical Pipeline for Segmentation of Brain MRI Pre-processing for noise reduction EM Segmentation Morphological or other post-processing pre-processing (noise removal) intensity-based classification

21. MIT AI Labtkapur@ai.mit.edu Typical Pipeline for Segmentation of Brain MRI Pre-processing for noise reduction EM Segmentation Morphological or other post-processing pre-processing (noise removal) post-processing (morphology/other) intensity-based classification

22. MIT AI Labtkapur@ai.mit.edu Contributions of Thesis Developed an integrated Bayesian Segmentation Method for MRI that incorporates de-noising and global geometric knowledge using priors into EM- Segmentation Applied integrated Bayesian method to segmentation of Brain and Knee MRI.

23. MIT AI Labtkapur@ai.mit.edu Contributions of Thesis The Priors –de-noising: novel use of a Mean-Field Approximation to a Gibbs random field in conjunction with EM-Segmentation (EM-MF) –geometric: novel statistical description of global spatial relationships between structures, used as a spatially varying prior in EM- Segmentation

24. MIT AI Labtkapur@ai.mit.edu Background to My Work Expectation-Maximization Algorithm EM-Segmentation

25. MIT AI Labtkapur@ai.mit.edu Expectation-Maximization Relevant Literature: –[Dempster, Laird, Rubin 1977] –[Neal 1998]

26. MIT AI Labtkapur@ai.mit.edu Expectation-Maximization (what?) Search Algorithm for Parameters of a Model to Maximize Likelihood of Data Data: some observed, some unobserved

27. MIT AI Labtkapur@ai.mit.edu Expectation-Maximization (how?) Initial Guess of Model Parameters Re-estimate Model Parameters: –E Step: compute PDF for hidden variables, given observations and current model parameters –M Step: compute ML model parameters assuming pdf for hidden variables is correct

28. MIT AI Labtkapur@ai.mit.edu Notation –Observed Variables: –Hidden Variables : –Model Parameters: Expectation-Maximization (how exactly?)

29. MIT AI Labtkapur@ai.mit.edu Initial Guess: Successive Estimation of –E Step: –M Step: Expectation-Maximization (how exactly?)

30. MIT AI Labtkapur@ai.mit.edu Expectation-Maximization Summary/Intuition: –If we had complete data, maximize likelihood –Since some data is missing, approximate likelihood with its expectation –Converges to local maximum of likelihood

31. MIT AI Labtkapur@ai.mit.edu EM-Segmentation [Wells 1994] Observed Signal is modeled as a product of the true signal and a corrupting gain field due to the imaging equipment Expectation-Maximization is used on log- transformed observations for iterative estimation of –tissue classification –corrupting bias field (inhomogeneity correction)

32. MIT AI Labtkapur@ai.mit.edu M-Step E-Step EM-Segmentation [Wells 1994]

33. MIT AI Labtkapur@ai.mit.edu Estimate intensity correction using residuals based on current posteriors. Compute tissue posteriors using current intensity correction. M-Step E-Step EM-Segmentation [Wells 1994]

34. MIT AI Labtkapur@ai.mit.edu Observed Variables –log transformed intensities in image Hidden Variables –indicator variables for classification Model Parameters –the slowly varying corrupting bias field ( refer to variables at voxel s in image) EM-Segmentation [Wells 1994]

35. MIT AI Labtkapur@ai.mit.edu Initial Guess: Successive Estimation of –E Step: –M Step: EM-Segmentation [Wells 1994]

36. MIT AI Labtkapur@ai.mit.edu Initial Guess: Successive Estimation of –E Step: –M Step: EM-Segmentation [Wells 1994]

37. MIT AI Labtkapur@ai.mit.edu Situating My Work Prior in EM-Segmentation: –Independent and Spatially Stationary My contribution is addition of two priors: –a spatially stationary Gibbs prior to model local interactions between neighbors (thermal noise) –spatially varying prior to model global relationships between geometry of structures

38. MIT AI Labtkapur@ai.mit.edu The Gibbs Prior Gibbs Random Field (GRF) –natural way to model piecewise homogeneous phenomena –used in image restoration [Geman&Geman 84] –Probability Model on a lattice –Partially Relaxes independence assumption to allow interactions between neighbors

39. MIT AI Labtkapur@ai.mit.edu EM-MF Segmentation: EM + Gibbs Prior We model tissue classification W as a Gibbs random field:

40. MIT AI Labtkapur@ai.mit.edu We model tissue classification W as a Gibbs random field: EM-MF Segmentation: Gibbs Prior on Classification

41. MIT AI Labtkapur@ai.mit.edu To fully specify the Gibbs model: –define neighborhood system as a first order neighborhood system i.e. 6 closest voxels –use to define EM-MF Segmentation: Gibbs Prior on Classification

42. MIT AI Labtkapur@ai.mit.edu EM-MF Segmentation: Gibbs form of Posterior Gibbs prior and Gaussian Measurement Models lead to Gibbs form for Posterior:

43. MIT AI Labtkapur@ai.mit.edu Gibbs prior and Gaussian Measurement Models lead to Gibbs form for Posterior: EM-MF Segmentation: Gibbs form of Posterior

44. MIT AI Labtkapur@ai.mit.edu EM-MF Segmentation For E-Step: Need values for

45. MIT AI Labtkapur@ai.mit.edu EM-MF Segmentation For E-Step: Need values for Cannot compute directly from Gibbs form

46. MIT AI Labtkapur@ai.mit.edu EM-MF Segmentation For E-Step: Need values for Cannot compute directly from Gibbs form Note

47. MIT AI Labtkapur@ai.mit.edu EM-MF Segmentation For E-Step: Need values for Cannot compute directly from Gibbs form Note Can approximate – Mean-Field Approximation to GRF

48. MIT AI Labtkapur@ai.mit.edu Mean-Field Approximation Deterministic Approximation to GRF [Parisi84] –the mean/expected value of a GRF is obtained as a solution to a set of consistency equations Update Equation is obtained using derivative of partition function with respect to the external field g. [Elfadel 93] Used in image reconstruction [Geiger, Yuille, Girosi 91]

49. MIT AI Labtkapur@ai.mit.edu Mean-Field Approximation to Posterior GRF Intuition: denominator is normalizer numerator captures: effect of labels at neighbors measurement at voxel itself

50. MIT AI Labtkapur@ai.mit.edu Summary of EM-MF Segmentation Modeled piecewise homogeneity of tissue using a Gibbs prior on classification Lead to Gibbs form for Posteriors Posterior Probabilities in E-Step are approximated as a Mean-Field solution

51. MIT AI Labtkapur@ai.mit.edu EM-MF Results Application: Brain MRI –white matter, gray matter, fluid/air, skin/scalp Results Comparison with Manual Segmentation

52. MIT AI Labtkapur@ai.mit.edu Some Results EM EM-MF

53. MIT AI Labtkapur@ai.mit.edu Some Results EM EM-MF

54. MIT AI Labtkapur@ai.mit.edu More Results Noisy MRIEM Segmentation EM-MF Segmentation

55. MIT AI Labtkapur@ai.mit.edu Posterior Probabilities (EM) White matter Gray matter

56. MIT AI Labtkapur@ai.mit.edu Posterior Probabilities (EM-MF) White matter Gray matter

57. MIT AI Labtkapur@ai.mit.edu Results

58. MIT AI Labtkapur@ai.mit.edu Modeling Global Geometric Relationships between Structures

59. MIT AI Labtkapur@ai.mit.edu Relative Geometry Models Motivate Using Knee MRI Brain MRI Example Modeling Global Geometric Relationships between Structures

60. MIT AI Labtkapur@ai.mit.edu Segmented Knee MRI Femur Tibia Femoral Cartilage Tibial Cartilage MERL, SPL, MIT, CMU Surgical Simulation (Sarah Gibson, PI)

61. MIT AI Labtkapur@ai.mit.edu Motivation Primary Structures –image well –easy to segment Secondary Structures –image poorly –relative to primary Tibial Cartilage Femoral Cartilage Tibia Femur

62. MIT AI Labtkapur@ai.mit.edu Relative Geometric Prior Approach Select primary/secondary structures Measure geometric relation between primary and secondary structures from training data Given novel image –segment primary structures –use geometric relation as prior on secondary structure in EM-MF Segmentation

63. MIT AI Labtkapur@ai.mit.edu Segment Primary Structures: Femur, Tibia SeedRegion GrowingBoundary Localization

64. MIT AI Labtkapur@ai.mit.edu Status Have Bone Want Cartilage

65. MIT AI Labtkapur@ai.mit.edu Measure Geometric Relationship between Primary and Secondary Structures Femur Tibia Femoral Cartilage Tibial Cartilage Using primitives such as –distances between surfaces –local normals of primary structures –local curvature of primary structures –etc.

66. MIT AI Labtkapur@ai.mit.edu Femur Tibia Femoral Cartilage Tibial Cartilage Measure Geometric Relationship between Primary and Secondary Structures

67. MIT AI Labtkapur@ai.mit.edu Estimate of

68. MIT AI Labtkapur@ai.mit.edu Status Have Bone Have spatial relation between Bone and Cartilage Need Cartilage

69. MIT AI Labtkapur@ai.mit.edu Use Relative Geometric Prior in EM Segmentation Replace stationary prior with relative geometric prior:

70. MIT AI Labtkapur@ai.mit.edu Results: Segmentation of Femoral & Tibial Cartilage MRI Image Model-Based Segmentation Manual Segmentation

71. MIT AI Labtkapur@ai.mit.edu Relative Geometric Priors for Brain Tissue Prior Estimation –Select primary structures (boundary of skin, ventricles) –Estimate Using Prior in Segmentation –Segment primary structures: skin, ventricles –Use as geometric prior

72. MIT AI Labtkapur@ai.mit.edu White MatterGray Matter Estimate distance to Ventricles distance to Skin

73. MIT AI Labtkapur@ai.mit.edu Resultant Segmentation MRIEM Segmentation EM-MF with Geometric Prior

74. MIT AI Labtkapur@ai.mit.edu Posterior Probabilities Gray Matter White Matter EM-MF+Geometric PriorEM

75. MIT AI Labtkapur@ai.mit.edu In Summary Incorporated robustness to thermal noise by using Mean-Field Approximation to Gibbs model in conjunction with EM Segmentation. Applied to Brain MRI. Introduced Relative-Geometry Models and applied to Brain and Knee MRI.

76. MIT AI Labtkapur@ai.mit.edu Future Work Further development of Relative-Geometry Models: –Automatic selection of primary/secondary structures –Additional primitives for Spatial Relationships

77. MIT AI Labtkapur@ai.mit.edu STOP

78. MIT AI Labtkapur@ai.mit.edu d1 (distance to Ventricles) d2 (distance to Skin) White Matter Grey Matter Fluid Air Left Caudate Right Caudate Class Conditional Density

79. MIT AI Labtkapur@ai.mit.edu Unified Bayesian Segmentation Method Simultaneous noise reduction intensity-based classification use of geometric information for segmentation.

80. MIT AI Labtkapur@ai.mit.edu Rest of Talk: 1.The Unified Segmentation Method 2.Two Priors 3.Results on Brain, Knee Segmentation 4.Conclusions

81. MIT AI Labtkapur@ai.mit.edu The Gibbs Prior To fully specify the Gibbs model: –define neighborhood system as a first order neighborhood system i.e. 6 closest voxels –use to define

82. MIT AI Labtkapur@ai.mit.edu Components of EM Framework Measurement models for tissue Prior models for tissue Model for bias field –piecewise smooth

83. MIT AI Labtkapur@ai.mit.edu Addition of Two Priors Gibbs prior on tissue appearance –models tissue as piecewise constant Geometric Prior to encode spatial relations –gray matter is outside ventricles and inside skull

84. MIT AI Labtkapur@ai.mit.edu Gibbs Model

85. MIT AI Labtkapur@ai.mit.edu Gibbs Model probability model on a lattice independence assumption is partially relaxed spatial range of interaction is local neighborhood

86. MIT AI Labtkapur@ai.mit.edu

87. MIT AI Labtkapur@ai.mit.edu Gibbs Model probability model on a lattice independence assumption is partially relaxed spatial range of interaction is local neighborhood Mean-Field Approximation Approximates neighboring random variables with their mean values: - algebraic and computational simplicity

88. MIT AI Labtkapur@ai.mit.edu Contributions of Thesis MRI+NoiseEM Segmentation EM-MF with Geometric Prior

89. MIT AI Labtkapur@ai.mit.edu Proposed MRI Segmentation Method Bayesian Statistical Classification Scheme that uses Expectation-Maximization Replaces pipeline with Priors on intensity and geometry

90. MIT AI Labtkapur@ai.mit.edu Proposed MRI Segmentation Method Previous Work [Wells 1994, 1996] –derived as a special case –spatially stationary, independent priors –piecewise smooth inhomogeneity model This Work: –locally interacting prior for intensity –spatially varying prior for geometry

91. MIT AI Labtkapur@ai.mit.edu Next Background on Expectation-Maximization

92. MIT AI Labtkapur@ai.mit.edu 1. Use Bayes’ rule, and independence between and to write: 2. Specify Measurement Models as Gaussian Computation of

93. MIT AI Labtkapur@ai.mit.edu 3. Specify Prior on Classification as a Gibbs Random Field. 4. Gibbs Prior + Gaussian Measurement Model imply is also a GRF. 5. Approximate using Mean- Field Solution for the GRF. Computation of

94. MIT AI Labtkapur@ai.mit.edu Recap: 1. Bayes’ Rule to rewrite … 2. Gaussian Measurement Models  3. Gibbs form of Prior  4. Gibbs form of Posterior  5. Mean-Field Approximation to Gibbs form Computation of

95. MIT AI Labtkapur@ai.mit.edu The Gibbs Prior Tissue-class interaction Matrix Stationary Prior

96. MIT AI Labtkapur@ai.mit.edu The Mean-Field Solution Gibbs models can be solved using –[Metropolis 1953], [Geman and Geman 1984], [Besag 1986] etc. We use Mean-Field Approximation to estimate the expected value of the posterior GRF as a solution to a set of consistency equations.

97. MIT AI Labtkapur@ai.mit.edu The Mean-Field Approximation Deterministic Approximation Update Equation is obtained using derivative of partition function with respect to the external field g.

98. MIT AI Labtkapur@ai.mit.edu The Mean-Field Solution Intuition: denominator is normalizer numerator captures: effect of labels at neighbors measurement at voxel itself

99. MIT AI Labtkapur@ai.mit.edu Initial Guess: Successive Estimation of –E Step: Estimate as Mean-Field Solution to a Gibbs Random Field –M Step: Compute same as [Wells 1996] EM-MF Summary