Model-Based Organ Segmentation in CT Scans Jiun-Hung Chen

Input Output

Organ Segmentation in 3D CT scans First step for further analysis Computer-aided detection, computer-aided diagnosis, radiation treatment, or organ shape analysis. Manual organ segmentation: accurate but very time-consuming An efficient and effective automatic or semi-automatic organ segmentation method

Problem Statement Learn how to segment unseen testing CT images from a set of training CT images with ground truths Goal: Minimize the training errors while can generalize to unseen testing CT images

Why Difficult? (Shape Variations)

Why Difficult? (Similar Appearances)

Why Difficult? (Appearance Changes)

Why Difficult? (Position Changes)

Outline Introduction Previous Work Initial Results Proposal Conclusions

Active Shape Model Based Framework Input Training CT Volumes 3D point correspondence Learning Statistical Shape Models Known point correspondences Learning Organ Detection Learning Boundary Intensity Model Boundary Refinement Organ Detection A Testing CT Volume Shape Model Initialization Learned Statistical Shape Model Learned Boundary Intensity Model Organ Detector Detected Bounding Box Initial Shape Final Shape … Training phase Testing phase … …

P1 P2 P3 Organ 1 Organ 2 Organ N

Input Output Correspondence 1 Correspondence 2 P1 P2 P3 P1 P2 P3 P1 P2

P1 P2 P3 x y z Input Output

Learned Boundary Intensity Model Input Output Intensity profiles P1 P2 P3 Learned Boundary Intensity Model Prob. Intensity profiles

x,y,z rx ry rz sx sy sz … Input Output

Learned Statistical Shape Model Organ Detector Test Organ x,y,z rx ry rz sx sy sz Test Organ Test Organ

P Learned Boundary Intensity Model SSM SSM

Outline Introduction Previous Work Initial Results Proposal Conclusions

3D Point Correspondence (MDL) Goal: Find 3D Point Correspondence Idea: Minimize MDL-based objective function Evaluate the quality of the correspondence How: Gradient descent Manipulate correspondences by parameterization and re-parameterization. Davies et al. [IEEE TMI’02]

Statistical Shape Models Principal Component Analysis (PCA) Kernel PCA Boundary Intensity Models Gaussian distribution AdaBoosted histogram classifiers Heuristics Cootes et al. [IEEE PAMI 01], Twining et al. [BMVC’01] Cootes et al. [IEEE PAMI 01], Li [ICCV’05], Kainmuller et al. [MICCAI’07]

Organ Detection (MSL) Goal: Find the bounding box Idea The parameter space is 9D. 3D positions, 3D scales and 3D orientations. Idea Uniform and exhaustive search is unnecessary How: decompose the problem into three steps position estimation, position-scale estimation and finally position-scale-orientation estimation. Zheng et al. [ICCV’07]

Two ASM-based Systems Ling et al. [CVPR’08] Statistical shape models Kainmuller et al. [MICCAI’07] Ling et al. [CVPR’08] Statistical shape models PCA 43 CT volumes Boundary intensity model Heuristics Liver detection Lungs detection and DICOM info Performance Ranked first in a recent liver segmentation competition. 10 testing volumes 1.1mm (the average symmetric surface distance) 15 minutes. Statistical Shape Models Statistical Shape models PCA, hierarchical shape pyramids 75 volumes Boundary intensity model A boundary classifier Liver detection MSL Performance 5 fold cross validation 1.59 mm (the average symmetric surface distance) 1.38 mm (the median) 12 seconds. Boundary Intensity Model Liver Detection Performance

Other approaches Active contour Level Sets Graph cuts Adaptive thresholding Kass et al. [IJCV 88], Osher et al. [J. Comput. Phys. 88], Boykov et. al. [IEEE PAMI 01], Shi et. al. [IEEE PAMI 00], Kobashi et. al. [PR 95]

Outline Introduction Previous Work Initial Results Proposal Conclusions

ASM Framework with Our Methods Training CT Volumes MDL with 2DPCA for 3D Point Correspondence Tensor-based Statistical Shape Models Known point correspondences Learning Organ Detection Learning Boundary Intensity Model Graph Cut based Boundary Refinement Organ Detection A Testing CT Volume Shape Model Initialization Learned Statistical Shape Model Organ Detector Detected Bounding Box Initial Shape Final Shape Learned Boundary Intensity Model Training phase Testing phase

Experimental Setting Datasets: Leave-one-out cross validation 4 types of organs (livers, left kidneys, right kidneys, spleens) 15-20 subjects Leave-one-out cross validation Measure the reconstruction error Metrics: Euclidean and Hausdorff distance

MDL-2DPCA MDL-based objective function Idea: Generalize the objective function to 2DPCA space Replace eigenvalues from PCA with from 2DPCA How: Gradient descent Comparisons: original MDL vs. MDL-2DPCA * Chen and Shapiro [to appear in EMBC’09]

Results (3D Point Correspondences) Livers Left Kidneys Reconstruction error Original MDL MDL-2DPCA # eigenvectors Spleens Right Kidneys

Tensor-based SSM Idea: Tensor-based dimension reduction methods 2DPCA Parafac model Tucker decomposition Comparisons: PCA vs. Tensor-based dimension reduction * Chen and Shapiro [to appear in EMBC’09]

Results (Statistical Shape Models) Livers Left Kidneys Parafac Reconstruction error PCA Tucker 2DPCA # eigenvectors Right Kidneys Spleens

Organ Detection (Boosting Approach) Idea: Classify whether an image block contains an organ of interest How: Partition slices into non-overlapping 32x32 blocks Global features: gray-tone histogram of the image slice and its slice index Local features: the position of a block, the mean and variance of its intensity values, and its intensity histogram. 20,000 SVM linear classifiers + Adaboosting Comparisons: Manual vs. Adaboosting

Results (Organ Detection) Positive (predicted) Negative (predicted) Positive (actual) 96.23% 3.77% Negative (actual) 4.57% 95.43% Livers (Training) Positive (predicted) Negative (predicted) Positive (actual) 91.23% 8.77% Negative (actual) 6.57% 93.43% Livers (Testing)

True Positive : Green, False Positive: Blue False Negative: Red, True Negative: Cyan Sub. 1 Sub. 2

Graph Cuts Based Boundary Refinement Idea: Adding hard constraints to min s-t cuts Min s-t cuts with side constraints NP-hard in general cases Approximation algorithm: standard rounding algorithm Comparisons: with constraints vs. without constraints Chen and Shapiro [ICPR’08]

Results (Boundary Refinement) Initial Contour Slice 1 Slice 2 without with

Outline Introduction Previous Work Initial Results Proposal Conclusions

Framework with Proposal Training CT Volumes Nonlinear and Robust MDL for 3D Point Correspondence Tensor-based Statistical Shape Models Known point correspondences Boosting/Bagging MSL Learning Boundary Intensity Model Robust Graph Cut based Boundary Refinement with TU constraints Organ Detection A Testing CT Volume Shape Model Initialization Learned Statistical Shape Model Organ Detector Detected Bounding Box Initial Shape Final Shape Learned Boundary Intensity Model Training phase Testing phase

Nonlinearity and Robustness Idea: generalize MDL objective function to KPCA, tensor-based decompositions and robust PCA How: replace eigenvalues from PCA with Eigenvalues from KPCA High-order singular values from Tensor decomposition Eigenvalues from Robust PCA Gradient descent for optimization

Boosting/Bagging MSL Goal: Improve MSL for organ detection Idea: Boosting/Bagging MSL How: Treat MSL as a base classifier Use AdaBoosting/Bagging to combine many MSL classifiers

Robust Graph Cuts Goal: Achieve robustness by Minimax minimize the worst case loss in objective values Idea: Some edge weights are unreliable Minimize maximum cut values caused by unreliable edge weights How: Solve a series of modified min s-t cut problems Polynomial time algorithm

Totally Unimodular (TU) Hard Constraints Goal: Balance the performance and efficiency Hard constraints: helpful but NP-hard Idea: Special cases where global optimal solutions can be found in polynomial time How: Find interesting semantic constraints that can be represented as TU constraints Specialized algorithms for TU problems

Outline Introduction Previous Work Initial Results Proposal Conclusions

Training CT Volumes MDL with 2DPCA for 3D Point Correspondence Tensor-based Statistical Shape Models Known point correspondences Learning Organ Detection Learning Boundary Intensity Model Graph Cut based Boundary Refinement Organ Detection A Testing CT Volume Shape Model Initialization Learned Statistical Shape Model Organ Detector Detected Bounding Box Initial Shape Final Shape Learned Boundary Intensity Model Training CT Volumes 3D point correspondence Learning Statistical Shape Models Known point correspondences Learning Organ Detection Learning Boundary Intensity Model Boundary Refinement Organ Detection A Testing CT Volume Shape Model Initialization Learned Statistical Shape Model Learned Boundary Intensity Model Organ Detector Detected Bounding Box Initial Shape Final Shape Training CT Volumes Nonlinear and Robust MDL for 3D Point Correspondence Tensor-based Statistical Shape Models Known point correspondences Boosting/Bagging MSL Learning Boundary Intensity Model Robust Graph Cut based Boundary Refinement with TU constraints Organ Detection A Testing CT Volume Shape Model Initialization Learned Statistical Shape Model Organ Detector Detected Bounding Box Initial Shape Final Shape Learned Boundary Intensity Model

Proposal General Approach and Many Applications 2D point correspondence Image segmentation Stereo Image restoration Interactive image segmentation Face alignment Clustering

Acknowledgement Committee Members Linda’s group members Professor Linda Shapiro (CSE) as Chair Professor James F. Brinkley (BioStr) Professor Jenq-Neng Hwang (EE) Professor Rajesh Rao (CSE) Professor Mark Ganter (ME) as GSR Linda’s group members

Thank You.

Gantt Chart Summer 09 Autumn 10 Winter 10 Spring 10 A working system   Summer 09 Autumn 10 Winter 10 Spring 10 A working system Better organ detector Better boundary refinement Better 3D point correspondence System Integration /Tuning/Testing Dissertation Writing Defense