1. MEDICAL IMAGE ANALYSIS Marek Brejl Vital Images, Inc.

2. 2 Vital Images, Inc. 2001 Outline Brief introduction to Medical Image Analysis Recent development in: Model based detection and segmentation Automated training/model generation

3. 3 Vital Images, Inc. 2001 Medical Image Data CT MR X-ray Nuclear Ultrasound

4. 4 Vital Images, Inc. 2001 Image Processing in Medical Settings Acquisition Processing Visualization/Reporting Detection Analysis Segmentation Preprocessing Diagnoses

5. 5 Vital Images, Inc. 2001 Data Processing 1.Preprocessing Filtering, registration 2.Detection Finding objects (nodules, polyps, organs) 3.Segmentation Exact delimitation of objects 4.Analysis Measurement (volume, curvature); Functional Imaging (Perfusion) 5.Classification/diagnoses

6. 6 Vital Images, Inc. 2001 Examples: Preprocessing Filtering/image enhancement OriginalEnhanced

7. 7 Vital Images, Inc. 2001 Examples: Preprocessing Registration Target Template registration

8. 8 Vital Images, Inc. 2001 Detection Find location of objects of interest (find or detect objects without prior knowledge about their location/existence) Bones Organs Polyps in colon Nodules in lungs

9. 9 Vital Images, Inc. 2001 Segmentation Exactly delimitate objects, once they are detected (found) Any object of predictable shape (organs, bones, vessel segments) Liver Cardiac imaging (left ventricle) Brain

10. 10 Vital Images, Inc. 2001 Segmentation

11. 11 Vital Images, Inc. 2001 Segmentation Brain segmentation Heart Lung Body

12. 12 Vital Images, Inc. 2001 Analyses Measurement Volume - growth rate Vessel stenosis Functional imaging Stroke Cardiac perfusion Tumor perfusion Cardiac function LV motion Injection fraction

13. 13 Vital Images, Inc. 2001 Classification/Diagnoses Comparison to developed atlases Use of knowledge databases Classify as normal/abnormal (brain structure) Classify lung nodules as benign/malignant Determine cancer/non-cancer

14. 14 Vital Images, Inc. 2001 Data Processing – Example Lung screening 1.Preprocessing Reduce noise, threshold image 2.Detection Find lungs, find nodules location 3.Segmentation Accurately segment out nodules 4.Analysis Measure volume, texture, curvature 5.Classification/diagnoses Classify as benign or malignant V=35mm 3 S=25mm 2 V=12mm 3 S=5mm 2 - benign - malignant

15. 15 Vital Images, Inc. 2001 Model-based detection and segmentation Acquisition Processing Visualization/Reporting Detection Analysis Segmentation Preprocessing Diagnoses

16. 16 Vital Images, Inc. 2001 Image Segmentation 1.Thresholding 2.Region-based 3.Edge-based Advanced (edge-based) OriginalThreshold Region-basedEdge-based OriginalThreshold Region-basedEdge-based

17. 17 Vital Images, Inc. 2001 Advanced Image Segmentation 1. Segmentation criterion design 2. Segmentation criterion optimization graph searching gradient descent

18. 18 Vital Images, Inc. 2001 Goals Design of methodology for automated model-based image segmentation (segmentation via boundary detection)

19. 19 Vital Images, Inc. 2001 Model-Based Segmentation 1.Training set design 2.Training a. Shape Model b. Border Appearance Model 3.Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

20. 20 Vital Images, Inc. 2001 Training Set Design Selection of examples Contains expected variability in local border appearance Contains expected variability in object shape Segmentation examples Outlined objects Registration landmarks manual outline manual landmarks computed landmarks

21. 21 Vital Images, Inc. 2001 Training Purpose: To create representation of the objects of interest presented in the training set Statistical models: (mean, variance) Shape Model Border Appearance Model Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

22. 22 Vital Images, Inc. 2001 Shape Model Point Distribution Model (PDM) (Cootes et al., 1992) Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection Example training shapes (N=12) Alligned shapes

23. 23 Vital Images, Inc. 2001 Shape Model Point Distribution Model Object representation: Shape Model:

24. 24 Vital Images, Inc. 2001 Model Size Reduction Eigenvalues and eigenvectors of the variance matrix Eigenvectors – directions of the main modes of variation Eigenvalues – importance of the variation Size reduction – keep only most important modes

25. 25 Vital Images, Inc. 2001 Border Appearance Model Gray levels Edge values 1 pixel 31 pixels 1 2 3 1 2 3 1:2:3: 1:2:3: Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

26. 26 Vital Images, Inc. 2001 BAM – Basic Idea BA - represented by appearance vector Training vectors grouped to create the model

27. 27 Vital Images, Inc. 2001 BAM – Basic Idea Comparison of image data to the BA model

28. 28 Vital Images, Inc. 2001 BAM - Clustering Fuzzy c-means clustering (Bezdek et al.,1981) Input vectors: f i Partitioning matrix: U Cluster centers: f Objective function to be minimized:

29. 29 Vital Images, Inc. 2001 fit Value Computation Fuzzy model Euclidean distance Gaussian model F performs some inversion Best fit:

30. 30 Vital Images, Inc. 2001 Border Appearance Model BAM – tool for: Modeling local border properties Comparing image data to the model Computing cost values based on the comparison Further improvements Spatial information Counter-examples Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

31. 31 Vital Images, Inc. 2001 BAM – Spatial Information BAM records: Appearance vectors Location of every vector on the original border

32. 32 Vital Images, Inc. 2001 Border Appearance Vector

33. 33 Vital Images, Inc. 2001 Training - Summary Shape Model Border Appearance Model border appearance appearance spatial information Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

34. 34 Vital Images, Inc. 2001 Approximate Object Location Algorithm for detection of approximate object location Requirements: detects objects defined by the models detects objects modified by rigid transforms (rotation, translation, scaling) Insensitive to shape variability (captured in the training set) Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

35. 35 Vital Images, Inc. 2001 Generalized Hough Transform (Ballard, 1981) Detection of objects of arbitrary a priori known shapes Object Shape R-Table ImageAccumulator Array 1. 2a.2b. 3.

36. 36 Vital Images, Inc. 2001 R-Table 1. R-Table construction: 2. Accumulator array updates: Object Shape R-Table ImageAccumulator Array 1. 2a.2b. 3.

37. 37 Vital Images, Inc. 2001 Problems of Generalized HT Searches for strong edges Shape-Variant Hough TransformRemedy: Shape-Variant Hough Transform Cannot handle shape variance

38. 38 Vital Images, Inc. 2001 Shape-variant Hough Transform Use of Border Appearance Model: Gradient magnitudeBAM General Spatial information

39. 39 Vital Images, Inc. 2001 SVHT – Shape Variance 1.Alignment of training shapes Same algorithm as for PDM 2.Encoding shape variance in the R-Table All border directions are encoded 3.Shape reconstruction Several shapes available for reconstruction Selection of the most probable shape

40. 40 Vital Images, Inc. 2001 SVHT – R-Table Construction

41. 41 Vital Images, Inc. 2001 SVHT – Object Reconstruction Reconstructed all, selected the best fit 1. fit=98 fit=45 fit=8 2. fit=65 fit=97 fit=30 3. fit=6 fit=32 fit=94 Original Imagefit values

42. 42 Vital Images, Inc. 2001 Hough TransformActive Hough Transform Shape Variance - Comparisons Hough TransformActive Hough TransformHough Transform Original Image

43. 43 Vital Images, Inc. 2001 Shape-Variant Hough Transform Summary Algorithm for detection of approximate object location Automated, information derived form training set BAM removed dependency on high gradient magnitude Insensitive to shape variability Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

44. 44 Vital Images, Inc. 2001 Accurate Boundary Detection Any edge-based image segmentation algorithm Automated design of cost function - BAM Fit values replace gradient magnitude Examples Dynamic Programming Snakes Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

45. 45 Vital Images, Inc. 2001 Segmentation - Summary Active Hough Transform Automated detection of approximate object location Provides initialization for accurate border detection algorithms Accurate border detection Any algorithm fit values based on BAM substitutes gradient-based terms in segmentation criterion Automated Segmentation 1. Training set design 2. Training a. Shape Model b. Border Appearance Model 3. Segmentation Step 1: Approximate object location Step 2: Accurate boundary detection

46. 46 Vital Images, Inc. 2001 Experimental Methods 5 segmentation tasks 1.Epicardial border in MR images of thorax 58 images, 21 used for training 2.Endocardial border in MR images of thorax 58 images, 21 used for training 3.Corpus Callosum in MR images of brain 90 images, 15 used for training 4.Cerebellum in MR images of brain 90 images, 6 used for training 5.Vertebrae in MR images of spine 55 images, 15 used for training, (235 vertebrae to segment)

47. 47 Vital Images, Inc. 2001 Results Original ImageExpected Border Dynamic ProgrammingSV Hough Transform

48. 48 Vital Images, Inc. 2001 Results Original ImageExpected Border Dynamic ProgrammingSV Hough Transform

49. 49 Vital Images, Inc. 2001 Results Original ImageExpected Border Dynamic ProgrammingSV Hough Transform

50. 50 Vital Images, Inc. 2001 Results Original ImageExpected Border Dynamic ProgrammingSV Hough Transform

51. 51 Vital Images, Inc. 2001 Results Original ImageExpected Border Dynamic ProgrammingSV Hough Transform

52. 52 Vital Images, Inc. 2001 Results Original ImageExpected Border Dynamic ProgrammingSV Hough Transform

53. 53 Vital Images, Inc. 2001 Detection Accuracy

54. 54 Vital Images, Inc. 2001 Comparison of Approximate and Accurate Detection

55. 55 Vital Images, Inc. 2001 Study 1: Dependency on Number of Training Data

56. 56 Vital Images, Inc. 2001 Study 1: Statistical Significance

57. 57 Vital Images, Inc. 2001 Study 2: Dependency on Training Data Selection

58. 58 Vital Images, Inc. 2001 Study 2: Statistical Significance

59. 59 Vital Images, Inc. 2001 Discussion - Conclusion Methodology for automated model-based image segmentation 1. Training set design 2. Training (Shape Model, Border Appearance Model) 3. Segmentation (Shape-Variant Hough Transform, Accurate Border Detection) Fully automated – requires small set of training examples Two step segmentation approximate object location accurate boundary detection

60. 60 Vital Images, Inc. 2001 Discussion - Conclusion The method is applicable to the following segmentation tasks: Objects with well defined shape Objects with reasonably consistent border appearance Representative set of examples is available Edge-based image segmentation method is appropriate for the segmentation task